Log In Sign Up

broadwaypodcastnetwork.com
robots.txt

Robots Exclusion Standard data for broadwaypodcastnetwork.com

Resource Scan

Scan Details

Site Domain broadwaypodcastnetwork.com
Base Domain broadwaypodcastnetwork.com
Scan Status Ok
Last Scan2026-02-05T19:10:47+00:00
Next Scan 2026-02-12T19:10:47+00:00

Last Scan

Scanned2026-02-05T19:10:47+00:00
URL https://broadwaypodcastnetwork.com/robots.txt
Domain IPs 2600:9000:2175:400:17:aadf:e840:93a1, 2600:9000:2175:4600:17:aadf:e840:93a1, 2600:9000:2175:7e00:17:aadf:e840:93a1, 2600:9000:2175:8200:17:aadf:e840:93a1, 2600:9000:2175:a800:17:aadf:e840:93a1, 2600:9000:2175:ae00:17:aadf:e840:93a1, 2600:9000:2175:c600:17:aadf:e840:93a1, 2600:9000:2175:d000:17:aadf:e840:93a1, 52.222.236.9, 52.222.236.92, 52.222.236.98, 52.222.236.99
Response IP 3.169.71.18
Found Yes
Hash 2c34b36beb4ed9966d6c6e249e87db41dbe54550ae39c7c7b74b059f9d4d001f
SimHash 511cdc30f7a0

Groups

*

Rule Path
Allow /
Disallow /*/*/*/customFields
Disallow /*/*/meta
Disallow /unknown/*
Disallow /api/
Disallow /admin/

Other Records

Field Value
crawl-delay 1

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

bomborabot

Rule Path
Disallow /

amazonbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

googlebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

duckduckbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 2

gptbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

chatgpt-user

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

claude-web

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

anthropic-ai

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

claudebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

perplexitybot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

applebot-extended

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

google-extended

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://broadwaypodcastnetwork.com/sitemap.xml
Back to top

Comments

  • Broadway Podcast Network - robots.txt
  • Allow all crawlers by default with rate limiting
  • Disallow internal/admin paths
  • Block aggressive SEO/marketing bots entirely (they provide no value)
  • Block TikTok crawler - extremely aggressive
  • Block other known aggressive crawlers
  • Major search engines - allow with reasonable rate limiting
  • LLM crawlers - allow with rate limiting
  • Sitemap location
Back to top