Log In Sign Up

gomuraw.com
robots.txt

Robots Exclusion Standard data for gomuraw.com

Resource Scan

Scan Details

Site Domain gomuraw.com
Base Domain gomuraw.com
Scan Status Ok
Last Scan2025-12-06T11:18:43+00:00
Next Scan 2026-01-05T11:18:43+00:00

Last Scan

Scanned2025-12-06T11:18:43+00:00
URL https://gomuraw.com/robots.txt
Domain IPs 104.21.39.3, 172.67.141.179, 2606:4700:3035::6815:2703, 2606:4700:3037::ac43:8db3
Response IP 104.21.39.3
Found Yes
Hash 368003083d7017e5b3060c7eaf4877ed0f3acbac4fdde1f70f21a4eab4b94433
SimHash 6509fa51c597

Groups

*

Rule Path
Disallow /_next/data/

googlebot

Rule Path
Disallow /_next/data/

bingbot

Rule Path
Disallow /_next/data/

Other Records

Field Value
crawl-delay 5

slurp

Rule Path
Disallow /_next/data/

Other Records

Field Value
crawl-delay 5

duckduckbot

Rule Path
Disallow /_next/data/

Other Records

Field Value
crawl-delay 5

baiduspider

Rule Path
Disallow /_next/data/

Other Records

Field Value
crawl-delay 5

yandex

Rule Path
Disallow /_next/data/

Other Records

Field Value
crawl-delay 5

gptbot

Rule Path
Disallow /_next/data/

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://gomuraw.com/sitemap.xml
Back to top

Comments

  • ===========================
  • Global rules (apply to all bots)
  • ===========================
  • ===========================
  • Major Search Engines (good bots)
  • ===========================
  • Googlebot (Google ignores Crawl-delay → control in Search Console)
  • Bingbot (respects Crawl-delay)
  • Yahoo Slurp (uses Bing’s indexer)
  • DuckDuckBot
  • Baiduspider
  • YandexBot
  • GPTBot (OpenAI crawler)
Back to top