Log In Sign Up

bugs.chromium.org
robots.txt

Robots Exclusion Standard data for bugs.chromium.org

Resource Scan

Scan Details

Site Domain bugs.chromium.org
Base Domain chromium.org
Scan Status Ok
Last Scan2024-04-24T14:07:04+00:00
Next Scan 2024-05-24T14:07:04+00:00

Last Scan

Scanned2024-04-24T14:07:04+00:00
URL https://bugs.chromium.org/robots.txt
Domain IPs 2404:6800:4003:c06::79, 64.233.170.121
Response IP 74.125.68.121
Found Yes
Hash 10d48d0844e3bff0cdd5375c3b04439a6a4de0e57f4c8eb31fbefe141cdec7ec
SimHash 8834d9888375

Groups

*

Rule Path
Disallow /
Allow /$
Allow /hosting
Allow /p/*/adminIntro
Allow /prpc/*
Allow /static/*
Allow /p/*/issues/detail?id=*
Allow /p/*/issues/detail_ezt?id=*
Disallow /p/*/issues/detail?id=*&*
Disallow /p/*/issues/detail?*&id=*

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • Start by disallowing everything.
  • Some specific things are okay, though.
  • Allow files needed to render the new UI
  • Query strings are hard. We only allow ?id=N, no other parameters.
  • 10 second crawl delay for bots that honor it.
Back to top