Log In Sign Up

coh3stats.com
robots.txt

Robots Exclusion Standard data for coh3stats.com

Resource Scan

Scan Details

Site Domain coh3stats.com
Base Domain coh3stats.com
Scan Status Ok
Last Scan2024-11-07T05:54:30+00:00
Next Scan 2024-11-21T05:54:30+00:00

Last Scan

Scanned2024-11-07T05:54:30+00:00
URL https://coh3stats.com/robots.txt
Domain IPs 104.21.62.212, 172.67.139.87, 2606:4700:3030::6815:3ed4, 2606:4700:3037::ac43:8b57
Response IP 172.67.139.87
Found Yes
Hash 0b8307c5a653ab48c0f5c9b73f31ca66f4e1efdf8ce97a8289ff914157fd37ed
SimHash 60349c41e6b3

Groups

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

timpibot/0.9

Rule Path
Disallow /

bingbot

Rule Path
Disallow /leaderboards
Disallow /players
Allow /

gptbot

Rule Path
Disallow /leaderboards
Disallow /players
Allow /

*

Rule Path
Disallow /api
Disallow /tests
Disallow /leaderboards?*start=
Allow /
Back to top

Other Records

Field Value
sitemap https://coh3stats.com/sitemap.xml
sitemap https://coh3stats.com/sitemap.txt
Back to top

Comments

  • We might change to block all robots / and allow only search engines
  • Block all the stupid bots
  • Bing bot is retarded, it's crawling leaderboards and all players on the platform
  • GPTBot is retarded the same way the Bing bot is
  • Ban crawling of the leaderboards via the start parameter
  • Host
  • Sitemaps
Back to top

Warnings

  • `host` is not a known field.
Back to top