www.staging2.horseracingnation.com
robots.txt

Robots Exclusion Standard data for www.staging2.horseracingnation.com

Resource Scan

Scan Details

Site Domain www.staging2.horseracingnation.com
Base Domain horseracingnation.com
Scan Status Ok
Last Scan2024-10-24T08:27:06+00:00
Next Scan 2024-11-23T08:27:06+00:00

Last Scan

Scanned2024-10-24T08:27:06+00:00
URL https://www.staging2.horseracingnation.com/robots.txt
Domain IPs 3.230.229.153, 44.221.162.8, 52.20.199.203, 54.85.242.210
Response IP 54.85.242.210
Found Yes
Hash d8ce5baa8e04b0917cab5c2a6c0cefe5a46d84f178bef2932533437942ee7309
SimHash 305a4bc1c764

Groups

*

Rule Path
Disallow /edit
Disallow /forgotpassword.aspx
Disallow /Login.aspx
Disallow /login.aspx
Disallow /signup.aspx
Disallow /session
Disallow /terms

ai2bot
ai2bot-dolma
amazonbot
applebot
applebot-extended
bytespider
ccbot
chatgpt-user
claude-web
claudebot
diffbot
facebookbot
friendlycrawler
gptbot
google-extended
googleother
googleother-image
googleother-video
icc-crawler
imagesiftbot
meta-externalagent
meta-externalfetcher
oai-searchbot
perplexitybot
petalbot
scrapy
the knowledge ai
timpibot
velenpublicwebcrawler
webzio-extended
youbot
anthropic-ai
cohere-ai
facebookexternalhit
iaskspider/2.0
img2dataset
omgili
omgilibot

Rule Path
Disallow /

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • AI / ML scrapers