Log In Sign Up

innonthecliff.com
robots.txt

Robots Exclusion Standard data for innonthecliff.com

Resource Scan

Scan Details

Site Domain innonthecliff.com
Base Domain innonthecliff.com
Scan Status Ok
Last Scan2024-11-05T21:55:31+00:00
Next Scan 2024-12-05T21:55:31+00:00

Last Scan

Scanned2024-11-05T21:55:31+00:00
URL http://innonthecliff.com/robots.txt
Domain IPs 34.196.202.11
Response IP 34.196.202.11
Found Yes
Hash 642b7e7974b26ec2bd3a0ffb218b2c859939360ca2013d6abe7b84007cc271ff
SimHash d31455d23f09

Groups

*

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

blexbot
mj12bot
goodzer
ahrefsbot
spbot
dotbot
dotbot
turnitinbot
seznambot
the knowledge ai
checkmarknetwork
ever accountable
seekportbot
claudebot
mauibot
houzzbot
baiduspider
baiduspider-image
serpstatbot
sogou blog
sogou inst spider
sogou news spider
sogou orion spider
sogou spider2
sogou web spider
uptimebot
yandex
yandexmobilebot
zoominfobot
megaindex.ru
alphaseobot-sa
proximic
amazonbot
petalbot
re-re studio
barkrowler
siteauditbot
awariorssbot
awariosmartbot
bytespider
dataforseobot
ioncrawl
nutch
oai-searchbot
gptbot
chatgpt-user

Rule Path
Disallow /

semrushbot-ba
semrushbot

Rule Path
Disallow /
Back to top

Comments

  • See if Semrush can behave - March 2023.
  • Update Nov 2023, no, they can't behave.
Back to top

Warnings

  • 1 invalid line.
Back to top