Log In Sign Up

rainadmin.com
robots.txt

Robots Exclusion Standard data for rainadmin.com

Resource Scan

Scan Details

Site Domain rainadmin.com
Base Domain rainadmin.com
Scan Status Ok
Last Scan2024-11-08T11:03:09+00:00
Next Scan 2024-12-08T11:03:09+00:00

Last Scan

Scanned2024-11-08T11:03:09+00:00
URL https://www.rainadmin.com/robots.txt
Domain IPs 108.138.26.21, 108.138.26.46, 108.138.26.65, 108.138.26.96, 2600:9000:248c:1600:8:8f39:1c00:93a1, 2600:9000:248c:3600:8:8f39:1c00:93a1, 2600:9000:248c:400:8:8f39:1c00:93a1, 2600:9000:248c:6a00:8:8f39:1c00:93a1, 2600:9000:248c:6c00:8:8f39:1c00:93a1, 2600:9000:248c:7e00:8:8f39:1c00:93a1, 2600:9000:248c:ba00:8:8f39:1c00:93a1, 2600:9000:248c:d600:8:8f39:1c00:93a1
Response IP 3.164.206.8
Found Yes
Hash 642b7e7974b26ec2bd3a0ffb218b2c859939360ca2013d6abe7b84007cc271ff
SimHash d31455d23f09

Groups

*

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

blexbot
mj12bot
goodzer
ahrefsbot
spbot
dotbot
dotbot
turnitinbot
seznambot
the knowledge ai
checkmarknetwork
ever accountable
seekportbot
claudebot
mauibot
houzzbot
baiduspider
baiduspider-image
serpstatbot
sogou blog
sogou inst spider
sogou news spider
sogou orion spider
sogou spider2
sogou web spider
uptimebot
yandex
yandexmobilebot
zoominfobot
megaindex.ru
alphaseobot-sa
proximic
amazonbot
petalbot
re-re studio
barkrowler
siteauditbot
awariorssbot
awariosmartbot
bytespider
dataforseobot
ioncrawl
nutch
oai-searchbot
gptbot
chatgpt-user

Rule Path
Disallow /

semrushbot-ba
semrushbot

Rule Path
Disallow /
Back to top

Comments

  • See if Semrush can behave - March 2023.
  • Update Nov 2023, no, they can't behave.
Back to top

Warnings

  • 1 invalid line.
Back to top