Log In Sign Up

therainshed.com
robots.txt

Robots Exclusion Standard data for therainshed.com

Resource Scan

Scan Details

Site Domain therainshed.com
Base Domain therainshed.com
Scan Status Ok
Last Scan2025-09-12T02:13:28+00:00
Next Scan 2025-10-12T02:13:28+00:00

Last Scan

Scanned2025-09-12T02:13:28+00:00
URL https://therainshed.com/robots.txt
Domain IPs 18.66.218.18, 18.66.218.22, 18.66.218.24, 18.66.218.76, 2600:9000:206f:2800:18:5421:5ec0:93a1, 2600:9000:206f:6400:18:5421:5ec0:93a1, 2600:9000:206f:7c00:18:5421:5ec0:93a1, 2600:9000:206f:800:18:5421:5ec0:93a1, 2600:9000:206f:ac00:18:5421:5ec0:93a1, 2600:9000:206f:d800:18:5421:5ec0:93a1, 2600:9000:206f:e600:18:5421:5ec0:93a1, 2600:9000:206f:e800:18:5421:5ec0:93a1
Response IP 18.165.72.64
Found Yes
Hash 53c9a7d6ca3d66ecec83396de732e4df7dba04677ee7a9d1a8ac05cfa8fd246c
SimHash 335c75925d89

Groups

*

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

blexbot
mj12bot
goodzer
ahrefsbot
spbot
dotbot
dotbot
turnitinbot
seznambot
the knowledge ai
checkmarknetwork
ever accountable
seekportbot
claudebot
mauibot
houzzbot
baiduspider
baiduspider-image
serpstatbot
sogou blog
sogou inst spider
sogou news spider
sogou orion spider
sogou spider2
sogou web spider
uptimebot
yandex
yandexmobilebot
zoominfobot
megaindex.ru
alphaseobot-sa
proximic
amazonbot
petalbot
re-re studio
barkrowler
siteauditbot
awariorssbot
awariosmartbot
bytespider
dataforseobot
ioncrawl
nutch
gptbot
chatgpt-user
imagesiftbot

Rule Path
Disallow /

semrushbot-ba
semrushbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap /sitemap.xml
Back to top

Comments

  • Need to allow the below bot, OAI-SearchBot, in order for our merchants products to show up in open ai. Im leaving the bot in the file, commented out, in case someone else notices the bot and wants to block it
  • User-agent: OAI-SearchBot
  • See if Semrush can behave - March 2023.
  • Update Nov 2023, no, they can't behave.
Back to top

Warnings

  • 1 invalid line.
Back to top