Log In Sign Up

rash-id.com
robots.txt

Robots Exclusion Standard data for rash-id.com

Resource Scan

Scan Details

Site Domain rash-id.com
Base Domain rash-id.com
Scan Status Ok
Last Scan2025-08-26T13:40:38+00:00
Next Scan 2025-09-09T13:40:38+00:00

Last Scan

Scanned2025-08-26T13:40:38+00:00
URL https://rash-id.com/robots.txt
Redirect https://www.rash-id.com/robots.txt
Redirect Domain www.rash-id.com
Redirect Base rash-id.com
Domain IPs 76.76.21.21
Redirect IPs 66.33.60.193, 76.76.21.22
Response IP 76.76.21.164
Found Yes
Hash f1c076e347266cd494a48579383d6ab7c81ae16748db2c5e591cc447f697a85d
SimHash 7233da50e6a2

Groups

googlebot
bingbot
slurp
duckduckbot
baiduspider
yandexbot
facebookexternalhit
twitterbot
linkedinbot
whatsapp
applebot

Rule Path
Allow /
Disallow /admin/
Disallow /api/

Other Records

Field Value
crawl-delay 0

gptbot
chatgpt-user

Rule Path
Allow /
Disallow /admin/
Disallow /api/

Other Records

Field Value
crawl-delay 1

anthropic-ai
claude-web

Rule Path
Allow /
Disallow /admin/
Disallow /api/

Other Records

Field Value
crawl-delay 1

google-extended
googleother
googleother-image
googleother-video

Rule Path
Allow /
Disallow /admin/
Disallow /api/

Other Records

Field Value
crawl-delay 1

facebookbot

Rule Path
Allow /
Disallow /admin/
Disallow /api/

Other Records

Field Value
crawl-delay 1

applebot-extended

Rule Path
Allow /
Disallow /admin/
Disallow /api/

Other Records

Field Value
crawl-delay 1

cohere-ai

Rule Path
Allow /
Disallow /admin/
Disallow /api/

Other Records

Field Value
crawl-delay 1

perplexitybot
youbot

Rule Path
Allow /
Disallow /admin/
Disallow /api/

Other Records

Field Value
crawl-delay 2

ccbot
bytespider
diffbot
amazonbot
awariorssbot
awariosmartbot
dataforseobot
magpie-crawler
newsnow
news-please
peer39_crawler
peer39_crawler/1.0
omgili
omgilibot
icc-crawler
img2dataset
scrapy

Rule Path
Disallow /

ahrefsbot
semrushbot
mj12bot
dotbot
screaming frog seo spider

Rule Path
Allow /
Disallow /admin/
Disallow /api/

Other Records

Field Value
crawl-delay 2

sentibot
blexbot
bubing
buck
ltx71
mb2345browser
megaindex
mediatoolkitbot
mj12bot
npbot
nutch
piplbot
python-requests
python-urllib
python-urllib3
r6_bot
randombot
repomonkey
riddler
rogerbot
semrushbot
seznambot
sitebot
turnitinbot
vagabondo
voilabot
wbsearchbot
www-mechanize
xenu

Rule Path
Disallow /

ia_archiver
wayback machine

Rule Path
Allow /
Disallow /admin/
Disallow /api/

Other Records

Field Value
crawl-delay 5

*

Rule Path
Allow /
Disallow /admin/
Disallow /api/
Disallow /private/
Disallow /*.json$
Disallow /*?*
Disallow /search

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://www.rash-id.com/sitemap.xml
Back to top

Comments

  • Robots.txt for www.rash-id.com
  • Last updated: January 2025
  • ===========================================
  • TRADITIONAL SEARCH ENGINES - ALLOW
  • ===========================================
  • ===========================================
  • TRUSTED AI COMPANIES - ALLOW FOR TRAINING
  • ===========================================
  • OpenAI
  • Anthropic
  • Google AI
  • Meta/Facebook AI
  • Apple AI
  • Cohere
  • ===========================================
  • AI SEARCH/ANSWER ENGINES - ALLOW WITH LIMITS
  • ===========================================
  • ===========================================
  • UNTRUSTED AI/DATA CRAWLERS - BLOCK
  • ===========================================
  • ===========================================
  • SEO & ANALYTICS TOOLS - ALLOW
  • ===========================================
  • ===========================================
  • AGGRESSIVE/MALICIOUS BOTS - BLOCK
  • ===========================================
  • ===========================================
  • ARCHIVING BOTS - YOUR CHOICE
  • ===========================================
  • ===========================================
  • DEFAULT FOR ALL OTHERS
  • ===========================================
  • ===========================================
  • SITEMAP
  • ===========================================
Back to top