driftdreamer.com
robots.txt

Robots Exclusion Standard data for driftdreamer.com

Resource Scan

Scan Details

Site Domain driftdreamer.com
Base Domain driftdreamer.com
Scan Status Ok
Last Scan2025-09-27T09:17:35+00:00
Next Scan 2025-10-27T09:17:35+00:00

Last Scan

Scanned2025-09-27T09:17:35+00:00
URL https://driftdreamer.com/robots.txt
Domain IPs 104.21.27.188, 172.67.143.116, 2606:4700:3031::ac43:8f74, 2606:4700:3037::6815:1bbc
Response IP 104.21.27.188
Found Yes
Hash 32355db04f3536df20a9190ff0225e408aa6e2d466529c3b1004d2e56c023647
SimHash 5a9a8940ea19

Groups

yandexbot

Rule Path
Disallow /

yadirectfetcher

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yandexvideo

Rule Path
Disallow /

yandexnews

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

*

Rule Path
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /wp-content/themes/
Disallow /cgi-bin/
Disallow /api/

Comments

  • Block AI training crawlers