Log In Sign Up

dreamescape.to
robots.txt

Robots Exclusion Standard data for dreamescape.to

Resource Scan

Scan Details

Site Domain dreamescape.to
Base Domain dreamescape.to
Scan Status Ok
Last Scan2024-09-23T20:25:34+00:00
Next Scan 2024-09-30T20:25:34+00:00

Last Scan

Scanned2024-09-23T20:25:34+00:00
URL https://dreamescape.to/robots.txt
Domain IPs 78.141.234.102
Response IP 78.141.234.102
Found Yes
Hash 4db5c32a8157cf87494662cd8a70777df8311435c11e6f4d2d3aa5f2b4e1fa55
SimHash f6345959e5a1

Groups

*

Rule Path
Disallow /?s=*
Disallow /?properties*
Disallow /go/
Disallow /wp-admin/
Disallow /wp-content/plugins/
Disallow /wp-includes/
Disallow /*deepspace
Disallow /?deepspace
Allow /wp-admin/admin-ajax.php

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 3

yahoo

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

baiduspider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

yandex

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

bleriot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

applebot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

semrushbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

semrushbot-sa

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

dotbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

ahrefsbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

sistrix

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

dataforseobot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

firstra

Rule Path
Disallow /

wow64

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

trendsmapresolver

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

alittle client

Rule Path
Disallow /

fast

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

mail.ru_bot

Rule Path Comment
Disallow / blocks access to the entire site

wesee

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

geedobot

Rule Path
Disallow /
Back to top

Comments

  • AI BOTS
  • CRAWL LIMITED BOTS
  • Baiduspider
  • Yandex
  • BLOCKED BOTS
  • Copied from wikipedia robots.txt
  • Crawlers that are kind enough to obey, but which we'd rather not have
  • unless they're feeding search engines.
  • Some bots are known to be trouble, particularly those designed to copy
  • entire sites. Please obey robots.txt.
  • Misbehaving: requests much too fast:
  • The 'grub' distributed client has been *very* poorly behaved.
  • Doesn't follow robots.txt anyway, but...
  • Hits many times per second, not acceptable
  • http://www.nameprotect.com/botinfo.html
  • A capture bot, downloads gazillions of pages with no public benefit
  • http://www.webreaper.net/
  • Bots added after reviewing BBQ Pro blocks
Back to top