pdcst.to
robots.txt

Robots Exclusion Standard data for pdcst.to

Resource Scan

Scan Details

Site Domain pdcst.to
Base Domain pdcst.to
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-15T03:53:29+00:00
Next Scan 2024-12-14T03:53:29+00:00

Last Successful Scan

Scanned2024-01-27T03:01:58+00:00
URL https://pdcst.to/robots.txt
Domain IPs 104.21.51.44, 172.67.220.184, 2606:4700:3031::ac43:dcb8, 2606:4700:3034::6815:332c
Response IP 104.21.51.44
Found Yes
Hash 6584dc38f904c7f241b5c310cb60537c0d7901a0e71ac8d0a10f5169424ade0e
SimHash 42544f7bf618

Groups

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

megaindex

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

voltron

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

linkdexbot/2.0

Rule Path
Disallow /

getintentcrawler

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

vegebot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 60

vegi bot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 60

adbeat_bot

Rule Path
Disallow /

jooblebot

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /