adoist.com
robots.txt

Robots Exclusion Standard data for adoist.com

Resource Scan

Scan Details

Site Domain adoist.com
Base Domain adoist.com
Scan Status Ok
Last Scan2024-10-30T22:20:06+00:00
Next Scan 2024-11-06T22:20:06+00:00

Last Scan

Scanned2024-10-30T22:20:06+00:00
URL https://adoist.com/robots.txt
Domain IPs 18.155.68.10, 18.155.68.22, 18.155.68.39, 18.155.68.90
Response IP 18.155.68.90
Found Yes
Hash ae1b19fb5f5b3df86a367c2a37aadfbde88cc856774afbbf358f0dee56491e9a
SimHash 0256d0306a4b

Groups

*

Rule Path
Disallow /en/
Disallow /hu/

yadirectbot

Rule Path
Disallow

voilabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dotbot

Rule Path
Disallow /

foobar

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baidu

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /