anglotopia.net
robots.txt

Robots Exclusion Standard data for anglotopia.net

Resource Scan

Scan Details

Site Domain anglotopia.net
Base Domain anglotopia.net
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-08-29T20:57:16+00:00
Next Scan 2024-11-27T20:57:16+00:00

Last Successful Scan

Scanned2024-02-08T20:29:05+00:00
URL https://anglotopia.net/robots.txt
Domain IPs 78.153.200.109
Response IP 78.153.200.109
Found Yes
Hash 425381bce38cb5a5fa7fee76210b422d8406ae01a82ca2ea65079b00cfd47aa9
SimHash 5a105848a2b7

Groups

*

Rule Path
Disallow /wp-login.php
Disallow /wp-admin/

Other Records

Field Value
crawl-delay 600

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /