shahid.news
robots.txt

Robots Exclusion Standard data for shahid.news

Resource Scan

Scan Details

Site Domain shahid.news
Base Domain shahid.news
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-11-11T04:06:36+00:00
Next Scan 2025-02-09T04:06:36+00:00

Last Successful Scan

Scanned2024-04-16T03:34:15+00:00
URL https://shahid.news/robots.txt
Domain IPs 173.255.194.134, 198.58.118.167, 45.33.18.44, 45.33.2.79, 45.33.20.235, 45.33.23.183, 45.33.30.197, 45.56.79.23, 45.79.19.196, 72.14.178.174, 72.14.185.43, 96.126.123.244
Response IP 72.14.178.174
Found Yes
Hash c0afd7f31bd33b9260f152dbab90c0e0142dc074b268448c94e664252e83ff0c
SimHash 4a54d4954550

Groups

googlebot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

mediapartners-google

Rule Path
Disallow

slurp

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

msnbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

bingbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

ia_archiver

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /