journalist.id
robots.txt

Robots Exclusion Standard data for journalist.id

Resource Scan

Scan Details

Site Domain journalist.id
Base Domain journalist.id
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-10-04T20:08:16+00:00
Next Scan 2024-10-11T20:08:16+00:00

Last Successful Scan

Scanned2024-09-26T19:46:57+00:00
URL https://journalist.id/robots.txt
Domain IPs 103.217.144.250
Response IP 103.217.144.250
Found Yes
Hash 2ae93ce485a1a90b48f6b405985fa1ba9651e0f10594d0633420380a3504b57c
SimHash 0d3c5c74c384

Groups

serpstatbot

Rule Path
Disallow /

googlebot

Rule Path
Disallow /api/
Disallow /pdf-post/*

Other Records

Field Value
crawl-delay 1800

Other Records

Field Value
sitemap https://journalist.id/sitemap.xml