thelead.com
robots.txt

Robots Exclusion Standard data for thelead.com

Resource Scan

Scan Details

Site Domain thelead.com
Base Domain thelead.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-10-05T18:58:12+00:00
Next Scan 2025-01-03T18:58:12+00:00

Last Successful Scan

Scanned2021-08-15T04:52:48+00:00
URL https://thelead.com/robots.txt
Found Yes
Hash 9bb57b55e5985696ed20eb938e4c0c7a64039a674c8ff40b00ce2ab6a8f84ec4
SimHash e8008808ed13

Groups

*

Rule Path
Disallow /categories/cat-*
Disallow /index.rss

Other Records

Field Value
sitemap http://thelead.com/config/sitemap.xml