demodirectory.com
robots.txt

Robots Exclusion Standard data for demodirectory.com

Resource Scan

Scan Details

Site Domain demodirectory.com
Base Domain demodirectory.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-06-10T17:20:34+00:00
Next Scan 2024-06-17T17:20:34+00:00

Last Successful Scan

Scanned2024-05-10T17:20:06+00:00
URL https://demodirectory.com/robots.txt
Domain IPs 104.26.0.28, 104.26.1.28, 172.67.70.221, 2606:4700:20::681a:11c, 2606:4700:20::681a:1c, 2606:4700:20::ac43:46dd
Response IP 104.26.1.28
Found Yes
Hash 59e1d833c81ffdbfc38261f644ee72b75010ad81e783322aa1eaf889e00b4eee
SimHash 4117c046c377

Groups

*

Rule Path
Disallow /custom/domain*
Disallow /custom/log
Disallow /custom/profile
Disallow /custom/tmp
Disallow /layout
Disallow /includes
Disallow /search/map/summary
Allow /custom/domain*/sitemap*
Allow /custom/domain*/tmp*.js
Allow /custom/domain*/tmp*.css
Allow /custom/domain*/theme*
Allow /custom/domain*/image_files/*
Allow /custom/domain*/content_files/*
Allow /

Other Records

Field Value
crawl-delay 1