digitalfront.org
robots.txt

Robots Exclusion Standard data for digitalfront.org

Resource Scan

Scan Details

Site Domain digitalfront.org
Base Domain digitalfront.org
Scan Status Ok
Last Scan2025-10-11T03:51:57+00:00
Next Scan 2025-11-10T03:51:57+00:00

Last Scan

Scanned2025-10-11T03:51:57+00:00
URL https://digitalfront.org/robots.txt
Redirect https://www.digitalfront.org/robots.txt
Redirect Domain www.digitalfront.org
Redirect Base digitalfront.org
Domain IPs 104.21.20.127, 172.67.192.229, 2606:4700:3033::6815:147f, 2606:4700:3035::ac43:c0e5
Redirect IPs 104.21.20.127, 172.67.192.229, 2606:4700:3033::6815:147f, 2606:4700:3035::ac43:c0e5
Response IP 104.21.20.127
Found Yes
Hash fdca4bad029606ee857c4aa2c9611095418d91b2979507a003deb192f6721178
SimHash c8dc58800766

Groups

*

Rule Path
Allow /
Allow /wp-content/uploads/*
Allow /wp-content/*.js
Allow /wp-includes/*.js
Disallow /cgi-bin
Disallow /wp-content/themes/
Disallow /wp-includes/
Disallow /*/attachment/
Disallow /*?
Disallow /*/trackback
Allow /feed/$
Disallow /*/*/feed/rss/$
Disallow /*/*/*/feed/$
Disallow /*/*/*/feed/rss/$
Disallow /*/*/*/trackback/$

slurpbot

Rule Path
Disallow /Sitemap

duckduckbot

Rule Path
Disallow /Sitemap

googlebot

Rule Path
Disallow /Sitemap

yandex

Rule Path
Disallow /Sitemap

bingbot

Rule Path
Disallow /Sitemap

msiecrawler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

httrack

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

linko

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.digitalfront.org/sitemapindex.xml