k12digest.com
robots.txt

Robots Exclusion Standard data for k12digest.com

Resource Scan

Scan Details

Site Domain k12digest.com
Base Domain k12digest.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-12-14T05:05:36+00:00
Next Scan 2026-03-14T05:05:36+00:00

Last Successful Scan

Scanned2025-04-24T07:40:35+00:00
URL https://k12digest.com/robots.txt
Domain IPs 104.26.8.124, 104.26.9.124, 172.67.72.42, 2606:4700:20::681a:87c, 2606:4700:20::681a:97c, 2606:4700:20::ac43:482a
Response IP 172.67.72.42
Found Yes
Hash f93943ef8476573587795f348d1b9e81065807b1720f5ef3bea0572f3bf8c8a9
SimHash 011596704680

Groups

*

Rule Path
Disallow /cgi-bin
Disallow /wp-admin
Disallow /wp-includes

Other Records

Field Value
crawl-delay 10

bingbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

dotbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

mail.ru

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

mediapartners-google

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

adsbot-google

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

googlebot-mobile

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://www.k12digest.com/sitemap_index.xml