anengineersblog.org
robots.txt

Robots Exclusion Standard data for anengineersblog.org

Resource Scan

Scan Details

Site Domain anengineersblog.org
Base Domain anengineersblog.org
Scan Status Ok
Last Scan2025-12-03T22:17:00+00:00
Next Scan 2026-01-02T22:17:00+00:00

Last Scan

Scanned2025-12-03T22:17:00+00:00
URL http://anengineersblog.org/robots.txt
Redirect http://www.anengineersblog.org/robots.txt
Redirect Domain www.anengineersblog.org
Redirect Base anengineersblog.org
Domain IPs 199.34.228.71
Redirect IPs 199.34.228.71
Response IP 199.34.228.71
Found Yes
Hash 464f51338333815d8662ff97c7b082b4274a55aaa8a62ada77fcb6411b1144bf
SimHash a854d8762e93

Groups

nerdybot

Rule Path
Disallow /

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /ajax/
Disallow /apps/
Disallow /projects.html
Disallow /teardowns.html

Other Records

Field Value
sitemap http://www.anengineersblog.org/sitemap.xml