teachwithouttears.com
robots.txt

Robots Exclusion Standard data for teachwithouttears.com

Resource Scan

Scan Details

Site Domain teachwithouttears.com
Base Domain teachwithouttears.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-09-22T05:13:30+00:00
Next Scan 2025-09-29T05:13:30+00:00

Last Successful Scan

Scanned2025-09-14T04:45:59+00:00
URL https://teachwithouttears.com/robots.txt
Domain IPs 104.21.47.173, 172.67.149.93, 2606:4700:3035::6815:2fad, 2606:4700:3035::ac43:955d
Response IP 104.21.47.173
Found Yes
Hash 8350abb77c9f1c76172f34c1e0a94914c4139870823ded4a5ae79f8582aa330d
SimHash 6408588603e1

Groups

scrapy

Rule Path
Allow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

*

Rule Path
Disallow /cgi-bin
Disallow /trackback
Disallow /*/trackback
Disallow /comment
Disallow /comments
Disallow /wp-login.php
Disallow /xmlrpc.php

*

Rule Path
Disallow /*.php$
Disallow /*.inc$
Disallow /*.cgi$
Disallow /*.xhtml$
Disallow /*.xlsx$
Disallow /*.doc$
Disallow /*.pdf$
Disallow /*.zip$