wyblog.us
robots.txt

Robots Exclusion Standard data for wyblog.us

Resource Scan

Scan Details

Site Domain wyblog.us
Base Domain wyblog.us
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2025-07-08T10:54:34+00:00
Next Scan 2025-10-06T10:54:34+00:00

Last Successful Scan

Scanned2023-08-26T06:52:51+00:00
URL https://wyblog.us/robots.txt
Domain IPs 104.21.45.60, 172.67.210.107, 2606:4700:3031::ac43:d26b, 2606:4700:3034::6815:2d3c
Response IP 172.67.210.107
Found Yes
Hash 66dc5a9e35d27e690d3cb6fde91240ce7ea4777c9cc50e6dbbcf1ebee911d0d0
SimHash 77d446c8cd96

Groups

*

Rule Path
Disallow /blog/2010/
Disallow /blog/2009/
Disallow /blog/2008/
Disallow /blog/2007/
Disallow /base_code/
Disallow /bin/
Disallow /cgi-bin/
Disallow /demo/
Disallow /fork_code/
Disallow /java_code/
Disallow /htbin/
Disallow /proxy_code/
Disallow /pvbin/
Disallow /script_code/
Disallow /serverdoc/
Disallow /sysosf/
Disallow /system/
Disallow /%24mapimage/
Disallow /%24omnimap/
Disallow /%24testcgi/
Disallow /tarserv/
Disallow /phonedir.html

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60.0

Comments

  • shut off robots
  • Yahoo