robrobinette.com
robots.txt

Robots Exclusion Standard data for robrobinette.com

Resource Scan

Scan Details

Site Domain robrobinette.com
Base Domain robrobinette.com
Scan Status Ok
Last Scan2024-10-05T21:06:17+00:00
Next Scan 2024-10-19T21:06:17+00:00

Last Scan

Scanned2024-10-05T21:06:17+00:00
URL https://robrobinette.com/robots.txt
Domain IPs 73.108.29.116
Response IP 73.108.29.116
Found Yes
Hash 0afce8c6428f67df21e927484680bbed4d39424be7a3b113e203184705568639
SimHash e14f99c554f3

Groups

*

Rule Path
Disallow
Disallow /cgi-bin/

Other Records

Field Value
sitemap https://robrobinette.com/sitemap.htm

Comments

  • /robots.txt file for http://webcrawler.com/
  • mail webmaster@webcrawler.com for constructive criticism
  • User-agent: webcrawler
  • Disallow:
  • User-agent: lycra
  • Disallow: /
  • User-agent: *
  • Disallow: /tmp
  • Disallow: /logs
  • robots.txt generated at http://www.mcanerin.com