thesimonandgarfunkelstory.com
robots.txt

Robots Exclusion Standard data for thesimonandgarfunkelstory.com

Resource Scan

Scan Details

Site Domain thesimonandgarfunkelstory.com
Base Domain thesimonandgarfunkelstory.com
Scan Status Ok
Last Scan2025-05-10T11:40:44+00:00
Next Scan 2025-06-09T11:40:44+00:00

Last Scan

Scanned2025-05-10T11:40:44+00:00
URL https://thesimonandgarfunkelstory.com/robots.txt
Domain IPs 109.109.227.178
Response IP 109.109.227.178
Found Yes
Hash e772a0758ae440a8733dd6398ce8cd2373961684cbbe3222054db06da13e1f4e
SimHash 25559c82d410

Groups

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

googlebot-mobile

Rule Path
Disallow

mediapartners-google

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

msnbot

Rule Path
Disallow

slurp

Rule Path
Disallow

teoma

Rule Path
Disallow

gigabot

Rule Path
Disallow

nutch

Rule Path
Disallow

ia_archiver

Rule Path
Disallow

yahoo-mmcrawler

Rule Path
Disallow

asterias

Rule Path
Disallow

yahoo-blogs/v3.9

Rule Path
Disallow

scrubby

Rule Path
Disallow

robozilla

Rule Path
Disallow

mozilla/2.0

Rule Path
Disallow

askjeeves

Rule Path
Disallow

jeeves

Rule Path
Disallow

fast-webcrawler

Rule Path
Disallow

architextspider

Rule Path
Disallow

lycos

Rule Path
Disallow

uk searcher spider

Rule Path
Disallow

inktomi slurp

Rule Path
Disallow

inktomi

Rule Path
Disallow

*

Rule Path
Disallow

Other Records

Field Value
sitemap http://www.thesimonandgarfunkelstory.com/sitemap.xml.gz