Log In Sign Up

snarkfood.com
robots.txt

Robots Exclusion Standard data for snarkfood.com

Resource Scan

Scan Details

Site Domain snarkfood.com
Base Domain snarkfood.com
Scan Status Ok
Last Scan2024-11-06T16:32:22+00:00
Next Scan 2024-11-13T16:32:22+00:00

Last Scan

Scanned2024-11-06T16:32:22+00:00
URL https://snarkfood.com/robots.txt
Domain IPs 64.91.249.62
Response IP 64.91.249.62
Found Yes
Hash b18b9c0dbbd9cc25d7912d0e701ee49274bf131df81549bd8106f050cd30ceee
SimHash 2af94d134e57

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /*.inc$
Disallow /*.txt$
Disallow /wp-admin/
Disallow /wp-content/upgrade/
Disallow /wp-content/w3tc/
Disallow /ads/
Disallow /go/
Disallow /stats*

googlebot-image

Rule Path
Disallow
Allow /*
Back to top

Other Records

Field Value
sitemap http://snarkfood.com/sitemap.xml
Back to top

Comments

  • disallow files in the following folders
  • disallow all files ending in .php
  • Disallow: /*.js$
  • Disallow: /*.css$
  • disallow all files in /wp- directorys
  • Disallow: /wp-includes/
  • Disallow: /wp-content/plugins/
  • Disallow: /wp-content/themes/
  • Disallow: /wp-content/wptouch-data/
  • disallow all files with ? in url
  • disallow any files that are stats related
  • Sitemap location for auto-discovery
Back to top