Log In Sign Up

peterharrington.co.uk
robots.txt

Robots Exclusion Standard data for peterharrington.co.uk

Resource Scan

Scan Details

Site Domain peterharrington.co.uk
Base Domain peterharrington.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-06T17:38:18+00:00
Next Scan 2024-11-05T17:38:18+00:00

Last Successful Scan

Scanned2024-06-16T17:37:09+00:00
URL https://peterharrington.co.uk/robots.txt
Domain IPs 18.132.144.20, 35.178.47.150
Response IP 35.178.47.150
Found Yes
Hash a05c871529ed1ee15467a2bb2e9a79b53668897a8c00f0daaa05f83e328db4ae
SimHash 0935f1464bb3

Groups

*

Rule Path
Disallow /

googlebot
googlebot-image
mediapartners-google
msnbot
msnbot-media
slurp
yahoo-blogs
mj12bot
yahoo-mmcrawler
nostocrawlerbot
zend_http_client
pingdom.com_bot_version_1.4_(http://www.pingdom.com/)

Rule Path
Disallow /404/
Disallow /app/
Disallow /cgi-bin/
Disallow /downloader/
Disallow /includes/
Disallow /lib/
Disallow /magento/
Disallow /pkginfo/
Disallow /report/
Disallow /stats/
Disallow /var/
Disallow */account/*
Disallow /index.php/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /catalog/product/gallery/
Disallow /checkout/
Disallow /control/
Disallow /customer/
Disallow /customize/
Disallow /newsletter/
Disallow /poll/
Disallow /review/
Disallow /sendfriend/
Disallow /tag/
Disallow /wishlist/
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /STATUS.txt
Allow /blog/wp-content/uploads/

*

Rule Path
Disallow /lib/
Disallow /*.php$
Disallow /pkginfo/
Disallow /report/
Disallow /var/
Disallow /catalog/
Disallow /customer/
Disallow /sendfriend/
Disallow /review/
Disallow /*SID%3D
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /pub/
Disallow /tag/
Disallow /review/
Disallow /blog/wp-content/plugins/
Disallow /blog/wp-admin/
Disallow /blog/readme.html
Disallow /blog/refer/
Allow /blog/wp-content/uploads/
Allow /blog/
Back to top

Other Records

Field Value
sitemap http://www.peterharrington.co.uk/sitemap.xml
Back to top

Comments

  • but allow only important bots
  • SITEMAP
  • PAGES
  • MAGENTO DIRECTORIES & FILES
  • Directories
  • Disallow: /search*
  • Paths (clean URLs)
  • Disallow: /catalogsearch/
  • Disallow: /contacts/
  • Files
  • QUERY STRING BLOCKER
  • Uncomment if the site is a brand new un-cached site.
  • Disallow: /*?*
  • WORDPRESS DIRECTORIES & FILES
  • Uncomment if using Wordpress in subdirectory
  • Disallow: /blog/wp-content/upgrade/
  • Disallow: /blog/wp-admin/
  • Disallow: /blog/wp-includes/
  • SCREAMING FROG
  • User-agent: Screaming Frog SEO Spider
  • Allow: /
  • Disallow: /*.gif$
  • Disallow: /*.jpg$
  • Disallow: /*.png$
  • Disallow: /*.bmp$
  • Disallow: /*.xml$
  • Disallow: /*.css$
  • Disallow: /*.js$root
Back to top