Log In Sign Up

hancocks.co.uk
robots.txt

Robots Exclusion Standard data for hancocks.co.uk

Resource Scan

Scan Details

Site Domain hancocks.co.uk
Base Domain hancocks.co.uk
Scan Status Ok
Last Scan2024-06-23T17:25:44+00:00
Next Scan 2024-07-23T17:25:44+00:00

Last Scan

Scanned2024-06-23T17:25:44+00:00
URL https://hancocks.co.uk/robots.txt
Redirect https://www.hancocks.co.uk/robots.txt
Redirect Domain www.hancocks.co.uk
Redirect Base hancocks.co.uk
Domain IPs 104.26.2.204, 104.26.3.204, 172.67.72.160, 2606:4700:20::681a:2cc, 2606:4700:20::681a:3cc, 2606:4700:20::ac43:48a0
Redirect IPs 104.26.2.204, 104.26.3.204, 172.67.72.160, 2606:4700:20::681a:2cc, 2606:4700:20::681a:3cc, 2606:4700:20::ac43:48a0
Response IP 104.26.3.204
Found Yes
Hash ac623e7eaf7de71d6632b2eb0e2318af21bc251d487e945edbbccca3e429a37a
SimHash 8ff0bb762562

Groups

googlebot

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

yandexbot

Rule Path
Disallow

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /
Disallow /README.md
Disallow /ecosystem.config.js
Disallow /i18n.js
Disallow /jsconfig.json
Disallow /newrelic.js
Disallow /next.config.js
Disallow /package.json
Disallow /.env
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?limit=
Disallow /*?limit=is_scroll
Disallow /*?limit=
Disallow /*?mode*
Disallow /*?SID=
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /.config/
Disallow /.next
Disallow /.npm
Disallow /.pm2
Disallow /locales
Disallow /node_modules
Disallow /src
Back to top

Other Records

Field Value
sitemap https://a333916.sitemaphosting7.com/4403589/sitemap_4403589.xml
Back to top

Comments

  • GENERAL SETTINGS
  • Enable robots.txt rules for all crawlers
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl common files
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
Back to top