Log In Sign Up

fabflour.co.uk
robots.txt

Robots Exclusion Standard data for fabflour.co.uk

Resource Scan

Scan Details

Site Domain fabflour.co.uk
Base Domain fabflour.co.uk
Scan Status Ok
Last Scan2024-11-11T02:09:41+00:00
Next Scan 2024-12-11T02:09:41+00:00

Last Scan

Scanned2024-11-11T02:09:41+00:00
URL https://fabflour.co.uk/robots.txt
Domain IPs 104.21.35.92, 172.67.216.110, 2606:4700:3030::6815:235c, 2606:4700:3032::ac43:d86e
Response IP 172.67.216.110
Found Yes
Hash 0668f97cac661da55b8418d27e963fc7cd406ec8b3c38a6382daf0d61dfd91da
SimHash 211499224f3e

Groups

*

Rule Path
Disallow /cpresources/
Disallow /vendor/
Disallow /.env
Disallow /cache/

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://fabflour.co.uk/sitemaps-2-sitemap.xml
Back to top

Comments

  • robots.txt for https://fabflour.co.uk/
  • live - don't allow web crawlers to index cpresources/ or vendor/
  • Disallow ChatGPT bot, as there's no benefit to allowing it to index your site
  • Disallow Google Bard and Vertex AI bots, as there's no benefit to allowing it to index your site
  • Disallow Perplexity bot, as there's no benefit to allowing it to index your site
Back to top