Log In Sign Up

milesweb.co.uk
robots.txt

Robots Exclusion Standard data for milesweb.co.uk

Resource Scan

Scan Details

Site Domain milesweb.co.uk
Base Domain milesweb.co.uk
Scan Status Ok
Last Scan2026-02-28T19:56:50+00:00
Next Scan 2026-03-30T19:56:50+00:00

Last Scan

Scanned2026-02-28T19:56:50+00:00
URL https://milesweb.co.uk/robots.txt
Redirect https://www.milesweb.co.uk/robots.txt
Redirect Domain www.milesweb.co.uk
Redirect Base milesweb.co.uk
Domain IPs 104.26.4.158, 104.26.5.158, 172.67.74.72, 2606:4700:20::681a:49e, 2606:4700:20::681a:59e, 2606:4700:20::ac43:4a48
Redirect IPs 104.26.4.158, 104.26.5.158, 172.67.74.72, 2606:4700:20::681a:49e, 2606:4700:20::681a:59e, 2606:4700:20::ac43:4a48
Response IP 172.67.74.72
Found Yes
Hash 6a513553e251257d4263e09be233570d81c536028bb26329dbcd0b1b29fae822
SimHash 551099755555

Groups

googlebot

Rule Path
Allow /*.js*
Allow /*.css*
Disallow /old_website/*
Disallow /demo-pages/*
Disallow /demosite/*
Disallow /temp-demo/*
Disallow /popup/*
Disallow /*?utm=*

gptbot

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

google-extended

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.milesweb.co.uk/sitemap.xml
sitemap https://www.milesweb.co.uk/blog/post-sitemap.xml
sitemap https://www.milesweb.co.uk/hosting-faqs/post-sitemap1.xml
sitemap https://www.milesweb.co.uk/hosting-faqs/post-sitemap2.xml
Back to top

Comments

  • Allow ChatGPT / OpenAI crawlers
  • (Optional) allow Google's AI extension if you want
Back to top