Log In Sign Up

thomasmorris.co.uk
robots.txt

Robots Exclusion Standard data for thomasmorris.co.uk

Resource Scan

Scan Details

Site Domain thomasmorris.co.uk
Base Domain thomasmorris.co.uk
Scan Status Ok
Last Scan2026-01-31T02:52:27+00:00
Next Scan 2026-03-02T02:52:27+00:00

Last Scan

Scanned2026-01-31T02:52:27+00:00
URL https://thomasmorris.co.uk/robots.txt
Redirect https://www.thomasmorris.co.uk/robots.txt
Redirect Domain www.thomasmorris.co.uk
Redirect Base thomasmorris.co.uk
Domain IPs 104.21.60.223, 172.67.201.243, 2606:4700:3035::ac43:c9f3, 2606:4700:3036::6815:3cdf
Redirect IPs 104.21.60.223, 172.67.201.243, 2606:4700:3035::ac43:c9f3, 2606:4700:3036::6815:3cdf
Response IP 172.67.201.243
Found Yes
Hash ffa658470067e39f485663b715565ff70f054b7a63a6ff67226d8b34e7e56d27
SimHash 7116bad0ee35

Groups

*

Rule Path
Disallow /my-thomas-morris/
Disallow /home-worth/
Disallow /uploads/property_pdf/
Disallow /uploads/asset_file/
Allow /images/favicon.ico

gptbot

Rule Path
Allow /*.md$

chatgpt-user

Rule Path
Allow /*.md$

oai-searchbot

Rule Path
Allow /*.md$

claudebot

Rule Path
Allow /*.md$

claude-web

Rule Path
Allow /*.md$

anthropic-ai

Rule Path
Allow /*.md$

claude-searchbot

Rule Path
Allow /*.md$

google-extended

Rule Path
Allow /*.md$

perplexitybot

Rule Path
Allow /*.md$

perplexity-user

Rule Path
Allow /*.md$
Disallow /*%21
Back to top

Other Records

Field Value
sitemap https://www.thomasmorris.co.uk/googlesitemap.xml
sitemap https://www.thomasmorris.co.uk/properties/lettings_googlesitemap.xml
sitemap https://www.thomasmorris.co.uk/properties/sales_googlesitemap.xml
sitemap https://www.thomasmorris.co.uk/properties/locations_googlesitemap.xml
Back to top

Comments

  • ----------------------------------------
  • Allow specific AI crawlers to access .md pages
  • ----------------------------------------
  • OpenAI / ChatGPT
  • Anthropic / Claude
  • Google AI
  • Perplexity
Back to top

Warnings

  • 1 invalid line.
Back to top