Log In Sign Up

manchestervacs.co.uk
robots.txt

Robots Exclusion Standard data for manchestervacs.co.uk

Resource Scan

Scan Details

Site Domain manchestervacs.co.uk
Base Domain manchestervacs.co.uk
Scan Status Ok
Last Scan2025-10-17T12:45:53+00:00
Next Scan 2025-11-16T12:45:53+00:00

Last Scan

Scanned2025-10-17T12:45:53+00:00
URL https://manchestervacs.co.uk/robots.txt
Domain IPs 104.21.57.213, 172.67.167.8, 2606:4700:3031::6815:39d5, 2606:4700:3031::ac43:a708
Response IP 172.67.167.8
Found Yes
Hash 8800678b87b5288122bf296a905cd8af7fea70d58eaa85c030e3e249bc3e08fb
SimHash 4b1079d28c63

Groups

*

Rule Path
Disallow /DysonForum/*action%3D
Disallow /DysonForum/*PHPSESSID
Disallow /DysonForum/*%3Bsa%3D
Disallow /DysonForum/*msg%3D
Disallow /DysonForum/*wap
Disallow /DysonForum/*imode
Disallow /DysonForum/*wap2
Disallow /DysonForum/*action%3Drecent
Disallow /DysonForum/*action%3Dunread
Disallow /DysonForum/*prev_next*
Allow /DysonForum/
Allow /DysonForum/*board*%2C*0.html$
Allow /DysonForum/*topic*%2C*0.html$
Allow /DysonForum/*.xml
Allow /*.css
Allow /*.js
Allow /*.png$
Allow /*.jpg$
Allow /*.jpeg$
Allow /*.gif$

baidu

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

teoma

Rule Path
Disallow /

shoemoneytoolsbot

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

ia_archiver

Rule Path
Allow /

archive.org_bot

Rule Path
Allow /

gptbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ccbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

perplexitybot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

anthropic-ai

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

amazonbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

claudebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

chatgpt-user

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

oai-searchbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /*?
Disallow /*%26
Allow /catalog/view/
Allow /system/storage/
Back to top

Other Records

Field Value
sitemap https://manchestervacs.co.uk/sitemap.xml
Back to top

Comments

  • robots.txt for Manchester Vacs Forum
  • Forum rules - block junk parameters, allow boards & topics
  • Junk/abusive bots blocked
  • Extra China scrapers
  • Archive bots - allowed
  • AI crawlers - allowed but slowed
  • Allow essential assets for rendering
  • Unified sitemap index (covers main site, shop, and forum)
Back to top

Warnings

  • 4 invalid lines.
Back to top