dalstonmillfabrics.co.uk
robots.txt

Robots Exclusion Standard data for dalstonmillfabrics.co.uk

Resource Scan

Scan Details

Site Domain dalstonmillfabrics.co.uk
Base Domain dalstonmillfabrics.co.uk
Scan Status Ok
Last Scan2025-10-27T04:06:57+00:00
Next Scan 2025-11-26T04:06:57+00:00

Last Scan

Scanned2025-10-27T04:06:57+00:00
URL https://dalstonmillfabrics.co.uk/robots.txt
Domain IPs 104.20.24.89, 172.66.162.241, 2606:4700:10::6814:1859, 2606:4700:10::ac42:a2f1
Response IP 104.20.24.89
Found Yes
Hash 7fa1f9cccc07b4392796979685126e7e4333de1fd60dfcb67bafdfeb48990dc3
SimHash 505cd570b630

Groups

amazonbot

Rule Path
Disallow /

rainbot

Rule Path
Disallow /

test-both

Rule Path
Disallow /

fidget-spinner-bot

Rule Path
Disallow /

thesis-research-bot

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

awariobot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

gptbot/1.0

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

facebookexternalhit/1.1

Rule Path
Allow /

barkrowler

Rule Path
Disallow /

turnitin

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

owler

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

chatglm-spider

Rule Path
Disallow /

barkrowler|netestate

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

*

Rule Path
Allow /