scimitarweb.co.uk
robots.txt

Robots Exclusion Standard data for scimitarweb.co.uk

Resource Scan

Scan Details

Site Domain scimitarweb.co.uk
Base Domain scimitarweb.co.uk
Scan Status Ok
Last Scan2025-09-06T23:08:01+00:00
Next Scan 2025-10-06T23:08:01+00:00

Last Scan

Scanned2025-09-06T23:08:01+00:00
URL https://scimitarweb.co.uk/robots.txt
Domain IPs 104.21.4.108, 172.67.132.1, 2606:4700:3030::6815:46c, 2606:4700:3033::ac43:8401
Response IP 172.67.132.1
Found Yes
Hash 849254f1e18b8ad9e2d3f1334f664a68b767133e940d73731f3eb613372ddb7e
SimHash df44d0969646

Groups

magpie-crawler

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou web spider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandex

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /