Log In Sign Up

mr-resistor.co.uk
robots.txt

Robots Exclusion Standard data for mr-resistor.co.uk

Resource Scan

Scan Details

Site Domain mr-resistor.co.uk
Base Domain mr-resistor.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-12-06T00:55:17+00:00
Next Scan 2026-01-05T00:55:17+00:00

Last Successful Scan

Scanned2025-10-13T07:18:15+00:00
URL https://mr-resistor.co.uk/robots.txt
Redirect https://www.mr-resistor.co.uk/robots.txt
Redirect Domain www.mr-resistor.co.uk
Redirect Base mr-resistor.co.uk
Domain IPs 51.104.28.65
Redirect IPs 13.107.226.59, 13.107.253.59, 2620:1ec:46::59, 2620:1ec:bdf::59
Response IP 13.107.226.59
Found Yes
Hash ad3dbb35e525be9df900f0d1a0067cc975753b71ff4f9aa2a5800e32a8f62bc8
SimHash c816d6722432

Groups

baiduspider

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

uolbot

Rule Path
Disallow /

radarurl

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Allow /
Disallow /products?o=
Disallow /Products?o=
Disallow /products*%26o%3D
Disallow /Products*%26o%3D
Back to top

Other Records

Field Value
sitemap https://www.mr-resistor.co.uk/sitemap.xml
Back to top

Comments

  • Block Baidu (China's biggest search engine)
  • Block Sogou
  • Block 360 Search
  • Block Shenma (Alibaba)
  • Block SOSO (Tencent)
  • Block other lesser-known Chinese crawlers
  • Block Yandex's bot (Russia)
  • Block UOLBot (Universo Online - Brazil)
  • Block RadarURL crawler (Brazil)
  • Rate-limit Ahrefs SEO crawler (Ahrefs - Singapore / Global)
  • Rate-limit Semrush crawler (SEMrush - United States / Global)
  • Rate-limit Moz crawler (DotBot - United States)
Back to top

Warnings

  • 2 invalid lines.
Back to top