theedinburghreporter.co.uk
robots.txt

Robots Exclusion Standard data for theedinburghreporter.co.uk

Resource Scan

Scan Details

Site Domain theedinburghreporter.co.uk
Base Domain theedinburghreporter.co.uk
Scan Status Ok
Last Scan2024-11-02T16:21:14+00:00
Next Scan 2024-11-09T16:21:14+00:00

Last Scan

Scanned2024-11-02T16:21:14+00:00
URL https://theedinburghreporter.co.uk/robots.txt
Domain IPs 192.124.249.8
Response IP 192.124.249.8
Found Yes
Hash ef1b792cf9694b0877a5d84025c9df220c6f4bece4e84d75c86b6958a08104c2
SimHash 6b11d413e7b4

Groups

bingbot
msnbot
msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /ads/
Disallow /2011/01/whats-on-this-week-in-edinburgh-17th-january-2011/*
Disallow /2016/05/whats-on-in-edinburgh-this-week-69/amp/*

Comments

  • Group 1
  • Group 2