the-bgn.co.uk
robots.txt

Robots Exclusion Standard data for the-bgn.co.uk

Resource Scan

Scan Details

Site Domain the-bgn.co.uk
Base Domain the-bgn.co.uk
Scan Status Ok
Last Scan2024-11-13T22:24:13+00:00
Next Scan 2024-11-20T22:24:13+00:00

Last Scan

Scanned2024-11-13T22:24:13+00:00
URL https://the-bgn.co.uk/robots.txt
Redirect https://www.the-bgn.co.uk/robots.txt
Redirect Domain www.the-bgn.co.uk
Redirect Base the-bgn.co.uk
Domain IPs 216.250.122.254
Redirect IPs 216.250.122.254
Response IP 216.250.122.254
Found Yes
Hash 5ed51efd8d61261ace952f9abc7a2213e6ca74ebbf618f4ea1e7e38a47b3e19b
SimHash 4a0fc8e242b3

Groups

*

Rule Path
Allow /

Other Records

Field Value
crawl-delay 3

Other Records

Field Value
sitemap https://www.the-bgn.co.uk/sitemap.xml
sitemap https://the-bgn.co.uk/sitemap1.xml
sitemap https://the-bgn.co.uk/sitemap2.xml
sitemap https://the-bgn.co.uk/sitemap3.xml
sitemap https://the-bgn.co.uk/sitemap4.xml
sitemap https://the-bgn.co.uk/sitemap5.xml