cap.org.uk
robots.txt

Robots Exclusion Standard data for cap.org.uk

Resource Scan

Scan Details

Site Domain cap.org.uk
Base Domain cap.org.uk
Scan Status Ok
Last Scan2024-06-21T13:26:44+00:00
Next Scan 2024-07-05T13:26:44+00:00

Last Scan

Scanned2024-06-21T13:26:44+00:00
URL https://cap.org.uk/robots.txt
Domain IPs 104.21.44.169, 172.67.201.151, 2606:4700:3032::ac43:c997, 2606:4700:3036::6815:2ca9
Response IP 172.67.201.151
Found Yes
Hash 8b1c80562411f3cd3e5c4ba853d51e902e9e5221f9e6d3e352cf5dba9cc38c65
SimHash 2b0a9a43c5b0

Groups

*

Rule Path
Disallow */type/capcode/code_rule/*
Disallow */type/bcapcode/code_rule/*
Disallow */account/*

slurp

Rule Path
Allow /

Other Records

Field Value
crawl-delay 120

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

Other Records

Field Value
sitemap https://www.asa.org.uk/sitemap.xml

Comments

  • robots.txt for https://www.asa.org.uk/