gbps.org.uk
robots.txt

Robots Exclusion Standard data for gbps.org.uk

Resource Scan

Scan Details

Site Domain gbps.org.uk
Base Domain gbps.org.uk
Scan Status Ok
Last Scan3/10/2025, 2:18:19 PM
Next Scan 4/9/2025, 2:18:19 PM

Last Scan

Scanned3/10/2025, 2:18:19 PM
URL https://gbps.org.uk/robots.txt
Domain IPs 104.21.93.10, 172.67.201.223, 2606:4700:3030::6815:5d0a, 2606:4700:3031::ac43:c9df
Response IP 172.67.201.223
Found Yes
Hash c2dbe84399e0eb9a1d76ab0d16822b1f781e32d405a3d748450a2439c96c75b5
SimHash a31fdc5aca38

Groups

*

Rule Path
Disallow /boards
Disallow /displays

Other Records

Field Value
crawl-delay 2

adsbot*

Rule Path
Disallow /boards/
Disallow /displays/

ahrefsbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.gbps.org.uk/sitemap.php