berksbirds.co.uk
robots.txt

Robots Exclusion Standard data for berksbirds.co.uk

Resource Scan

Scan Details

Site Domain berksbirds.co.uk
Base Domain berksbirds.co.uk
Scan Status Ok
Last Scan2024-09-28T20:05:41+00:00
Next Scan 2024-10-05T20:05:41+00:00

Last Scan

Scanned2024-09-28T20:05:41+00:00
URL https://berksbirds.co.uk/robots.txt
Domain IPs 92.205.227.171
Response IP 92.205.227.171
Found Yes
Hash a6ffb9cfd007780ebdb8766faa6f56b2bb2e3c8dbaa02206f6da2eb68b70acad
SimHash 6017d4b88e00

Groups

*

Rule Path
Disallow /webadmin/
Disallow /cache/
Disallow /500.asp
Disallow /asp2netbridge.asp
Disallow /brcreport.asp
Disallow /brcreport2.asp
Disallow /error.aspx
Disallow /getnews.asp
Disallow /keepalive.asp
Disallow /message.asp
Disallow /offline.html
Disallow /peregrine.asp
Disallow /sendnotification.aspx
Disallow /sendtweet.aspx
Disallow /wip.aspx

ahrefsbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

blexbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

fidget-spinner-bot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

obot

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

test-bot

Rule Path
Disallow /

thesis-research-bot

Rule Path
Disallow /