ndrfc.co.uk
robots.txt

Robots Exclusion Standard data for ndrfc.co.uk

Resource Scan

Scan Details

Site Domain ndrfc.co.uk
Base Domain ndrfc.co.uk
Scan Status Ok
Last Scan2024-05-21T07:03:36+00:00
Next Scan 2024-06-20T07:03:36+00:00

Last Scan

Scanned2024-05-21T07:03:36+00:00
URL http://ndrfc.co.uk/robots.txt
Redirect https://northdorsetrfc.rfu.club//robots.txt
Redirect Domain northdorsetrfc.rfu.club
Redirect Base rfu.club
Domain IPs 94.136.40.82
Redirect IPs 130.0.83.93
Response IP 130.0.83.93
Found Yes
Hash a9d32efd543aa24fc74b3615e881b0dc2ed5fb0ff06a07b33196df062364bb36
SimHash 000043414336

Groups

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

*

Rule Path
Allow /
Disallow /teams/
Disallow /competitions/

Comments

  • robots.txt for northdorsetrfc.rfu.club