Log In Sign Up

northdorsetrfc.rfu.club
robots.txt

Robots Exclusion Standard data for northdorsetrfc.rfu.club

Resource Scan

Scan Details

Site Domain northdorsetrfc.rfu.club
Base Domain rfu.club
Scan Status Ok
Last Scan2024-05-21T06:56:04+00:00
Next Scan 2024-06-20T06:56:04+00:00

Last Scan

Scanned2024-05-21T06:56:04+00:00
URL https://northdorsetrfc.rfu.club/robots.txt
Domain IPs 130.0.83.93
Response IP 130.0.83.93
Found Yes
Hash a9d32efd543aa24fc74b3615e881b0dc2ed5fb0ff06a07b33196df062364bb36
SimHash 000043414336

Groups

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

*

Rule Path
Allow /
Disallow /teams/
Disallow /competitions/
Back to top

Comments

  • robots.txt for northdorsetrfc.rfu.club
Back to top