oxfordowl.co.uk
robots.txt

Robots Exclusion Standard data for oxfordowl.co.uk

Resource Scan

Scan Details

Site Domain oxfordowl.co.uk
Base Domain oxfordowl.co.uk
Scan Status Ok
Last Scan2024-10-28T06:05:09+00:00
Next Scan 2024-11-27T06:05:09+00:00

Last Scan

Scanned2024-10-28T06:05:09+00:00
URL https://oxfordowl.co.uk/robots.txt
Redirect https://www.oxfordowl.co.uk/robots.txt
Redirect Domain www.oxfordowl.co.uk
Redirect Base oxfordowl.co.uk
Domain IPs 3.248.171.19, 34.248.180.208, 54.228.36.24
Redirect IPs 3.248.171.19, 34.248.180.208, 54.228.36.24
Response IP 54.228.36.24
Found Yes
Hash d194ddb2ac8d0495d3c609ee372c9cb55ddb0120f2466163de8fc4238d4635a2
SimHash 8014094bc7e4

Groups

*

Rule Path
Disallow

*

Rule Path
Disallow *ELS_handbook.pdf

magpie-crawler

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • Allow all bots to access and index the site