Log In Sign Up

msrc.co.uk
robots.txt

Robots Exclusion Standard data for msrc.co.uk

Resource Scan

Scan Details

Site Domain msrc.co.uk
Base Domain msrc.co.uk
Scan Status Ok
Last Scan2026-01-01T01:32:55+00:00
Next Scan 2026-01-31T01:32:55+00:00

Last Scan

Scanned2026-01-01T01:32:55+00:00
URL https://msrc.co.uk/robots.txt
Domain IPs 104.21.44.187, 172.67.202.224, 2606:4700:3032::6815:2cbb, 2606:4700:3033::ac43:cae0
Response IP 104.21.44.187
Found Yes
Hash fcf0a480123ee5bd58bf9edfbcf3b6e2a70d5ec616b618edfe3e25f381475060
SimHash bb0c47fcde81

Groups

teoma

Rule Path
Disallow /control/
Disallow /report/

*

Rule Path
Disallow /control/
Disallow /report/
Disallow /details/goldenbull2007john/
Disallow /stream/goldenbull2007john/
Disallow /download/goldenbull2007john/
Disallow /14/items/goldenbull2007john/goldenbull2007john_djvu.txt
Back to top

Other Records

Field Value
sitemap http://archive.org/sitemap/sitemap.xml
sitemap http://archive.org/sitemap/sitemap.xml
Back to top

Comments

  • Welcome to the Archive!
  • Please crawl our files.
  • We appreciate if you can crawl responsibly.
  • Stay open!
  • slow down the ask jeeves crawler which was hitting our SE a little too fast
  • via collection pages. --Feb2008 tracey--
Back to top