Log In Sign Up

sverve.com
robots.txt

Robots Exclusion Standard data for sverve.com

Resource Scan

Scan Details

Site Domain sverve.com
Base Domain sverve.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a server error.
Last Scan2025-09-17T15:13:35+00:00
Next Scan 2025-10-17T15:13:35+00:00

Last Successful Scan

Scanned2025-07-27T14:31:37+00:00
URL https://sverve.com/robots.txt
Redirect https://www.sverve.com/robots.txt
Redirect Domain www.sverve.com
Redirect Base sverve.com
Domain IPs 104.21.94.131, 172.67.136.79, 2606:4700:3034::6815:5e83, 2606:4700:3034::ac43:884f
Redirect IPs 104.21.94.131, 172.67.136.79, 2606:4700:3034::6815:5e83, 2606:4700:3034::ac43:884f
Response IP 104.21.94.131
Found Yes
Hash fcf0a480123ee5bd58bf9edfbcf3b6e2a70d5ec616b618edfe3e25f381475060
SimHash bb0c47fcde81

Groups

teoma

Rule Path
Disallow /control/
Disallow /report/

*

Rule Path
Disallow /control/
Disallow /report/
Disallow /details/goldenbull2007john/
Disallow /stream/goldenbull2007john/
Disallow /download/goldenbull2007john/
Disallow /14/items/goldenbull2007john/goldenbull2007john_djvu.txt
Back to top

Other Records

Field Value
sitemap http://archive.org/sitemap/sitemap.xml
sitemap http://archive.org/sitemap/sitemap.xml
Back to top

Comments

  • Welcome to the Archive!
  • Please crawl our files.
  • We appreciate if you can crawl responsibly.
  • Stay open!
  • slow down the ask jeeves crawler which was hitting our SE a little too fast
  • via collection pages. --Feb2008 tracey--
Back to top