Log In Sign Up

aero-web.org
robots.txt

Robots Exclusion Standard data for aero-web.org

Resource Scan

Scan Details

Site Domain aero-web.org
Base Domain aero-web.org
Scan Status Ok
Last Scan2025-10-07T07:25:08+00:00
Next Scan 2025-11-06T07:25:08+00:00

Last Scan

Scanned2025-10-07T07:25:08+00:00
URL https://aero-web.org/robots.txt
Domain IPs 162.144.77.12
Response IP 162.144.77.12
Found Yes
Hash fcf0a480123ee5bd58bf9edfbcf3b6e2a70d5ec616b618edfe3e25f381475060
SimHash bb0c47fcde81

Groups

teoma

Rule Path
Disallow /control/
Disallow /report/

*

Rule Path
Disallow /control/
Disallow /report/
Disallow /details/goldenbull2007john/
Disallow /stream/goldenbull2007john/
Disallow /download/goldenbull2007john/
Disallow /14/items/goldenbull2007john/goldenbull2007john_djvu.txt
Back to top

Other Records

Field Value
sitemap http://archive.org/sitemap/sitemap.xml
sitemap http://archive.org/sitemap/sitemap.xml
Back to top

Comments

  • Welcome to the Archive!
  • Please crawl our files.
  • We appreciate if you can crawl responsibly.
  • Stay open!
  • slow down the ask jeeves crawler which was hitting our SE a little too fast
  • via collection pages. --Feb2008 tracey--
Back to top