Log In Sign Up

esotericarchives.com
robots.txt

Robots Exclusion Standard data for esotericarchives.com

Resource Scan

Scan Details

Site Domain esotericarchives.com
Base Domain esotericarchives.com
Scan Status Ok
Last Scan2025-09-25T19:24:50+00:00
Next Scan 2025-10-25T19:24:50+00:00

Last Scan

Scanned2025-09-25T19:24:50+00:00
URL https://esotericarchives.com/robots.txt
Domain IPs 71.39.101.39
Response IP 71.39.101.39
Found Yes
Hash fd0d201c415d6f4fd372e65d5a2ba9af3df926e0276f11778b727e617a9c8bb3
SimHash 9a17a972c67d

Groups

*

Rule Path
Disallow /style/*
Disallow /private/*
Disallow /ianua/*
Disallow /solomon/ad10862_087r.jpg
Back to top

Comments

  • This is a file retrieved by webwalkers a.k.a. spiders that
  • conform to a defacto standard.
  • See <URL:http://info.webcrawler.com/mak/projects/robots/robots.html>
  • The webmaster for this site is:
  • <name>Joe Peterson</name>
  • <address>joe@esotericarchives.com</address>
  • Format is:
  • User-agent: <name of spider>
  • Disallow: <nothing> | <path>
  • Date By Reason
  • Mar 31, 1997; JHP Added to website with unrestricted access
  • Aug 16, 1998; JHP Moved yasna to subdirectory, disallowed old parts
  • Oct 24, 2001; JHP Disallow /style/*
  • -----------------------------------------------------------------------------
  • The following allows access to all directories for all spiders
Back to top