Log In Sign Up

lumendatabase.org
robots.txt

Robots Exclusion Standard data for lumendatabase.org

Resource Scan

Scan Details

Site Domain lumendatabase.org
Base Domain lumendatabase.org
Scan Status Ok
Last Scan2025-10-02T00:14:26+00:00
Next Scan 2025-11-01T00:14:26+00:00

Last Scan

Scanned2025-10-02T00:14:26+00:00
URL https://lumendatabase.org/robots.txt
Domain IPs 128.103.64.94
Response IP 128.103.64.94
Found Yes
Hash 4d2220634741e1970cfd9d3284dd0a83ccfa10d4a75c7ab85f231c2ca44b47f9
SimHash a01409f553e0

Groups

google-legal-removals

Rule Path
Allow /

googlebot

Rule Path
Allow /$
Allow /pages
Disallow /notices
Disallow /faceted_search
Disallow /captcha_gateway

ia_archiver

Rule Path
Allow /
Disallow /faceted_search
Disallow /captcha_gateway

*

Rule Path
Disallow /
Disallow /notices
Disallow /faceted_search
Disallow /captcha_gateway
Allow /pages
Allow /$

Other Records

Field Value Comment
crawl-delay 86400 one day
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
Back to top