Log In Sign Up

floorwage.org
robots.txt

Robots Exclusion Standard data for floorwage.org

Resource Scan

Scan Details

Site Domain floorwage.org
Base Domain floorwage.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2025-12-21T10:15:10+00:00
Next Scan 2026-03-21T10:15:10+00:00

Last Successful Scan

Scanned2024-02-07T21:55:24+00:00
URL http://floorwage.org/robots.txt
Redirect https://resourcesnew.cleanclothes.org/robots.txt
Redirect Domain resourcesnew.cleanclothes.org
Redirect Base cleanclothes.org
Domain IPs 85.17.140.66
Redirect IPs 85.17.140.66
Response IP 85.17.140.66
Found Yes
Hash a75b90ae8f66ca4dd54872e199a592bd15f66ed8f486d1dfb6b3c7a8d5fc245f
SimHash ca1a495f0bd0

Groups

*

Rule Path
Disallow /filestore

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • Sample robots.txt file - ensures that a Google Appliance can still access the spider page (if configured)
  • and assumes an installation in the site root. For sites in a subfolder you must move the robots.txt file
  • to the site root and alter the paths accordingly.
Back to top