Log In Sign Up

findjobs.direct
robots.txt

Robots Exclusion Standard data for findjobs.direct

Resource Scan

Scan Details

Site Domain findjobs.direct
Base Domain findjobs.direct
Scan Status Ok
Last Scan2024-05-27T22:01:17+00:00
Next Scan 2024-06-03T22:01:17+00:00

Last Scan

Scanned2024-05-27T22:01:17+00:00
URL https://findjobs.direct/robots.txt
Domain IPs 104.21.45.76, 172.67.211.57, 2606:4700:3033::ac43:d339, 2606:4700:3035::6815:2d4c
Response IP 104.21.45.76
Found Yes
Hash 92b707bcfa8f58adf6a2529f368c981c002a52b2afef1646c7556db7feebb663
SimHash a2848d877770

Groups

*

Rule Path
Allow /
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
Back to top