Log In Sign Up

codementor.tech
robots.txt

Robots Exclusion Standard data for codementor.tech

Resource Scan

Scan Details

Site Domain codementor.tech
Base Domain codementor.tech
Scan Status Ok
Last Scan2024-11-01T20:45:04+00:00
Next Scan 2024-11-08T20:45:04+00:00

Last Scan

Scanned2024-11-01T20:45:04+00:00
URL https://codementor.tech/robots.txt
Domain IPs 52.220.155.145, 52.74.232.59
Response IP 52.74.232.59
Found Yes
Hash 90934a85de6c8a4dbcbca0dab4da8eb2a522fe3c547ab3eb72fdcd35658bdd5f
SimHash 0c70997a2303

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Disallow /nogooglebot/
Back to top

Comments

  • Allow all user agents.
  • User agent "Googlebot" should not crawl the `http://example.com/nogooglebot/` folder or any of its subdirectories.
Back to top