Log In Sign Up

adani.benchmarkdigital.com
robots.txt

Robots Exclusion Standard data for adani.benchmarkdigital.com

Resource Scan

Scan Details

Site Domain adani.benchmarkdigital.com
Base Domain benchmarkdigital.com
Scan Status Ok
Last Scan2026-01-12T13:55:51+00:00
Next Scan 2026-02-11T13:55:51+00:00

Last Scan

Scanned2026-01-12T13:55:51+00:00
URL https://adani.benchmarkdigital.com/robots.txt
Domain IPs 104.18.12.193, 104.18.13.193, 2606:4700::6812:cc1, 2606:4700::6812:dc1
Response IP 104.18.13.193
Found Yes
Hash 383f7323aacd110066c5e3f1d70a6461803421c34f9bf18cab638d6f9cf8418f
SimHash b825110bced7

Groups

*

Rule Path
Disallow /
Back to top

Comments

  • robots.txt file to restrict spiders from accessing the site. Most spiders will use this file for permission to site index
  • fot the spiders that do not adhere to this standard, a rule has been written in ISAPI to also disallow Spiders that are
  • known in the list.
  • all spiders where name is entered into http user-agent are not allowed to crawl through site.
Back to top