Log In Sign Up

g-site.com
robots.txt

Robots Exclusion Standard data for g-site.com

Resource Scan

Scan Details

Site Domain g-site.com
Base Domain g-site.com
Scan Status Ok
Last Scan2025-05-17T14:19:51+00:00
Next Scan 2025-06-16T14:19:51+00:00

Last Scan

Scanned2025-05-17T14:19:51+00:00
URL https://g-site.com/robots.txt
Domain IPs 74.208.236.145
Response IP 74.208.236.145
Found Yes
Hash 25e4476258369bd6a5a71780b938a54bc901a9bb21bd94dc8bed3ae4750b3fe0
SimHash 9b145942c9c6

Groups

*

Rule Path
Disallow /temp/
Disallow /test/
Disallow /private/
Disallow /cgi-bin/local_sales/
Disallow /cgi-bin/local_services/
Disallow /newyears/
Disallow /random_times/images/
Disallow /cgi-bin/pod/

ia_archiver

Rule Path
Disallow /
Disallow /webdesign/

asterias

Rule Path
Disallow /cgi-bin/pod/pod.cgi?dir=%2F*

semalt

Rule Path
Disallow /
Back to top

Comments

  • All robots will spider the domain except listed dirs
  • Disallow ia_archiver
  • Disallow asterias
Back to top