Log In Sign Up

iks.lt
robots.txt

Robots Exclusion Standard data for iks.lt

Resource Scan

Scan Details

Site Domain iks.lt
Base Domain iks.lt
Scan Status Ok
Last Scan2024-10-29T10:04:27+00:00
Next Scan 2024-11-28T10:04:27+00:00

Last Scan

Scanned2024-10-29T10:04:27+00:00
URL https://iks.lt/robots.txt
Domain IPs 37.156.219.92
Response IP 37.156.219.92
Found Yes
Hash db28a33d9c517cfbcde7e52842ed1bd634ed6c6b0cfd64ace7d2f2a39bba01bf
SimHash 3f4081630b51

Groups

*

Rule Path
Disallow /manual/
Disallow /manual-1.3/
Disallow /manual-2.0/
Disallow /manual-2.2/
Disallow /addon-modules/
Disallow /doc/
Disallow /images/
Disallow /all_our_e-mail_addresses
Disallow /admin/

stress-agent

Rule Path
Disallow /
Back to top

Comments

  • $Id: robots.txt 92365 2007-09-23 14:04:27Z oden $
  • $HeadURL: svn+ssh://svn.mandriva.com/svn/packages/cooker/apache-conf/current/SOURCES/robots.txt $
  • exclude help system from robots
  • the next line is a spam bot trap, for grepping the logs. you should _really_ change this to something else...
  • same idea here...
  • but allow htdig to index our doc-tree
  • User-agent: htdig
  • Disallow:
  • disallow stress test
Back to top