Log In Sign Up

toothycat.net
robots.txt

Robots Exclusion Standard data for toothycat.net

Resource Scan

Scan Details

Site Domain toothycat.net
Base Domain toothycat.net
Scan Status Ok
Last Scan2024-09-23T18:28:49+00:00
Next Scan 2024-10-23T18:28:49+00:00

Last Scan

Scanned2024-09-23T18:28:49+00:00
URL https://toothycat.net/robots.txt
Redirect https://www.toothycat.net/robots.txt
Redirect Domain www.toothycat.net
Redirect Base toothycat.net
Domain IPs 89.16.173.239
Redirect IPs 89.16.173.239
Response IP 89.16.173.239
Found Yes
Hash 16aa9987e9b32cc30bf427f5e14d84d75d4c3f1c10bd28db89a7baa3acd57f0b
SimHash 22048d51c544

Groups

*

Rule Path
Disallow /wiki/wiki.pl?SandBox
Disallow /wiki/wiki.pl?id
Disallow /wiki/wiki.pl?action
Disallow /wiki/wiki.pl?search
Disallow /wiki/wiki.pl?&
Disallow /wiki/wiki2.pl
Disallow /wiki/wikit.pl
Disallow /wiki/slang_wrapper.pl
Disallow /wiki/img.pl
Disallow /wiki/util.pl
Disallow /wiki/img2.pl
Disallow /wiki/dice.pl
Disallow /wiki/trap.pl?trap
Disallow /wiki/trap.pl/trap
Disallow /wiki/wiki.pl/wiki.pl
Disallow /cgi

fast

Rule Path
Disallow /

webaroobot

Rule Path
Disallow /

rufusbot

Rule Path
Disallow /

lsearch

Rule Path
Disallow /
Back to top

Comments

  • robots.txt file for toothycat.net
  • email comments to moonshadow@toothycat.net
  • allow everything to be indexed except wiki actions (don't want spurious login IDs created every time Google crawls the site) and image server pages (what would be the point?)
  • ..these don't seem to respect any of the other entries!
Back to top