Log In Sign Up

thechels.info
robots.txt

Robots Exclusion Standard data for thechels.info

Resource Scan

Scan Details

Site Domain thechels.info
Base Domain thechels.info
Scan Status Ok
Last Scan2026-02-03T23:20:24+00:00
Next Scan 2026-03-05T23:20:24+00:00

Last Scan

Scanned2026-02-03T23:20:24+00:00
URL http://thechels.info/robots.txt
Domain IPs 79.170.44.113
Response IP 79.170.44.113
Found Yes
Hash 2172ce2ed06adecb14e791c8a892811a5ad2ed136a52b7b721d36ce0dfba947c
SimHash 18769913c355

Groups

*

Rule Path
Disallow /w/

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2
Back to top

Comments

  • GF Edits - 2010-01-03
  • We want Google and other search engines to index the site, but MediaWiki
  • creates a lot of content that isn't useful and just overloads the web
  • server and creates a lot of useless rubbish in Google's indexes.
  • The following line should prevent that from happening but still
  • allow search engines to index useful content.
  • Be VERY careful with this command, see here for reference:
  • http://www.mediawiki.org/wiki/Robots.txt
Back to top