hamstercentral.com
robots.txt

Robots Exclusion Standard data for hamstercentral.com

Resource Scan

Scan Details

Site Domain hamstercentral.com
Base Domain hamstercentral.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-11-11T23:24:47+00:00
Next Scan 2026-02-09T23:24:47+00:00

Last Successful Scan

Scanned2024-04-21T05:50:44+00:00
URL https://hamstercentral.com/robots.txt
Domain IPs 104.21.35.10, 172.67.167.42, 2606:4700:3031::ac43:a72a, 2606:4700:3035::6815:230a
Response IP 172.67.167.42
Found Yes
Hash b46f63f115b249e7b39b02d3dd55153e0875cc7db69b4e3faf535570f65234c4
SimHash 6d149f37cc77

Groups

ia_archiver

Rule Path
Allow /

googlebot-image

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

*

Rule Path
Disallow /community/cgi-bin/
Disallow /community/clientscript/
Disallow /community/cpstyle/
Disallow /community/customavatars/
Disallow /community/customprofilepics/
Disallow /community/editpost.php
Disallow /community/gamercard.php
Disallow /community/images/
Disallow /community/includes/
Disallow /community/inlinemod.php
Disallow /community/install/
Disallow /community/member.php
Disallow /community/memberlist.php
Disallow /community/newreply.php
Disallow /community/newthread.php
Disallow /community/payments.php
Disallow /community/printthread.php
Disallow /community/subscription.php
Disallow /community/payments.php
Disallow /community/profile.php
Disallow /community/faq.php
Disallow /community/calendar.php
Disallow /community/private.php
Disallow /community/report.php
Disallow /community/sendmessage.php
Disallow /community/search.php
Disallow /community/showgroups.php
Disallow /community/showpost.php
Disallow /community/reputation.php
Disallow /community/report.php
Disallow /community/register.php
Disallow /community/login.php
Disallow /community/faq.php
Disallow /community/image.php
Disallow /community/cron.php
Disallow /community/joinrequests.php
Disallow /community/usercp.php
Disallow /community/usernote.php
Disallow /gallery/adm-index.php
Disallow /gallery/uploadphoto.php
Disallow /gallery/slideshow.php
Disallow /gallery/comments.php

mediapartners-google

Rule Path
Allow /community/member.php
Allow /community/private.php
Allow /community/usercp.php

Other Records

Field Value
sitemap http://www.hamstercentral.com/community/sitemap_index.xml.gz

Comments

  • Allow Archive.org to save snapshots of everything
  • Tame yahoo... it tends to eat a ton of resources without a delay
  • list individual pages and files here that all bots should ignore, as well as group extentions.
  • If you re-write everything to .html, you can disallow *.php, but note that if you don't have a CRR for custom pages, those will be blocked.
  • Finally, list the path to your sitemap: