Log In Sign Up

hamstercentral.com
robots.txt

Robots Exclusion Standard data for hamstercentral.com

Resource Scan

Scan Details

Site Domain hamstercentral.com
Base Domain hamstercentral.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-11-11T23:24:47+00:00
Next Scan 2026-02-09T23:24:47+00:00

Last Successful Scan

Scanned2024-04-21T05:50:44+00:00
URL https://hamstercentral.com/robots.txt
Domain IPs 104.21.35.10, 172.67.167.42, 2606:4700:3031::ac43:a72a, 2606:4700:3035::6815:230a
Response IP 172.67.167.42
Found Yes
Hash b46f63f115b249e7b39b02d3dd55153e0875cc7db69b4e3faf535570f65234c4
SimHash 6d149f37cc77

Groups

ia_archiver

Rule Path
Allow /

googlebot-image

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

*

Rule Path
Disallow /community/cgi-bin/
Disallow /community/clientscript/
Disallow /community/cpstyle/
Disallow /community/customavatars/
Disallow /community/customprofilepics/
Disallow /community/editpost.php
Disallow /community/gamercard.php
Disallow /community/images/
Disallow /community/includes/
Disallow /community/inlinemod.php
Disallow /community/install/
Disallow /community/member.php
Disallow /community/memberlist.php
Disallow /community/newreply.php
Disallow /community/newthread.php
Disallow /community/payments.php
Disallow /community/printthread.php
Disallow /community/subscription.php
Disallow /community/payments.php
Disallow /community/profile.php
Disallow /community/faq.php
Disallow /community/calendar.php
Disallow /community/private.php
Disallow /community/report.php
Disallow /community/sendmessage.php
Disallow /community/search.php
Disallow /community/showgroups.php
Disallow /community/showpost.php
Disallow /community/reputation.php
Disallow /community/report.php
Disallow /community/register.php
Disallow /community/login.php
Disallow /community/faq.php
Disallow /community/image.php
Disallow /community/cron.php
Disallow /community/joinrequests.php
Disallow /community/usercp.php
Disallow /community/usernote.php
Disallow /gallery/adm-index.php
Disallow /gallery/uploadphoto.php
Disallow /gallery/slideshow.php
Disallow /gallery/comments.php

mediapartners-google

Rule Path
Allow /community/member.php
Allow /community/private.php
Allow /community/usercp.php
Back to top

Other Records

Field Value
sitemap http://www.hamstercentral.com/community/sitemap_index.xml.gz
Back to top

Comments

  • Allow Archive.org to save snapshots of everything
  • Tame yahoo... it tends to eat a ton of resources without a delay
  • list individual pages and files here that all bots should ignore, as well as group extentions.
  • If you re-write everything to .html, you can disallow *.php, but note that if you don't have a CRR for custom pages, those will be blocked.
  • Finally, list the path to your sitemap:
Back to top