Log In Sign Up

501st.com
robots.txt

Robots Exclusion Standard data for 501st.com

Resource Scan

Scan Details

Site Domain 501st.com
Base Domain 501st.com
Scan Status Ok
Last Scan2025-12-26T20:44:58+00:00
Next Scan 2026-01-25T20:44:58+00:00

Last Scan

Scanned2025-12-26T20:44:58+00:00
URL https://501st.com/robots.txt
Domain IPs 52.72.11.249
Response IP 52.72.11.249
Found Yes
Hash cdc4fae7a7ca2d94c7ff042a5183f2a1063d110cd7ca226dbd7a97cf743bc7bb
SimHash 85d7d0d055b5

Groups

facebot

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

googlebot

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

googlebot-mobile

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

mediapartners-google

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

Other Records

Field Value
crawl-delay 20

adsbot-google

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

Other Records

Field Value
crawl-delay 20

archive.org_bot

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

bingbot

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

ia_archiver

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

Other Records

Field Value
crawl-delay 20

msnbot

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

Other Records

Field Value
crawl-delay 20

scoutjet

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

Other Records

Field Value
crawl-delay 20

yandex

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

Other Records

Field Value
crawl-delay 20

seznambot

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

Other Records

Field Value
crawl-delay 20

slurp

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

Other Records

Field Value
crawl-delay 20

twitterbot

Rule Path
Allow /
Disallow /mw501
Disallow /forum
Disallow /archived_things
Disallow /logoarchive
Disallow /memberdata

Other Records

Field Value
crawl-delay 20

*

Rule Path
Disallow /
Back to top

Comments

  • _________________ ____ __________
  • . . / | / \ . | \
  • . / ______ _____| . / \ | ___ | . .
  • \ \ | | / /\ \ | |___> |
  • . \ \ | | / /__\ \ . | _/ .
  • . ________> | | | . / \ | |\ \_______ .
  • | / | | / ______ \ | | \ |
  • |___________/ |___| /____/ \____\ |___| \__________| .
  • . ____ __ . _____ ____ . __________ . _________
  • \ \ / \ / / / \ | \ / | .
  • \ \/ \/ / / \ | ___ | / ______| .
  • \ / / /\ \ . | |___> | \ \
  • . \ / / /__\ \ | _/. \ \
  • \ /\ / / \ | |\ \______> | .
  • \ / \ / / ______ \ | | \ / .
  • . . \/ \/ /____/ \____\ |___| \____________/ LS
Back to top

Warnings

  • 1 invalid line.
Back to top