Log In Sign Up

redditstatic.s3.amazonaws.com
robots.txt

Robots Exclusion Standard data for redditstatic.s3.amazonaws.com

Resource Scan

Scan Details

Site Domain redditstatic.s3.amazonaws.com
Base Domain redditstatic.s3.amazonaws.com
Scan Status Ok
Last Scan2025-09-27T22:54:07+00:00
Next Scan 2025-10-11T22:54:07+00:00

Last Scan

Scanned2025-09-27T22:54:07+00:00
URL https://redditstatic.s3.amazonaws.com/robots.txt
Domain IPs 16.15.200.161, 16.182.65.217, 16.182.99.121, 3.5.27.73, 52.217.193.169, 52.217.46.100, 52.217.96.84, 54.231.195.49
Response IP 54.231.204.25
Found Yes
Hash 49aab5fa7caad50bfad12dac7a15becfe7b4c01d24157b49b0fb2a48405b1f56
SimHash 708edec0d0b5

Groups

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

bender

Rule Path
Disallow /my_shiny_metal_ass

gort

Rule Path
Disallow /earth

*

Rule Path
Disallow /*.json
Disallow /*.json-compact
Disallow /*.json-html
Disallow /*.mobile
Disallow /*.compact
Disallow /*.xml
Disallow /*.rss
Disallow /*.i
Disallow /*.embed
Disallow /*/comments/*?*sort=
Disallow /r/*/comments/*/*/c*
Disallow /comments/*/*/c*
Disallow /r/*/submit
Disallow /message/compose*
Disallow /api
Disallow /post
Disallow /submit
Disallow /goto
Disallow /*after%3D
Disallow /*before%3D
Disallow /domain/*t%3D
Disallow /login
Disallow /reddits/search
Disallow /search
Disallow /r/*/search
Allow /
Back to top

Comments

  • 80legs
  • MSNBot
Back to top

Warnings

  • 2 invalid lines.
Back to top