Log In Sign Up

developers.reddit.com
robots.txt

Robots Exclusion Standard data for developers.reddit.com

Resource Scan

Scan Details

Site Domain developers.reddit.com
Base Domain reddit.com
Scan Status Ok
Last Scan2025-03-09T00:51:39+00:00
Next Scan 2025-03-23T00:51:39+00:00

Last Scan

Scanned2025-03-09T00:51:39+00:00
URL https://developers.reddit.com/robots.txt
Domain IPs 151.101.1.140, 151.101.129.140, 151.101.193.140, 151.101.65.140, 2a04:4e42:200::396, 2a04:4e42:400::396, 2a04:4e42:600::396, 2a04:4e42::396
Response IP 151.101.193.140
Found Yes
Hash e1d811e01632f98a4de968f263e9d249406116d82bfb099d078e9acefbba666c
SimHash 0452fb08cf92

Groups

voltron

Rule Path
Disallow /

bender

Rule Path
Disallow /my_shiny_metal_ass

gort

Rule Path
Disallow /earth

mj12bot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

*

Rule Path
Allow /$
Allow /apps
Allow /docs
Allow /get-started
Allow /play
Disallow /
Back to top

Comments

  • Inspired by reddit-wide robots.txt for search engines
  • 80legs
  • 80legs' new crawler
  • landing page
Back to top

Warnings

  • 2 invalid lines.
Back to top