Log In Sign Up

drivethrustuff.com
robots.txt

Robots Exclusion Standard data for drivethrustuff.com

Resource Scan

Scan Details

Site Domain drivethrustuff.com
Base Domain drivethrustuff.com
Scan Status Ok
Last Scan2025-03-13T18:22:32+00:00
Next Scan 2025-04-12T18:22:32+00:00

Last Scan

Scanned2025-03-13T18:22:32+00:00
URL https://drivethrustuff.com/robots.txt
Redirect https://www.drivethrurpg.com/robots.txt
Redirect Domain www.drivethrurpg.com
Redirect Base drivethrurpg.com
Domain IPs 104.21.24.141, 172.67.218.250, 2606:4700:3036::ac43:dafa, 2606:4700:3037::6815:188d
Redirect IPs 18.219.213.122, 3.13.42.214
Response IP 18.219.213.122
Found Yes
Hash 6af9730263fa1a7ad456e9ad6e09c64babda7669c3643fdd050c6c1b5f677a6c
SimHash aa78954feef6

Groups

*

Rule Path
Allow /
Back to top

Comments

  • Our production robots.txt file; this robots.txt is extremely permissive because the damage that
  • can be done to the site by excluding important bots that drive our SEO rankings far outweights
  • the impact of any given misbehaving bots we are currently aware of. Future exclusions may be
  • added as needed.
  • It is for sites that have taken over a top level domain.
Back to top