Log In Sign Up

aven.com
robots.txt

Robots Exclusion Standard data for aven.com

Resource Scan

Scan Details

Site Domain aven.com
Base Domain aven.com
Scan Status Ok
Last Scan2024-06-20T11:02:06+00:00
Next Scan 2024-07-20T11:02:06+00:00

Last Scan

Scanned2024-06-20T11:02:06+00:00
URL https://aven.com/robots.txt
Redirect https://www.aven.com/robots.txt
Redirect Domain www.aven.com
Redirect Base aven.com
Domain IPs 13.35.18.128, 13.35.18.70, 13.35.18.83, 13.35.18.92
Redirect IPs 104.18.0.240, 104.18.1.240, 2606:4700::6812:1f0, 2606:4700::6812:f0
Response IP 104.18.1.240
Found Yes
Hash f4832c1a53134118c04d5065f3a36b8dbb74f94ce287d76f99df487d8b50c03d
SimHash 691572728465

Groups

*

Rule Path Comment
Allow /$ This is lit '/' instead of '/*' which is implied if you use just '/'
Allow /favicon.ico -
Allow /favicon.png -
Allow /education$ -
Allow /education/ -
Allow /privacy$ -
Allow /privacy/ -
Allow /app$ -
Allow /app/ -
Allow /reviews$ -
Allow /reviews/ -
Allow /careers$ -
Allow /careers/ -
Allow /disclosures$ -
Allow /disclosures/ -
Allow /about$ -
Allow /about/ -
Allow /support$ -
Allow /support/ -
Disallow / Equiv to '/*', follows for anything not specified above explicitly
Back to top

Comments

  • Test with https://technicalseo.com/tools/robots-txt/
  • Explicitly enable public homepage paths, disable everything else
  • This syntax says 'allow /education explicitly, and /education/* explicitly'
  • '/education' alone would allow urls like '/educationPotato', which we don't want to whitelist
Back to top