Log In Sign Up

hustleminds.pro
robots.txt

Robots Exclusion Standard data for hustleminds.pro

Resource Scan

Scan Details

Site Domain hustleminds.pro
Base Domain hustleminds.pro
Scan Status Ok
Last Scan2024-05-23T19:45:24+00:00
Next Scan 2024-06-22T19:45:24+00:00

Last Scan

Scanned2024-05-23T19:45:24+00:00
URL https://hustleminds.pro/robots.txt
Redirect https://www.hustleminds.pro/robots.txt
Redirect Domain www.hustleminds.pro
Redirect Base hustleminds.pro
Domain IPs 204.246.191.111, 204.246.191.118, 204.246.191.81, 204.246.191.83, 2600:9000:21f8:0:5:a69e:1bc0:93a1, 2600:9000:21f8:1400:5:a69e:1bc0:93a1, 2600:9000:21f8:2000:5:a69e:1bc0:93a1, 2600:9000:21f8:2600:5:a69e:1bc0:93a1, 2600:9000:21f8:6a00:5:a69e:1bc0:93a1, 2600:9000:21f8:b600:5:a69e:1bc0:93a1, 2600:9000:21f8:ca00:5:a69e:1bc0:93a1, 2600:9000:21f8:d000:5:a69e:1bc0:93a1
Redirect IPs 18.238.109.112, 18.238.109.119, 18.238.109.2, 18.238.109.25, 2600:9000:200f:2200:5:a69e:1bc0:93a1, 2600:9000:200f:3200:5:a69e:1bc0:93a1, 2600:9000:200f:3e00:5:a69e:1bc0:93a1, 2600:9000:200f:5200:5:a69e:1bc0:93a1, 2600:9000:200f:7c00:5:a69e:1bc0:93a1, 2600:9000:200f:9c00:5:a69e:1bc0:93a1, 2600:9000:200f:b600:5:a69e:1bc0:93a1, 2600:9000:200f:da00:5:a69e:1bc0:93a1
Response IP 18.165.171.54
Found Yes
Hash 6820bc08549aaba6b2c0aa4cd97b09c1e794b781c0e712253f827b4fa3bc3020
SimHash 383049030c98

Groups

*

Rule Path
Disallow /startTopic/
Disallow /discover/unread/
Disallow /markallread/
Disallow /staff/
Disallow /cookie/
Disallow /online/
Disallow /discover/
Disallow /leaderboard/
Disallow /search/
Disallow /tags/
Disallow /*?advancedSearchForm=
Disallow /register/
Disallow /lostpassword/
Disallow /login/
Disallow /*?sortby=
Disallow /*?filter=
Disallow /*?tab=
Disallow /*?do=
Disallow /*ref%3D
Disallow /*?forumId*
Disallow /profile/
Back to top

Other Records

Field Value
sitemap https://www.hustleminds.pro/sitemap.php
Back to top

Comments

  • Rules for Invision Community (https://invisioncommunity.com)
  • Block pages with no unique content
  • Block faceted pages and 301 redirect pages
  • Block profile pages as these have little unique value, consume a lot of crawl time and contain hundreds of 301 links
  • Sitemap URL
Back to top