Log In Sign Up

grufity.com
robots.txt

Robots Exclusion Standard data for grufity.com

Resource Scan

Scan Details

Site Domain grufity.com
Base Domain grufity.com
Scan Status Ok
Last Scan2024-10-01T20:36:11+00:00
Next Scan 2024-10-08T20:36:11+00:00

Last Scan

Scanned2024-10-01T20:36:11+00:00
URL https://grufity.com/robots.txt
Redirect https://www.grufity.com/robots.txt
Redirect Domain www.grufity.com
Redirect Base grufity.com
Domain IPs 76.76.21.21
Redirect IPs 76.76.21.142, 76.76.21.9
Response IP 76.76.21.241
Found Yes
Hash 64342922a26d8c4541ce20e7bfa04f745293c0d2f83f2787875acb74483cc00d
SimHash 0a090b580ad2

Groups

*

Rule Path
Allow /
Disallow /*.json$
Disallow /*_buildManifest.js$
Disallow /*_middlewareManifest.js$
Disallow /*_ssgManifest.js$
Disallow */api/*
Disallow /api
Disallow */undefined/*
Disallow /stock/*/Peers$
Disallow /stock/*/Charts$
Disallow /historical-data-chart-graph/*/*
Disallow /stock/*/Question-Answer$
Disallow /stock/*/Compare$
Back to top

Other Records

Field Value
sitemap https://www.grufity.com/sitemap.xml
Back to top

Comments

  • Next.JS Crawl Budget Performance Updates
  • Block files ending in .json, _buildManifest.js, _middlewareManifest.js, _ssgManifest.js, and any other JS files
  • The asterisks allows any file name
  • The dollar sign ensures it only matches the end of an URL and not a oddly formatted url (e.g. /locations.json.html)
  • Disallow: /*.js$
Back to top