Log In Sign Up

blog.appsignal.com
robots.txt

Robots Exclusion Standard data for blog.appsignal.com

Resource Scan

Scan Details

Site Domain blog.appsignal.com
Base Domain appsignal.com
Scan Status Ok
Last Scan2025-08-26T03:15:04+00:00
Next Scan 2025-09-25T03:15:04+00:00

Last Scan

Scanned2025-08-26T03:15:04+00:00
URL https://blog.appsignal.com/robots.txt
Domain IPs 76.76.21.164, 76.76.21.93
Response IP 66.33.60.130
Found Yes
Hash 38ab5911e36b38e8b14b35c73c58a6d8c95e07ac0e33dd821951e643dfd81188
SimHash 0a09091c06d0

Groups

*

Rule Path
Allow /
Disallow /*.json$
Disallow /*_buildManifest.js$
Disallow /*_middlewareManifest.js$
Disallow /*_ssgManifest.js$
Disallow /*.js$
Back to top

Other Records

Field Value
sitemap https://blog.appsignal.com/sitemap.xml
Back to top

Comments

  • Next.JS Crawl Budget Performance Updates
  • Block files ending in .json, _buildManifest.js, _middlewareManifest.js, _ssgManifest.js, and any other JS files
  • The asterisks allows any file name
  • The dollar sign ensures it only matches the end of an URL and not a oddly formatted url (e.g. /locations.json.html)
Back to top