Log In Sign Up

webdevtutor.net
robots.txt

Robots Exclusion Standard data for webdevtutor.net

Resource Scan

Scan Details

Site Domain webdevtutor.net
Base Domain webdevtutor.net
Scan Status Ok
Last Scan2025-10-26T17:43:18+00:00
Next Scan 2025-11-02T17:43:18+00:00

Last Scan

Scanned2025-10-26T17:43:18+00:00
URL https://webdevtutor.net/robots.txt
Redirect https://www.webdevtutor.net/robots.txt
Redirect Domain www.webdevtutor.net
Redirect Base webdevtutor.net
Domain IPs 104.21.42.226, 172.67.167.37, 2606:4700:3031::6815:2ae2, 2606:4700:3037::ac43:a725
Redirect IPs 104.21.42.226, 172.67.167.37, 2606:4700:3031::6815:2ae2, 2606:4700:3037::ac43:a725
Response IP 104.21.42.226
Found Yes
Hash f6a980f9c16f3791103bed9f80aede87367d53cbe343774c1f6e144100d23a67
SimHash 0a0d091c42d0

Groups

*

Rule Path
Disallow /*.json$
Disallow /*_buildManifest.js$
Disallow /*_middlewareManifest.js$
Disallow /*_ssgManifest.js$
Disallow /*.js$
Back to top

Other Records

Field Value
sitemap https://www.webdevtutor.net/sitemap.xml
Back to top

Comments

  • Next.JS Crawl Budget Performance Updates
  • Block files ending in .json, _buildManifest.js, _middlewareManifest.js, _ssgManifest.js and any other JS files
  • The asterisks allows any file name
  • The dollar sign ensures it only matches the end of an URL and not a oddly formatted url (e.g. /locations.json.html)
Back to top