Log In Sign Up

fchq.io
robots.txt

Robots Exclusion Standard data for fchq.io

Resource Scan

Scan Details

Site Domain fchq.io
Base Domain fchq.io
Scan Status Ok
Last Scan2025-09-27T09:25:18+00:00
Next Scan 2025-10-04T09:25:18+00:00

Last Scan

Scanned2025-09-27T09:25:18+00:00
URL https://fchq.io/robots.txt
Domain IPs 104.26.14.44, 104.26.15.44, 172.67.74.210, 2606:4700:20::681a:e2c, 2606:4700:20::681a:f2c, 2606:4700:20::ac43:4ad2
Response IP 104.26.14.44
Found Yes
Hash 187ec77697c47906fa1186d9cc5ca9828d5acc37a1b39fb250724f9fcab79fd4
SimHash 0a01091846f0

Groups

*

Rule Path
Disallow /*.json$
Disallow /*_buildManifest.js$
Disallow /*_middlewareManifest.js$
Disallow /*_ssgManifest.js$
Disallow /*.js$
Allow /api/og/*
Back to top

Comments

  • Next.JS Crawl Budget Performance Updates
  • Block files ending in .json, _buildManifest.js, _middlewareManifest.js, _ssgManifest.js, and any other JS files
  • The asterisks allows any file name
  • The dollar sign ensures it only matches the end of an URL and not a oddly formatted url (e.g. /locations.json.html)
Back to top