Log In Sign Up

optimisemedia.com
robots.txt

Robots Exclusion Standard data for optimisemedia.com

Resource Scan

Scan Details

Site Domain optimisemedia.com
Base Domain optimisemedia.com
Scan Status Ok
Last Scan2025-12-13T02:13:53+00:00
Next Scan 2026-01-12T02:13:53+00:00

Last Scan

Scanned2025-12-13T02:13:53+00:00
URL https://optimisemedia.com/robots.txt
Domain IPs 13.35.238.115, 13.35.238.117, 13.35.238.72, 13.35.238.96
Response IP 13.35.238.115
Found Yes
Hash acd93922d1b0454fc79e0eede1764d23e894c4e8afc5daf7205d2491e5e272eb
SimHash 0a09891c0ed2

Groups

*

Rule Path
Disallow /*_buildManifest.js$
Disallow /*_middlewareManifest.js$
Disallow /*_ssgManifest.js$
Disallow /_next/static/chunks/*.js$

ahrefsbot

No rules defined. All paths allowed.

Back to top

Other Records

Field Value
sitemap https://optimisemedia.com/sitemap.xml
Back to top

Comments

  • Next.JS Crawl Budget Performance Updates
  • Block files ending in .json, _buildManifest.js, _middlewareManifest.js, _ssgManifest.js, and any other JS files
  • The asterisks allows any file name
  • The dollar sign ensures it only matches the end of an URL and not a oddly formatted url (e.g. /locations.json.html)
Back to top

Warnings

  • `​crawl-delay` is not a known field.
Back to top