Log In Sign Up

coba-nextjs-git-okta-survivor-story-5-ekafyi.vercel.app
robots.txt

Robots Exclusion Standard data for coba-nextjs-git-okta-survivor-story-5-ekafyi.vercel.app

Resource Scan

Scan Details

Site Domain coba-nextjs-git-okta-survivor-story-5-ekafyi.vercel.app
Base Domain coba-nextjs-git-okta-survivor-story-5-ekafyi.vercel.app
Scan Status Ok
Last Scan2024-11-05T14:07:34+00:00
Next Scan 2024-11-19T14:07:34+00:00

Last Scan

Scanned2024-11-05T14:07:34+00:00
URL https://coba-nextjs-git-okta-survivor-story-5-ekafyi.vercel.app/robots.txt
Domain IPs 76.76.21.164, 76.76.21.61
Response IP 76.76.21.98
Found Yes
Hash e43b923f8470fe20d799705a7dfccfac77793c828e93044a06615497694ed1e3
SimHash 084cc9d27771

Groups

*

Rule Path
Allow /
Disallow /pr/*

mj12bot
ahrefsbot
semrushbot
mauibot
dotbot

Rule Path
Disallow /
Back to top

Comments

  • start disallowed routes
  • pr site belum
  • end disallowed routes
  • Sitemaps index
  • not yet
  • Block bad bots
  • https://studiofreya.com/2018/11/15/how-to-stop-bots-from-crawling-your-website
  • https://www.simtechdev.com/blog/good-and-bad-bots-to-control-to-save-server-resources-and-improve-performance/
Back to top