Log In Sign Up

busy.az
robots.txt

Robots Exclusion Standard data for busy.az

Resource Scan

Scan Details

Site Domain busy.az
Base Domain busy.az
Scan Status Ok
Last Scan2026-03-29T04:49:44+00:00
Next Scan 2026-04-05T04:49:44+00:00

Last Scan

Scanned2026-03-29T04:49:44+00:00
URL https://busy.az/robots.txt
Domain IPs 104.21.82.93, 172.67.167.249, 2606:4700:3032::ac43:a7f9, 2606:4700:3035::6815:525d
Response IP 172.67.167.249
Found Yes
Hash 6dd0aeb4796d6bd935838272734d27c00ff07731ee05818dd50e003b723b08b4
SimHash 660e9ad2ac21

Groups

*

Rule Path
Allow /
Allow /vacancies
Allow /companies
Allow /jobseekers
Allow /blog
Allow /about
Allow /contact
Disallow /dashboard/
Disallow /admin/
Disallow /api/
Disallow /login
Disallow /register
Disallow /*?*
Allow /vacancies?*
Allow /companies?*
Allow /jobseekers?*

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://busy.az/sitemap_all.xml
Back to top

Comments

  • robots.txt for Busy.az
  • Important pages for crawling
  • Block sensitive areas
  • Block duplicate content
  • Sitemap location
  • Crawl delay to be respectful
  • Specific bot instructions
  • Block aggressive crawlers
Back to top