Log In Sign Up

careeraidhub.com
robots.txt

Robots Exclusion Standard data for careeraidhub.com

Resource Scan

Scan Details

Site Domain careeraidhub.com
Base Domain careeraidhub.com
Scan Status Ok
Last Scan2025-10-28T14:18:31+00:00
Next Scan 2025-11-04T14:18:31+00:00

Last Scan

Scanned2025-10-28T14:18:31+00:00
URL https://careeraidhub.com/robots.txt
Domain IPs 104.21.3.234, 172.67.131.82, 2606:4700:3032::6815:3ea, 2606:4700:3033::ac43:8352
Response IP 104.21.3.234
Found Yes
Hash f61f0ac38799d715eaab118788e3f10ceb9e9ede480c2f906e5707f4fbf1f03a
SimHash 6ad2d8d2a43a

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /?s=
Disallow /page/*/?s=
Disallow /search/
Disallow /*/search/*
Disallow /wp-json/
Disallow /?rest_route=
Disallow /cgi-bin/
Disallow /trackback/
Disallow /comments/
Disallow /xmlrpc.php
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /readme.html
Disallow /license.txt
Disallow /.git
Disallow /*?replytocom=
Allow /wp-content/uploads/
Allow /wp-content/themes/
Allow /wp-content/plugins/
Allow /wp-content/astra-local-fonts/
Allow /wp-includes/

googlebot

Rule Path
Allow /*.css$
Allow /*.js$
Allow /*.jpg$
Allow /*.jpeg$
Allow /*.png$
Allow /*.gif$
Allow /*.webp$
Allow /*.svg$
Allow /*.woff$
Allow /*.woff2$
Allow /*.ttf$
Allow /*.otf$

googlebot-image

Rule Path
Allow /wp-content/uploads/
Allow /*.jpg$
Allow /*.jpeg$
Allow /*.png$
Allow /*.webp$
Allow /*.gif$
Allow /*.svg$

mediapartners-google

Rule Path
Allow /

google-structured-data-testing-tool

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

whatsapp

Rule Path
Allow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20
Back to top

Other Records

Field Value
sitemap https://careeraidhub.com/sitemap_index.xml
Back to top

Comments

  • ===============================
  • ✅ FINAL ROBOTS.TXT (YOAST + ADSENSE + SECURITY + CRAWLOPTIMIZED)
  • ===============================
  • ── General Bot Rules ──
  • 🔍 Block duplicate & internal search result pages
  • 🔐 Block REST API endpoints (JSON)
  • 🔐 Block unnecessary or sensitive WordPress paths
  • ✅ Allow essential static assets for rendering
  • 🎯 Googlebot (SEO + Ads compatibility)
  • 🖼️ Googlebot-Image
  • 💰 Google Ad Bot
  • ✅ Structured Data Testing
  • 🤖 Social Media Bots (optional but safe to allow)
  • 🚫 Block known aggressive bots
  • 🐌 Crawl-delay for high-impact bots
  • 🗺️ Yoast SEO Sitemap
  • ===============================
  • 🚀 END OF ROBOTS.TXT
  • ===============================
Back to top