Log In Sign Up

karangupta.com
robots.txt

Robots Exclusion Standard data for karangupta.com

Resource Scan

Scan Details

Site Domain karangupta.com
Base Domain karangupta.com
Scan Status Ok
Last Scan2026-02-21T13:08:36+00:00
Next Scan 2026-03-23T13:08:36+00:00

Last Scan

Scanned2026-02-21T13:08:36+00:00
URL https://karangupta.com/robots.txt
Redirect https://www.karangupta.com/robots.txt
Redirect Domain www.karangupta.com
Redirect Base karangupta.com
Domain IPs 104.26.12.14, 104.26.13.14, 172.67.71.161, 2606:4700:20::681a:c0e, 2606:4700:20::681a:d0e, 2606:4700:20::ac43:47a1
Redirect IPs 104.26.12.14, 104.26.13.14, 172.67.71.161, 2606:4700:20::681a:c0e, 2606:4700:20::681a:d0e, 2606:4700:20::ac43:47a1
Response IP 104.26.12.14
Found Yes
Hash 81eedfb9ddbe1ec4861164d516891777a04708ff47368473b2be34b066293c26
SimHash 629110b1ece9

Groups

*

Rule Path
Allow /
Allow /study-abroad/
Allow /services/
Allow /admissions-strategy/
Allow /success-stories/
Allow /blog/
Allow /reels/
Allow /scholarships
Allow /media
Allow /test-preparation/
Allow /learn/
Allow /about
Allow /contact
Allow /free-course
Allow /get-started

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

google-extended

Rule Path
Allow /

storebot-google

Rule Path
Allow /

googleother

Rule Path
Allow /

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

ccbot

Rule Path
Allow /

anthropic-ai

Rule Path
Allow /

claude-web

Rule Path
Allow /

claudebot

Rule Path
Allow /

cohere-ai

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

bytespider

Rule Path
Allow /

applebot

Rule Path
Allow /

applebot-extended

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /

facebookbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

amazonbot

Rule Path
Allow /

youbot

Rule Path
Allow /

diffbot

Rule Path
Allow /

omgili

Rule Path
Allow /

omgilibot

Rule Path
Allow /

bingbot

Rule Path
Allow /

bingpreview

Rule Path
Allow /

msnbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

yandexbot

Rule Path
Allow /

sogou

Rule Path
Allow /

exabot

Rule Path
Allow /

ia_archiver

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

pinterest

Rule Path
Allow /

pinterestbot

Rule Path
Allow /

slackbot

Rule Path
Allow /

whatsapp

Rule Path
Allow /

telegrambot

Rule Path
Allow /

discordbot

Rule Path
Allow /

embedly

Rule Path
Allow /

quora-bot

Rule Path
Allow /

redditbot

Rule Path
Allow /

semrushbot

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

mj12bot

Rule Path
Allow /

dotbot

Rule Path
Allow /

screaming frog seo spider

Rule Path
Allow /

rogerbot

Rule Path
Allow /

siteauditbot

Rule Path
Allow /

chrome-lighthouse

Rule Path
Allow /

jegl

Rule Path
Allow /

prerender

Rule Path
Allow /

prerender

Rule Path
Allow /

*

Rule Path
Disallow /admin/
Disallow /portal/
Disallow /staff/
Disallow /api/auth/
Disallow /api/admin/
Disallow /login
Disallow /register
Disallow /forgot-password
Disallow /reset-password
Disallow /search
Disallow /search?
Disallow /free-course/thank-you
Disallow /free-course/certificate
Disallow /free-course/day/
Disallow /psychometric-test/
Disallow /upload/
Disallow /resize.php
Disallow /forgotpassword/
Disallow /kcfinder/
Disallow /watch/
Disallow /reels/comment-
Disallow /reels/heres-
Disallow /reels/ivy-league-
Disallow /reels/study-abroad-
Disallow /reels/in-the-us-
Disallow /reels/watch-
Disallow /reels/while-
Disallow /reels/whats-
Disallow /reels/for-some-
Disallow /reels/is-pcmb-
Disallow /reels/are-your-
Disallow /reels/there-are-
Disallow /reels/from-maggie-
Disallow /reels/finally-
Disallow /reels/very-soon-
Disallow /reels/reel-
Disallow /reels/18
Disallow /reels/17
Disallow /reels/think-
Disallow /reels/doctor-
Disallow /reels/stop-
Disallow /reels/tea-or-
Disallow /reels/what-
Disallow /reels/your-
Disallow /reels/lets-
Disallow /reels/is-the-
Disallow /reels/as-the-
Disallow /reels/studying-
Disallow /reels/internships-
Disallow /reels/international-
Disallow /reels/want-to-
Disallow /reels/uk-scholarship-
Disallow /reels/sat-prep-
Disallow /reels/study-in-
Disallow /reels/non-russell-
Disallow /reels/curriculum-
Disallow /reels/ib-study-
Disallow /reels/scholarship-
Disallow /reels/mba-
Disallow /reels/ms-vs-
Disallow /reels/expensive-
Disallow /reels/fashion-
Disallow /reels/college-list-
Disallow /reels/top-paying-
Disallow /reels/uk-visa-
Disallow /reels/europe-
Disallow /test/welcome/
Disallow /blog/category/
Disallow /gyan/article/1
Disallow /gyan/article/3
Disallow /gyan/article/5
Disallow /gyan/article/6
Disallow /gyan/article/7
Disallow /gyan/article/8
Disallow /gyan/article/10
Disallow /*?no_redirect=
Disallow /*?from=country
Disallow /*?C=
Disallow /*?O=
Disallow /*?platform=
Disallow /*?utm_source=
Disallow /*?path=
Disallow /*?search=
Disallow /blog?tag=
Disallow /blog?category=
Disallow /blog?page=
Disallow /blog?q=
Disallow /blog?search=
Disallow /reels?tag=
Disallow /reels?category=
Disallow /testimonials?page=
Disallow /testimonials?no_redirect=
Disallow /student-reviews?page=
Disallow /student-reviews?no_redirect=
Disallow /media?page=
Disallow /media?no_redirect=
Disallow /success-stories?page=
Disallow /student-outcomes?country=
Disallow /*.json$
Disallow /static/js/*.map
Back to top

Other Records

Field Value
sitemap https://www.karangupta.com/sitemap.xml
Back to top

Comments

  • Karan Gupta Consulting - robots.txt
  • AI-Friendly & SEO Optimized
  • ===========================================
  • WELCOME ALL SEARCH ENGINE CRAWLERS
  • ===========================================
  • Critical Content Pillars
  • ===========================================
  • GOOGLE BOTS - FULL ACCESS
  • ===========================================
  • ===========================================
  • AI & LLM TRAINING BOTS - FULL ACCESS
  • ===========================================
  • ===========================================
  • OTHER SEARCH ENGINES - FULL ACCESS
  • ===========================================
  • ===========================================
  • SOCIAL MEDIA CRAWLERS - FULL ACCESS
  • ===========================================
  • ===========================================
  • SEO & ANALYTICS TOOLS - FULL ACCESS
  • ===========================================
  • ===========================================
  • PRERENDER & CACHING BOTS
  • ===========================================
  • ===========================================
  • RESTRICTED AREAS (All Bots)
  • ===========================================
  • Free Course - Only landing page is public, rest is behind login
  • Psychometric tests - Behind login
  • Block legacy URLs that no longer exist
  • Block broken /watch/ URLs (legacy video pages)
  • Block broken /reels/ URLs with slugs (legacy reel pages)
  • Block legacy test pages
  • Block legacy blog URLs with numeric IDs
  • Block query parameter variations (prevent duplicate content indexing)
  • Block development/debug files
  • ===========================================
  • SITEMAP LOCATION
  • ===========================================
Back to top