Log In Sign Up

shanthicabs.com
robots.txt

Robots Exclusion Standard data for shanthicabs.com

Resource Scan

Scan Details

Site Domain shanthicabs.com
Base Domain shanthicabs.com
Scan Status Ok
Last Scan2025-08-12T15:20:43+00:00
Next Scan 2025-09-11T15:20:43+00:00

Last Scan

Scanned2025-08-12T15:20:43+00:00
URL https://shanthicabs.com/robots.txt
Domain IPs 23.88.7.241
Response IP 23.88.7.241
Found Yes
Hash 942a0bf9bcff399de316df855e39250ee5b3460d056ccf06a9e38a2fdff310a4
SimHash 4015cbd3e519

Groups

*
googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /
Disallow /search?q=
Disallow /cart/
Disallow /checkout/
Disallow /account/
Allow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.shanthicabs.com/sitemap.xml
Back to top

Comments

  • Prevent indexing of duplicate or unnecessary pages
  • Allow all user-agents to crawl public pages
  • Block specific bots known for spam
Back to top