Log In Sign Up

ctgpratidin.com
robots.txt

Robots Exclusion Standard data for ctgpratidin.com

Resource Scan

Scan Details

Site Domain ctgpratidin.com
Base Domain ctgpratidin.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-11-14T02:21:04+00:00
Next Scan 2025-11-21T02:21:04+00:00

Last Successful Scan

Scanned2025-10-21T17:47:01+00:00
URL https://ctgpratidin.com/robots.txt
Domain IPs 104.21.10.152, 172.67.163.158, 2606:4700:3034::ac43:a39e, 2606:4700:3036::6815:a98
Response IP 104.21.10.152
Found Yes
Hash 15a45a1aed1dec76058dc9b302e535577641aa35e46b884af02a83b01f08ea24
SimHash 4b5cd8530730

Groups

oai-searchbot

Rule Path
Allow /

googlebot

Rule Path
Allow /wp-content/uploads/
Disallow /wp-admin/
Disallow /wp-content/plugins/
Disallow /wp-includes/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /readme.html
Disallow /refer/
Disallow /comments/feed/
Disallow /wp-content/themes/*/includes/libs/better-framework/assets/fonts/

googlebot-news

Rule Path
Allow /
Disallow /wp-admin/
Disallow /wp-content/plugins/

Other Records

Field Value
crawl-delay 5

googlebot-image

Rule Path
Allow /wp-content/uploads/
Disallow /wp-admin/
Disallow /wp-includes/

*

Rule Path
Allow /wp-content/uploads/
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-content/plugins/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /readme.html
Disallow /refer/
Disallow /comments/feed/

Other Records

Field Value
crawl-delay 20

yandex

Rule Path
Disallow /

amazon-kendra-web-crawler-*

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /do-not-crawl/

baiduspider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

seekport

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

macloud.host

Rule Path
Disallow /

macloud

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://ctgpratidin.com/sitemap-news.xml
sitemap https://ctgpratidin.com/sitemap.xml
sitemap https://ctgpratidin.com/sitemap-home.xml
sitemap https://ctgpratidin.com/sitemap-post_type-post.2021.xml
sitemap https://ctgpratidin.com/sitemap-post_type-post.2020.xml
sitemap https://ctgpratidin.com/sitemap-post_type-post.2019.xml
sitemap https://ctgpratidin.com/sitemap-post_type-post.2018.xml
sitemap https://ctgpratidin.com/sitemap-post_type-post.2017.xml
sitemap https://ctgpratidin.com/sitemap-post_type-post.2016.xml
sitemap https://ctgpratidin.com/sitemap-post_type-post.2015.xml
sitemap https://ctgpratidin.com/sitemap-post_type-page.xml
Back to top

Comments

  • ===============================
  • ROBOTS.TXT for ctgpratidin.com
  • Optimized for Google, Google News, OpenAI SearchBot, and SEO
  • ===============================
  • Allow OpenAI SearchBot
  • Googlebot (Web & News)
  • Googlebot-News
  • Googlebot-Image (image indexing)
  • General rules for all other bots
  • Aggressive / unwanted bots
  • -----------------------------
  • Sitemap files
  • -----------------------------
Back to top