Log In Sign Up

cloudtechtwitter.com
robots.txt

Robots Exclusion Standard data for cloudtechtwitter.com

Resource Scan

Scan Details

Site Domain cloudtechtwitter.com
Base Domain cloudtechtwitter.com
Scan Status Ok
Last Scan2025-12-27T12:38:33+00:00
Next Scan 2026-01-03T12:38:33+00:00

Last Scan

Scanned2025-12-27T12:38:33+00:00
URL https://cloudtechtwitter.com/robots.txt
Redirect https://www.cloudtechtwitter.com/robots.txt
Redirect Domain www.cloudtechtwitter.com
Redirect Base cloudtechtwitter.com
Domain IPs 2001:4860:4802:32::15, 2001:4860:4802:34::15, 2001:4860:4802:36::15, 2001:4860:4802:38::15, 216.239.32.21, 216.239.34.21, 216.239.36.21, 216.239.38.21
Redirect IPs 2404:6800:4003:c11::79, 74.125.68.121
Response IP 172.217.194.121
Found Yes
Hash 5d1e6bebf8e36528df4a50a1caeba21fb76eee24e73913a673864a4767578997
SimHash 23549b70e4b0

Groups

mediapartners-google

Rule Path
Allow /

google-display-ads-bot

Rule Path
Allow /

*

Rule Path
Disallow /search
Allow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

googlebot-image

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

adsbot-google

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.cloudtechtwitter.com/sitemap.xml
sitemap https://www.cloudtechtwitter.com/sitemap-pages.xml
Back to top

Comments

  • =========================================
  • CloudTechTwitter Robots Configuration
  • Optimized for Blogger + AdSense + SEO
  • Last Updated: Nov 2025
  • =========================================
  • ----- Google AdSense and Display Bots -----
  • ----- Googlebot & General Search Engines -----
  • ----- Block Known Crawler Spam & Content Scrapers -----
  • ----- Block AI Model Training & Data Collection Bots -----
  • ----- Allow Essential Google Services -----
  • ----- Sitemaps (Auto-Indexing Support) -----
  • =========================================
  • Notes:
  • 1. /search blocked to prevent duplicate content.
  • 2. AI and scraper bots blocked to protect content.
  • 3. All legitimate search & ad bots allowed.
  • 4. Sitemaps ensure Google auto-discovers new posts/pages.
  • 5. Verified for Blogger, AdSense & Search Console.
  • =========================================
Back to top