Log In Sign Up

techobserver.in
robots.txt

Robots Exclusion Standard data for techobserver.in

Resource Scan

Scan Details

Site Domain techobserver.in
Base Domain techobserver.in
Scan Status Ok
Last Scan2025-12-17T10:36:04+00:00
Next Scan 2025-12-24T10:36:04+00:00

Last Scan

Scanned2025-12-17T10:36:04+00:00
URL https://techobserver.in/robots.txt
Domain IPs 104.26.0.102, 104.26.1.102, 172.67.74.187, 2606:4700:20::681a:166, 2606:4700:20::681a:66, 2606:4700:20::ac43:4abb
Response IP 104.26.0.102
Found Yes
Hash 080d0170016c1599bd3e063abbb795702245dff14002fd909023d3599985ec14
SimHash fb111d00e7ed

Groups

*

Rule Path
Allow /
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /cgi-bin/
Disallow /tmp/
Disallow /junk/
Disallow /*?sessionid=
Disallow /*?replytocom=

googlebot

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

bingbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

gptbot

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

perplexity-user

Rule Path
Allow /

copilot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

google-extended

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://techobserver.in/sitemap.xml
sitemap https://techobserver.in/news-sitemap.xml
sitemap https://techobserver.in/local-sitemap.xml
sitemap https://techobserver.in/category-sitemap.xml
sitemap https://techobserver.in/page-sitemap.xml
sitemap https://techobserver.in/product-sitemap.xml
Back to top

Comments

  • TechObserver.in robots.txt
  • -----------------------------------------------------------------------------
  • NOTICE:
  • TechObserver.in content is made available for personal, non-commercial use
  • subject to our Terms of Service: https://techobserver.in/terms-conditions/
  • The use of any automated system, bot, or process to scrape, mine, harvest,
  • or otherwise collect TechObserver.in content is strictly prohibited without
  • prior written consent from TechObserver.in.
  • Prohibited uses include, but are not limited to:
  • (1) text and data mining for AI/ML/LLM training or dataset creation
  • (2) developing commercial tools, software, or services using our content
  • (3) redistributing or archiving TechObserver.in content for third parties
  • Unauthorized use may result in legal action.
  • ----------------------------------------------------------------------------
  • --- Search & News Indexing ---
  • --- AI Search Experiences (Allowed for citation & answer generation only) ---
  • --- AI Model Training Crawlers (Blocked) ---
  • --- Future catch-all for new AI bots ---
  • --- Sitemaps ---
Back to top

Warnings

  • 2 invalid lines.
Back to top