Log In Sign Up

bitcot.com
robots.txt

Robots Exclusion Standard data for bitcot.com

Resource Scan

Scan Details

Site Domain bitcot.com
Base Domain bitcot.com
Scan Status Ok
Last Scan2026-02-20T11:13:45+00:00
Next Scan 2026-02-27T11:13:45+00:00

Last Scan

Scanned2026-02-20T11:13:45+00:00
URL https://bitcot.com/robots.txt
Redirect https://www.bitcot.com/robots.txt
Redirect Domain www.bitcot.com
Redirect Base bitcot.com
Domain IPs 104.26.10.233, 104.26.11.233, 172.67.74.226, 2606:4700:20::681a:ae9, 2606:4700:20::681a:be9, 2606:4700:20::ac43:4ae2
Redirect IPs 104.26.10.233, 104.26.11.233, 172.67.74.226, 2606:4700:20::681a:ae9, 2606:4700:20::681a:be9, 2606:4700:20::ac43:4ae2
Response IP 104.26.10.233
Found Yes
Hash a4a27724204155e9dc2b2ebbc2e534eeea38f049dfaff6ba7947a1ee6eb6ef3c
SimHash 407ec27125be

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /cgi-bin/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /xmlrpc.php
Disallow /?s=
Disallow /search/
Allow /wp-admin/admin-ajax.php

*

Rule Path
Disallow /wp-content/uploads/wpforms/

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

yandexbot

Rule Path
Allow /

gptbot

Rule Path
Allow /

chatgpt-user

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

claudebot

Rule Path
Allow /

google-extended

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

perplexity-user

Rule Path
Allow /

bingai

Rule Path
Allow /

microsoftpreview

Rule Path
Allow /

facebookbot

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /

amazonbot

Rule Path
Allow /

applebot

Rule Path
Allow /

google-voice

Rule Path
Allow /

assistant

Rule Path
Allow /

siribot

Rule Path
Allow /

alexabot

Rule Path
Allow /

cortana

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

applenewsbot

Rule Path
Allow /

bingnews

Rule Path
Allow /

flipboardproxy

Rule Path
Allow /

feedly

Rule Path
Allow /

feedspot

Rule Path
Allow /

feedspotbot

Rule Path
Allow /

inoreader

Rule Path
Allow /

newsblur

Rule Path
Allow /

usabot

Rule Path
Allow /

commoncrawl

Rule Path
Allow /

ccbot

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

semrushbot

Rule Path
Allow /

mj12bot

Rule Path
Allow /

bytespider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.bitcot.com/sitemap_index.xml
Back to top

Comments

  • =========================================
  • Global Rules (All Crawlers)
  • =========================================
  • START WPFORMS BLOCK
  • END WPFORMS BLOCK
  • =========================================
  • Major Search Engines
  • =========================================
  • =========================================
  • AI Search / Generative AI Crawlers (AISO READY)
  • =========================================
  • OpenAI / ChatGPT
  • Anthropic / Claude
  • Google Gemini / AI Overviews
  • Perplexity AI
  • Microsoft Copilot / Bing AI
  • Meta AI
  • Amazon AI / Alexa
  • Apple AI / Siri / Spotlight
  • =========================================
  • Voice Search & Assistants
  • =========================================
  • =========================================
  • News Discovery & Content Platforms
  • =========================================
  • Google News
  • Apple News
  • Bing News
  • Flipboard
  • Feed Readers & Discovery
  • =========================================
  • USA & Enterprise Crawlers / Indexers
  • =========================================
  • =========================================
  • Block Aggressive / Low-Value / Risk Bots
  • =========================================
  • TikTok / ByteDance (aggressive, low SEO value)
  • Other heavy scrapers
  • =========================================
  • Sitemap
  • =========================================
Back to top