bizthon.com
robots.txt

Robots Exclusion Standard data for bizthon.com

Archived Snapshots

Resource Scan

Scan Details

Site Domain	bizthon.com
Base Domain	bizthon.com
Scan Status	Ok
Last Scan	2026-01-31T07:34:00+00:00
Next Scan	2026-03-02T07:34:00+00:00

Last Scan

Scanned	2026-01-31T07:34:00+00:00
URL	https://bizthon.com/robots.txt
Domain IPs	104.21.64.72, 172.67.178.115, 2606:4700:3030::ac43:b273, 2606:4700:3035::6815:4048
Response IP	172.67.178.115
Found	Yes
Hash	3762a0ae6fa27e3a256dfec4056a2196f1a04f96234aeba8b477e93aadc9b088
SimHash	4d1c8945d7b3

Groups

googlebot

Rule	Path
Disallow

Rule

Path

Disallow

bingbot

Rule	Path
Disallow

Rule

Path

Disallow

chatgpt-user

Rule	Path
Disallow

Rule

Path

Disallow

chatgpt-user/1.0

Rule	Path
Disallow

Rule

Path

Disallow

gptbot

Rule	Path
Disallow

Rule

Path

Disallow

gptbot/1.0

Rule	Path
Disallow

Rule

Path

Disallow

openai-user

Rule	Path
Disallow

Rule

Path

Disallow

claudebot

Rule	Path
Disallow

Rule

Path

Disallow

chatllama

Rule	Path
Disallow

Rule

Path

Disallow

perplexitybot

Rule	Path
Disallow

Rule

Path

Disallow

anthropic

Rule	Path
Disallow

Rule

Path

Disallow

deepmind

Rule	Path
Disallow

Rule

Path

Disallow

geminibot

Rule	Path
Disallow

Rule

Path

Disallow

google-extended

Rule	Path
Disallow

Rule

Path

Disallow

ccbot

Rule	Path
Disallow

Rule

Path

Disallow

youbot

Rule	Path
Disallow

Rule

Path

Disallow

replika

Rule	Path
Disallow

Rule

Path

Disallow

quora-poe

Rule	Path
Disallow

Rule

Path

Disallow

diffbot

Rule	Path
Disallow

Rule

Path

Disallow

botify

Rule	Path
Disallow

Rule

Path

Disallow

screaming frog seo spider

Rule	Path
Disallow

Rule

Path

Disallow

scrapy

Rule	Path
Disallow

Rule

Path

Disallow

ai-crawler

Rule	Path
Disallow

Rule

Path

Disallow

ai-agent

Rule	Path
Disallow

Rule

Path

Disallow

autogpt

Rule	Path
Disallow

Rule

Path

Disallow

babyagi

Rule	Path
Disallow

Rule

Path

Disallow

langchain

Rule	Path
Disallow

Rule

Path

Disallow

agentgpt

Rule	Path
Disallow

Rule

Path

Disallow

llmagent

Rule	Path
Disallow

Rule

Path

Disallow

browsergpt

Rule	Path
Disallow

Rule

Path

Disallow

upstage-llm

Rule	Path
Disallow

Rule

Path

Disallow

airstack-maintainer

Rule	Path
Disallow

Rule

Path

Disallow

zenoai

Rule	Path
Disallow

Rule

Path

Disallow

seekrbot

Rule	Path
Disallow

Rule

Path

Disallow

gptengine

Rule	Path
Disallow

Rule

Path

Disallow

crawlai

Rule	Path
Disallow

Rule

Path

Disallow

ai-search

Rule	Path
Disallow

Rule

Path

Disallow

ai-useragent

Rule	Path
Disallow

Rule

Path

Disallow

ai

Rule	Path
Disallow

Rule

Path

Disallow

*

Rule	Path
Disallow	/cgi-bin/
Disallow	/admin/
Disallow	/private/

Rule

Path

Disallow

/cgi-bin/

Disallow

/admin/

Disallow

/private/

Other Records

Field	Value
sitemap	https://bizthon.com/sitemap.xml

Field

Value

sitemap

https://bizthon.com/sitemap.xml

Comments

Allow Googlebot (search engine)
Allow Bingbot
Allow all AI Agents, LLM crawlers & web agents
General bots - restrict admin/private areas

bizthon.comrobots.txt

Resource Scan

Scan Details

Last Scan

Groups

googlebot

bingbot

chatgpt-user

chatgpt-user/1.0

gptbot

gptbot/1.0

openai-user

claudebot

chatllama

perplexitybot

anthropic

deepmind

geminibot

google-extended

ccbot

youbot

replika

quora-poe

diffbot

botify

screaming frog seo spider

scrapy

ai-crawler

ai-agent

autogpt

babyagi

langchain

agentgpt

llmagent

browsergpt

upstage-llm

airstack-maintainer

zenoai

seekrbot

gptengine

crawlai

ai-search

ai-useragent

ai

*

Other Records

Comments

bizthon.com
robots.txt