cgtalk.ru
robots.txt

Robots Exclusion Standard data for cgtalk.ru

Archived Snapshots

Resource Scan

Scan Details

Site Domain	cgtalk.ru
Base Domain	cgtalk.ru
Scan Status	Ok
Last Scan	2025-02-27T10:18:09+00:00
Next Scan	2025-03-29T10:18:09+00:00

Last Scan

Scanned	2025-02-27T10:18:09+00:00
URL	https://cgtalk.ru/robots.txt
Domain IPs	104.21.55.45, 172.67.144.134, 2606:4700:3030::ac43:9086, 2606:4700:3037::6815:372d
Response IP	172.67.144.134
Found	Yes
Hash	e0f6e6145a8f5e7d60b89baa3ef90af755eb86f6ba4d5940ba484f60c6df9ac3
SimHash	52156b10c280

Groups

*

Rule	Path
Disallow	/darla/
Disallow	/dynamic/
Disallow	/pageview/
Disallow	/fp_ms/

Rule

Path

Disallow

/darla/

Disallow

/dynamic/

Disallow

/pageview/

Disallow

/fp_ms/

admantx

Rule	Path
Disallow	/

Rule

Path

Disallow

alphabot

Rule	Path
Disallow	/

Rule

Path

Disallow

anthropic-ai

Rule	Path
Disallow	/

Rule

Path

Disallow

awariorssbot

Rule	Path
Disallow	/

Rule

Path

Disallow

ahrefsbot

Rule	Path
Disallow	/

Rule

Path

Disallow

semrushbot

Rule	Path
Disallow	/

Rule

Path

Disallow

rogerbot

Rule	Path
Disallow	/

Rule

Path

Disallow

mj12bot

Rule	Path
Disallow	/

Rule

Path

Disallow

spbot

Rule	Path
Disallow	/

Rule

Path

Disallow

serpstatbot

Rule	Path
Disallow	/

Rule

Path

Disallow

sistrixbot

Rule	Path
Disallow	/

Rule

Path

Disallow

admantx

Rule	Path
Disallow	/

Rule

Path

Disallow

blexbot

Rule	Path
Disallow	/

Rule

Path

Disallow

buzzbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bytespider

Rule	Path
Disallow	/

Rule

Path

Disallow

ccbot

Rule	Path
Disallow	/

Rule

Path

Disallow

chatgpt-user

Rule	Path
Disallow	/

Rule

Path

Disallow

claritybot

Rule	Path
Disallow	/

Rule

Path

Disallow

claude-web

Rule	Path
Disallow	/

Rule

Path

Disallow

claudebot

Rule	Path
Disallow	/

Rule

Path

Disallow

cohere-ai

Rule	Path
Disallow	/

Rule

Path

Disallow

diffbot

Rule	Path
Disallow	/

Rule

Path

Disallow

facebookbot

Rule	Path
Disallow	/

Rule

Path

Disallow

friendlycrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

google-extended

Rule	Path
Disallow	/

Rule

Path

Disallow

gptbot

Rule	Path
Disallow	/

Rule

Path

Disallow

huggingface

Rule	Path
Disallow	/

Rule

Path

Disallow

imagesiftbot

Rule	Path
Disallow	/

Rule

Path

Disallow

img2dataset

Rule	Path
Disallow	/

Rule

Path

Disallow

magpie-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

meltwater

Rule	Path
Disallow	/

Rule

Path

Disallow

neevabot

Rule	Path
Disallow	/

Rule

Path

Disallow

news-please

Rule	Path
Disallow	/

Rule

Path

Disallow

newsnow

Rule	Path
Disallow	/

Rule

Path

Disallow

nutch

Rule	Path
Disallow	/

Rule

Path

Disallow

omgili

Rule	Path
Disallow	/

Rule

Path

Disallow

omgilibot

Rule	Path
Disallow	/

Rule

Path

Disallow

panscient.com

Rule	Path
Disallow	/

Rule

Path

Disallow

perplexity-ai

Rule	Path
Disallow	/

Rule

Path

Disallow

perplexitybot

Rule	Path
Disallow	/

Rule

Path

Disallow

petalbot

Rule	Path
Disallow	/

Rule

Path

Disallow

piplbot

Rule	Path
Disallow	/

Rule

Path

Disallow

scoop.it

Rule	Path
Disallow	/

Rule

Path

Disallow

scrapy

Rule	Path
Disallow	/

Rule

Path

Disallow

seekr

Rule	Path
Disallow	/

Rule

Path

Disallow

sentibot

Rule	Path
Disallow	/

Rule

Path

Disallow

seznambot

Rule	Path
Disallow	/

Rule

Path

Disallow

turnitinbot

Rule

Path

Disallow

youbot

Rule

Path

Disallow

zumbot

Rule

Path

Disallow

cgtalk.rurobots.txt

Resource Scan

Scan Details

Last Scan

Groups

*

admantx

alphabot

anthropic-ai

awariorssbot

ahrefsbot

semrushbot

rogerbot

mj12bot

spbot

serpstatbot

sistrixbot

admantx

blexbot

buzzbot

bytespider

ccbot

chatgpt-user

claritybot

claude-web

claudebot

cohere-ai

diffbot

facebookbot

friendlycrawler

google-extended

gptbot

huggingface

imagesiftbot

img2dataset

magpie-crawler

meltwater

neevabot

news-please

newsnow

nutch

omgili

omgilibot

panscient.com

perplexity-ai

perplexitybot

petalbot

piplbot

scoop.it

scrapy

seekr

sentibot

seznambot

turnitinbot

youbot

zumbot

cgtalk.ru
robots.txt