critic.de
robots.txt

Robots Exclusion Standard data for critic.de

Resource Scan

Scan Details

Site Domain critic.de
Base Domain critic.de
Scan Status Ok
Last Scan2024-09-22T10:06:25+00:00
Next Scan 2024-09-29T10:06:25+00:00

Last Scan

Scanned2024-09-22T10:06:25+00:00
URL https://critic.de/robots.txt
Redirect https://www.critic.de/robots.txt
Redirect Domain www.critic.de
Redirect Base critic.de
Domain IPs 89.110.149.182
Redirect IPs 89.110.149.182
Response IP 89.110.149.182
Found Yes
Hash f75475e82f173204c2d6e15ae7e9022680ebc84316411667333f6e83ea66ecd6
SimHash 0dd4c151ac12

Groups

*

Rule Path
Allow /
Disallow /person/
Disallow /ov-movies-berlin/cinema/
Disallow /ov-movies-berlin/movie/

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandexvideoparser

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

exabot

Rule Path
Disallow /

exabot-thumbnails

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

expo9

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

g-i-g-a-b-o-t

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

spbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /