tagesanzeiger.ch
robots.txt

Robots Exclusion Standard data for tagesanzeiger.ch

Resource Scan

Scan Details

Site Domain tagesanzeiger.ch
Base Domain tagesanzeiger.ch
Scan Status Ok
Last Scan2024-09-21T06:53:37+00:00
Next Scan 2024-09-28T06:53:37+00:00

Last Scan

Scanned2024-09-21T06:53:37+00:00
URL https://tagesanzeiger.ch/robots.txt
Redirect https://www.tagesanzeiger.ch/robots.txt
Redirect Domain www.tagesanzeiger.ch
Redirect Base tagesanzeiger.ch
Domain IPs 13.226.2.114, 13.226.2.43, 13.226.2.46, 13.226.2.82, 2600:9000:2795:0:e:5a66:ac0:93a1, 2600:9000:2795:1a00:e:5a66:ac0:93a1, 2600:9000:2795:2a00:e:5a66:ac0:93a1, 2600:9000:2795:3600:e:5a66:ac0:93a1, 2600:9000:2795:5800:e:5a66:ac0:93a1, 2600:9000:2795:6200:e:5a66:ac0:93a1, 2600:9000:2795:8000:e:5a66:ac0:93a1, 2600:9000:2795:e000:e:5a66:ac0:93a1
Redirect IPs 2600:9000:2795:3e00:e:5a66:ac0:93a1, 2600:9000:2795:5800:e:5a66:ac0:93a1, 2600:9000:2795:6600:e:5a66:ac0:93a1, 2600:9000:2795:8000:e:5a66:ac0:93a1, 2600:9000:2795:9e00:e:5a66:ac0:93a1, 2600:9000:2795:a200:e:5a66:ac0:93a1, 2600:9000:2795:b200:e:5a66:ac0:93a1, 2600:9000:2795:de00:e:5a66:ac0:93a1, 3.164.85.11, 3.164.85.59, 3.164.85.64, 3.164.85.95
Response IP 52.85.49.116
Found Yes
Hash dc87aeeb96758268997c9a44913d7dcaf59ab337dbd428211492d260ae10b96b
SimHash 70168b60db3b

Groups

psbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

proximic

Rule Path
Disallow /

admantx

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

applebot

Rule Path
Disallow /

*

Rule Path
Allow /

Other Records

Field Value
sitemap https://www.tagesanzeiger.ch/sitemaps/sitemapindex.xml
sitemap https://www.tagesanzeiger.ch/sitemaps/news.xml

Comments

  • Disallow commercial bots to prevent ad fraud, see DISC-2117
  • Allow crawling for other bots

Warnings

  • 2 invalid lines.