tagesanzeiger.ch
robots.txt

Robots Exclusion Standard data for tagesanzeiger.ch

Resource Scan

Scan Details

Site Domain tagesanzeiger.ch
Base Domain tagesanzeiger.ch
Scan Status Ok
Last Scan2024-11-09T09:14:46+00:00
Next Scan 2024-11-16T09:14:46+00:00

Last Scan

Scanned2024-11-09T09:14:46+00:00
URL https://tagesanzeiger.ch/robots.txt
Redirect https://www.tagesanzeiger.ch/robots.txt
Redirect Domain www.tagesanzeiger.ch
Redirect Base tagesanzeiger.ch
Domain IPs 2600:9000:2795:4a00:e:5a66:ac0:93a1, 2600:9000:2795:5e00:e:5a66:ac0:93a1, 2600:9000:2795:6a00:e:5a66:ac0:93a1, 2600:9000:2795:8c00:e:5a66:ac0:93a1, 2600:9000:2795:a00:e:5a66:ac0:93a1, 2600:9000:2795:ba00:e:5a66:ac0:93a1, 2600:9000:2795:c200:e:5a66:ac0:93a1, 2600:9000:2795:ee00:e:5a66:ac0:93a1, 3.164.85.11, 3.164.85.59, 3.164.85.64, 3.164.85.95
Redirect IPs 2600:9000:2795:6400:e:5a66:ac0:93a1, 2600:9000:2795:7200:e:5a66:ac0:93a1, 2600:9000:2795:b800:e:5a66:ac0:93a1, 2600:9000:2795:c00:e:5a66:ac0:93a1, 2600:9000:2795:cc00:e:5a66:ac0:93a1, 2600:9000:2795:da00:e:5a66:ac0:93a1, 2600:9000:2795:e400:e:5a66:ac0:93a1, 2600:9000:2795:f000:e:5a66:ac0:93a1, 3.164.85.11, 3.164.85.59, 3.164.85.64, 3.164.85.95
Response IP 52.85.49.30
Found Yes
Hash 674a51908310b8e493dedab33785767999a3f4af4f293c9b848d4b99228f05c5
SimHash 50168b60db3b

Groups

psbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

proximic

Rule Path
Disallow /

admantx

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

applebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

*

Rule Path
Allow /

Other Records

Field Value
sitemap https://www.tagesanzeiger.ch/sitemaps/sitemapindex.xml
sitemap https://www.tagesanzeiger.ch/sitemaps/news.xml

Comments

  • Disallow commercial bots to prevent ad fraud, see DISC-2117
  • Allow crawling for other bots

Warnings

  • 2 invalid lines.