thunertagblatt.ch
robots.txt

Robots Exclusion Standard data for thunertagblatt.ch

Resource Scan

Scan Details

Site Domain thunertagblatt.ch
Base Domain thunertagblatt.ch
Scan Status Ok
Last Scan2024-09-20T13:56:59+00:00
Next Scan 2024-09-27T13:56:59+00:00

Last Scan

Scanned2024-09-20T13:56:59+00:00
URL https://thunertagblatt.ch/robots.txt
Redirect https://www.thunertagblatt.ch/robots.txt
Redirect Domain www.thunertagblatt.ch
Redirect Base thunertagblatt.ch
Domain IPs 2600:9000:2795:2600:e:5a66:ac0:93a1, 2600:9000:2795:2a00:e:5a66:ac0:93a1, 2600:9000:2795:3000:e:5a66:ac0:93a1, 2600:9000:2795:4a00:e:5a66:ac0:93a1, 2600:9000:2795:6600:e:5a66:ac0:93a1, 2600:9000:2795:cc00:e:5a66:ac0:93a1, 2600:9000:2795:d800:e:5a66:ac0:93a1, 2600:9000:2795:f800:e:5a66:ac0:93a1, 3.164.85.11, 3.164.85.59, 3.164.85.64, 3.164.85.95
Redirect IPs 2600:9000:2664:5800:e:5a66:ac0:93a1, 2600:9000:2664:5a00:e:5a66:ac0:93a1, 2600:9000:2664:6c00:e:5a66:ac0:93a1, 2600:9000:2664:8200:e:5a66:ac0:93a1, 2600:9000:2664:8600:e:5a66:ac0:93a1, 2600:9000:2664:8c00:e:5a66:ac0:93a1, 2600:9000:2664:ec00:e:5a66:ac0:93a1, 2600:9000:2664:f800:e:5a66:ac0:93a1, 3.164.85.11, 3.164.85.59, 3.164.85.64, 3.164.85.95
Response IP 13.226.2.43
Found Yes
Hash 2e751457ad038ac2571bfad744cd5fabcf5e1f475097db5ffac6c9aebb4ae4ef
SimHash d0268348db33

Groups

psbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

proximic

Rule Path
Disallow /

admantx

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

applebot

Rule Path
Disallow /

*

Rule Path
Allow /

Other Records

Field Value
sitemap https://www.thunertagblatt.ch/sitemaps/sitemapindex.xml
sitemap https://www.thunertagblatt.ch/sitemaps/news.xml

Comments

  • Disallow commercial bots to prevent ad fraud, see DISC-2117
  • Allow crawling for other bots

Warnings

  • 2 invalid lines.