Log In Sign Up

thunertagblatt.ch
robots.txt

Robots Exclusion Standard data for thunertagblatt.ch

Resource Scan

Scan Details

Site Domain thunertagblatt.ch
Base Domain thunertagblatt.ch
Scan Status Ok
Last Scan2024-06-07T05:25:11+00:00
Next Scan 2024-06-14T05:25:11+00:00

Last Scan

Scanned2024-06-07T05:25:11+00:00
URL https://thunertagblatt.ch/robots.txt
Redirect https://www.thunertagblatt.ch/robots.txt
Redirect Domain www.thunertagblatt.ch
Redirect Base thunertagblatt.ch
Domain IPs 18.164.174.12, 18.164.174.21, 18.164.174.55, 18.164.174.95, 2600:9000:25ef:3200:e:5a66:ac0:93a1, 2600:9000:25ef:5400:e:5a66:ac0:93a1, 2600:9000:25ef:5600:e:5a66:ac0:93a1, 2600:9000:25ef:6a00:e:5a66:ac0:93a1, 2600:9000:25ef:8600:e:5a66:ac0:93a1, 2600:9000:25ef:8c00:e:5a66:ac0:93a1, 2600:9000:25ef:b400:e:5a66:ac0:93a1, 2600:9000:25ef:c800:e:5a66:ac0:93a1
Redirect IPs 18.238.109.120, 18.238.109.121, 18.238.109.95, 18.238.109.98, 2600:9000:234d:2e00:e:5a66:ac0:93a1, 2600:9000:234d:9000:e:5a66:ac0:93a1, 2600:9000:234d:9800:e:5a66:ac0:93a1, 2600:9000:234d:c400:e:5a66:ac0:93a1, 2600:9000:234d:da00:e:5a66:ac0:93a1, 2600:9000:234d:de00:e:5a66:ac0:93a1, 2600:9000:234d:f000:e:5a66:ac0:93a1, 2600:9000:234d:fc00:e:5a66:ac0:93a1
Response IP 18.165.171.104
Found Yes
Hash 64deb6fe441f119d0a1e80d562dd74bd8620d1c3379c6dc684d74a37fa547d0d
SimHash d00681485b37

Groups

psbot
yandex
petalbot
mail.ru_bot
megaindex
baiduspider
yisouspider
bytespider
sogou web spider
sogou inst spider
proximic
admantx
seekport crawler
semrushbot
blexbot
mj12bot
dotbot
gptbot
ccbot
google-extended

Rule Path
Disallow /

*

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.thunertagblatt.ch/sitemaps/sitemapindex.xml
sitemap https://www.thunertagblatt.ch/sitemaps/news.xml
Back to top

Comments

  • Disallow commercial bots to prevent ad fraud, see DISC-2117
  • Allow crawling for other bots
Back to top

Warnings

  • 1 invalid line.
Back to top