Log In Sign Up

langenthalertagblatt.ch
robots.txt

Robots Exclusion Standard data for langenthalertagblatt.ch

Resource Scan

Scan Details

Site Domain langenthalertagblatt.ch
Base Domain langenthalertagblatt.ch
Scan Status Ok
Last Scan2024-06-02T18:06:33+00:00
Next Scan 2024-06-09T18:06:33+00:00

Last Scan

Scanned2024-06-02T18:06:33+00:00
URL https://langenthalertagblatt.ch/robots.txt
Redirect https://www.langenthalertagblatt.ch/robots.txt
Redirect Domain www.langenthalertagblatt.ch
Redirect Base langenthalertagblatt.ch
Domain IPs 13.226.2.114, 13.226.2.43, 13.226.2.46, 13.226.2.82, 2600:9000:21f8:1600:e:5a66:ac0:93a1, 2600:9000:21f8:7c00:e:5a66:ac0:93a1, 2600:9000:21f8:8800:e:5a66:ac0:93a1, 2600:9000:21f8:8e00:e:5a66:ac0:93a1, 2600:9000:21f8:9a00:e:5a66:ac0:93a1, 2600:9000:21f8:a600:e:5a66:ac0:93a1, 2600:9000:21f8:dc00:e:5a66:ac0:93a1, 2600:9000:21f8:e000:e:5a66:ac0:93a1
Redirect IPs 18.64.67.116, 18.64.67.124, 18.64.67.20, 18.64.67.49, 2600:9000:26cc:1e00:e:5a66:ac0:93a1, 2600:9000:26cc:2600:e:5a66:ac0:93a1, 2600:9000:26cc:4000:e:5a66:ac0:93a1, 2600:9000:26cc:ac00:e:5a66:ac0:93a1, 2600:9000:26cc:b600:e:5a66:ac0:93a1, 2600:9000:26cc:de00:e:5a66:ac0:93a1, 2600:9000:26cc:e00:e:5a66:ac0:93a1, 2600:9000:26cc:e200:e:5a66:ac0:93a1
Response IP 18.165.171.104
Found Yes
Hash 8907b422e738e36186f27517d38320274d7f133fa3e3a07bfdfcc2a6695a5f66
SimHash d04683405b3f

Groups

psbot
yandex
petalbot
mail.ru_bot
megaindex
baiduspider
yisouspider
bytespider
sogou web spider
sogou inst spider
proximic
admantx
seekport crawler
semrushbot
blexbot
mj12bot
dotbot
gptbot
ccbot
google-extended

Rule Path
Disallow /

*

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.langenthalertagblatt.ch/sitemaps/sitemapindex.xml
sitemap https://www.langenthalertagblatt.ch/sitemaps/news.xml
Back to top

Comments

  • Disallow commercial bots to prevent ad fraud, see DISC-2117
  • Allow crawling for other bots
Back to top

Warnings

  • 1 invalid line.
Back to top