Log In Sign Up

bernerzeitung.ch
robots.txt

Robots Exclusion Standard data for bernerzeitung.ch

Resource Scan

Scan Details

Site Domain bernerzeitung.ch
Base Domain bernerzeitung.ch
Scan Status Ok
Last Scan2024-05-14T21:50:07+00:00
Next Scan 2024-05-21T21:50:07+00:00

Last Scan

Scanned2024-05-14T21:50:07+00:00
URL https://bernerzeitung.ch/robots.txt
Redirect https://www.bernerzeitung.ch/robots.txt
Redirect Domain www.bernerzeitung.ch
Redirect Base bernerzeitung.ch
Domain IPs 18.164.174.12, 18.164.174.21, 18.164.174.55, 18.164.174.95, 2600:9000:21f8:1200:e:5a66:ac0:93a1, 2600:9000:21f8:1600:e:5a66:ac0:93a1, 2600:9000:21f8:2800:e:5a66:ac0:93a1, 2600:9000:21f8:4e00:e:5a66:ac0:93a1, 2600:9000:21f8:8600:e:5a66:ac0:93a1, 2600:9000:21f8:a200:e:5a66:ac0:93a1, 2600:9000:21f8:c200:e:5a66:ac0:93a1, 2600:9000:21f8:f000:e:5a66:ac0:93a1
Redirect IPs 18.164.174.12, 18.164.174.21, 18.164.174.55, 18.164.174.95, 2600:9000:24ba:3800:e:5a66:ac0:93a1, 2600:9000:24ba:5800:e:5a66:ac0:93a1, 2600:9000:24ba:5a00:e:5a66:ac0:93a1, 2600:9000:24ba:8200:e:5a66:ac0:93a1, 2600:9000:24ba:8600:e:5a66:ac0:93a1, 2600:9000:24ba:b800:e:5a66:ac0:93a1, 2600:9000:24ba:bc00:e:5a66:ac0:93a1, 2600:9000:24ba:fe00:e:5a66:ac0:93a1
Response IP 18.165.171.107
Found Yes
Hash c8456fa19db1b0e03bb32b4b396037fd67cd57c475af5a385c0a6a96f45c6f9a
SimHash 50068b605f37

Groups

psbot
yandex
petalbot
mail.ru_bot
megaindex
baiduspider
yisouspider
bytespider
sogou web spider
sogou inst spider
proximic
admantx
seekport crawler
semrushbot
blexbot
mj12bot
dotbot
gptbot
ccbot
google-extended

Rule Path
Disallow /

*

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.bernerzeitung.ch/sitemaps/sitemapindex.xml
sitemap https://www.bernerzeitung.ch/sitemaps/news.xml
Back to top

Comments

  • Disallow commercial bots to prevent ad fraud, see DISC-2117
  • Allow crawling for other bots
Back to top

Warnings

  • 1 invalid line.
Back to top