biernet.nl
robots.txt

Robots Exclusion Standard data for biernet.nl

Resource Scan

Scan Details

Site Domain biernet.nl
Base Domain biernet.nl
Scan Status Ok
Last Scan2024-11-10T21:49:20+00:00
Next Scan 2024-11-17T21:49:20+00:00

Last Scan

Scanned2024-11-10T21:49:20+00:00
URL https://biernet.nl/robots.txt
Redirect https://www.biernet.nl/robots.txt
Redirect Domain www.biernet.nl
Redirect Base biernet.nl
Domain IPs 83.149.93.69
Redirect IPs 83.149.93.69
Response IP 83.149.93.69
Found Yes
Hash 2d47f574f013c0b23af756b7e02c15210253e17ebc189933daaf1ebfd7e77c95
SimHash 642f44324f7b

Groups

*

Rule Path
Disallow /*.php$
Disallow /bja/
Disallow /koop/merken/
Disallow */lat%3A*
Disallow */lng%3A*
Disallow */zoom%3A*
Disallow */begindatum%3A*
Disallow */einddatum%3A*

ahrefsbot
baiduspider
baiduspider-video
baiduspider-image
dotbot
ichiro
mauibot
mj12bot
moget
naverbot
petalbot
semrushbot
seznambot
sogou spider
yandex
yeti
youdaobot

Rule Path
Disallow /

Comments

  • Alle urls met GET-parameters blocken (Versienummers worden hierdoor niet geblokkeerd.)
  • 05-05-22 update: door imgs commenten
  • Disallow: /*?*=
  • Lat + lng + zoom + begin/einddatum parameters blokkeren van de kaartenpagina
  • Bekendere bots overslaan