beslist.be
robots.txt

Robots Exclusion Standard data for beslist.be

Resource Scan

Scan Details

Site Domain beslist.be
Base Domain beslist.be
Scan Status Ok
Last Scan2024-06-13T15:48:02+00:00
Next Scan 2024-06-20T15:48:02+00:00

Last Scan

Scanned2024-06-13T15:48:02+00:00
URL https://beslist.be/robots.txt
Redirect https://www.beslist.be/robots.txt
Redirect Domain www.beslist.be
Redirect Base beslist.be
Domain IPs 13.226.2.117, 13.226.2.64, 13.226.2.94, 13.226.2.98
Redirect IPs 13.227.74.102, 13.227.74.53, 13.227.74.73, 13.227.74.74
Response IP 3.160.246.126
Found Yes
Hash f01e1afe44b5aa6de4ce6b0a793181d5e61c7b7716052de2035a2571efabd8c1
SimHash 821bdd5f8613

Groups

*

Rule Path
Disallow /compare/
Disallow /consumer/
Disallow /images/
Disallow /load_product.php
Disallow /outclick/redirect
Disallow /overig/
Disallow /php/
Disallow /products/overig/
Disallow /shoppingcart/
Disallow /ufah/
Disallow /webimages/
Disallow /recently_viewed/
Disallow /app.php
Disallow /modules/
Disallow /module/
Disallow /shopsearch/
Disallow /data/graphql
Disallow /noindex/

adsbot-google

Rule Path
Disallow /compare/
Disallow /consumer/
Disallow /images/
Disallow /load_product.php
Disallow /outclick/redirect
Disallow /overig/
Disallow /php/
Disallow /products/overig/
Disallow /shoppingcart/
Disallow /ufah/
Disallow /webimages/
Disallow /recently_viewed/
Disallow /app.php
Disallow /modules/
Disallow /module/
Disallow /shopsearch/
Disallow /data/graphql
Disallow /noindex/
Disallow /cddvdrom/
Disallow /computers/
Disallow /baby_peuter/
Disallow /meubilair/
Disallow /eten_drinken/
Disallow /mode/
Disallow /software/
Disallow /huis_tuin/
Disallow /sport_outdoor_vrije-tijd/
Disallow /cadeaus_gadgets_culinair/
Disallow /gezond_mooi/
Disallow /speelgoed_spelletjes/
Disallow /sieraden_horloges/
Disallow /kantoorartikelen/
Disallow /voor_volwassenen/
Disallow /elektronica/
Disallow /films-series/
Disallow /boeken/
Disallow /huishoudelijke_apparatuur/
Disallow /main_sanitair/
Disallow /parfum_aftershave/
Disallow /horloge/
Disallow /muziekinstrument/
Disallow /schoenen/
Disallow /mode_accessoires/
Disallow /dieren_accessoires/
Disallow /klussen/
Disallow /tuin_accessoires/
Disallow /autos/
Disallow /fietsen/
Disallow /accessoires/

googlebot-image

Rule Path
Disallow /assets/responsive/loaders/

omgili

Rule Path
Disallow /

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

teracent-feed-processing

Rule Path
Disallow /

ingrid

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

jyxobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

Other Records

Field Value
sitemap https://www.beslist.be/sitemapxml/be/current/sitemap-category-index.xml
sitemap https://www.beslist.be/sitemapxml/be/current/sitemap-browse-index.xml
sitemap https://www.beslist.be/sitemapxml/be/current/sitemap-landing-index.xml