vidaxl.de
robots.txt

Robots Exclusion Standard data for vidaxl.de

Resource Scan

Scan Details

Site Domain vidaxl.de
Base Domain vidaxl.de
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-10T10:49:44+00:00
Next Scan 2025-01-08T10:49:44+00:00

Last Successful Scan

Scanned2023-02-26T09:34:10+00:00
URL https://vidaxl.de/robots.txt
Redirect https://www.vidaxl.de/robots.txt
Redirect Domain www.vidaxl.de
Redirect Base vidaxl.de
Domain IPs 104.19.254.84, 104.19.255.84, 2606:4700::6813:fe54, 2606:4700::6813:ff54
Redirect IPs 104.19.254.84, 104.19.255.84, 2606:4700::6813:fe54, 2606:4700::6813:ff54
Response IP 104.19.255.84
Found Yes
Hash d93abf0672b877a5ad24ccd4e16fbd438a5df5e000c689e1aba0a45d4bcc4ea4
SimHash e4635374e33e

Groups

googlebot-image

Rule Path
Allow /

googlebot

Rule Path
Allow /

*

Rule Path
Disallow /on/demandware.store/*
Disallow *Login-*
Disallow *Register-*
Disallow *Wishlist-*
Disallow *Cart-*
Disallow *Checkout-*
Disallow *Account-*
Disallow *Order-*
Disallow /account
Disallow /cart
Disallow /checkout
Disallow /wishlist
Disallow /write-review
Disallow /auction/*/*
Disallow /thankyou?ID=*
Disallow /product-recalls
Disallow /*null*
Disallow /*/passwordreset/
Disallow /*/account/
Disallow /*/orders/
Disallow /*/profile/
Allow /*?dwvar
Allow /*utm_medium%3Dmerchant
Disallow /*pmin*
Disallow /*pmax*
Disallow /*srule%3D*
Disallow /*?q=*
Disallow /*prefv*%3D*
Disallow /*prefn*%3D*
Disallow /*?attributes*
Disallow /*?order_by=*

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebookbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mj12bot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

slurp

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

uptimerobot

Rule Path
Disallow /

contentking

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.vidaxl.de/sitemap_index.xml

Comments

  • Transactional and Content
  • Search and Query Params
  • Slow crawling rate for some agents
  • Disallow aggressive agents
  • Sitemaps

Warnings

  • 2 invalid lines.