vidaxl.co.uk
robots.txt

Robots Exclusion Standard data for vidaxl.co.uk

Resource Scan

Scan Details

Site Domain vidaxl.co.uk
Base Domain vidaxl.co.uk
Scan Status Ok
Last Scan2026-03-05T08:54:33+00:00
Next Scan 2026-04-04T08:54:33+00:00

Last Scan

Scanned2026-03-05T08:54:33+00:00
URL https://vidaxl.co.uk/robots.txt
Redirect https://www.vidaxl.co.uk/robots.txt
Redirect Domain www.vidaxl.co.uk
Redirect Base vidaxl.co.uk
Domain IPs 104.18.39.199, 172.64.148.57, 2606:4700:440d::6812:27c7, 2a06:98c1:310d::ac40:9439
Redirect IPs 104.18.39.199, 172.64.148.57, 2606:4700:440d::6812:27c7, 2a06:98c1:310d::ac40:9439
Response IP 104.18.39.199
Found Yes
Hash 3c8f85e8eaba86f377f04c0b401510c8b339b1aff3e3f094e29746290f2dfa17
SimHash ec625274eb3a

Groups

*

Rule Path
Disallow /on/demandware.store/*
Disallow *Login-*
Disallow *Register-*
Disallow *Wishlist-*
Disallow *Cart-*
Disallow *Checkout-*
Disallow *Account-*
Disallow *Order-*
Disallow /account
Disallow /cart
Disallow /checkout
Disallow /wishlist
Disallow /write-review
Disallow /auction/*/*
Disallow /thankyou?ID=*
Disallow /product-recalls
Disallow /*null*
Disallow /*/passwordreset/
Disallow /*/account/
Disallow /*/orders/
Disallow /*/profile/
Allow /*?dwvar
Allow /*utm_medium%3Dmerchant
Disallow /*pmin*
Disallow /*pmax*
Disallow /*srule%3D*
Disallow /*?q=*
Disallow /*prefv*%3D*
Disallow /*prefn*%3D*
Disallow /*?attributes*
Disallow /*?order_by=*

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebookbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mj12bot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

slurp

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

uptimerobot

Rule Path
Disallow /

contentking

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.vidaxl.co.uk/sitemap_index.xml

Comments

  • Transactional and Content
  • Search and Query Params
  • Slow crawling rate for some agents
  • Disallow aggressive agents
  • Sitemaps

Warnings

  • 2 invalid lines.