vidaxl.com.au
robots.txt

Robots Exclusion Standard data for vidaxl.com.au

Resource Scan

Scan Details

Site Domain vidaxl.com.au
Base Domain vidaxl.com.au
Scan Status Ok
Last Scan2026-02-05T12:47:58+00:00
Next Scan 2026-03-07T12:47:58+00:00

Last Scan

Scanned2026-02-05T12:47:58+00:00
URL https://vidaxl.com.au/robots.txt
Redirect https://www.vidaxl.com.au/robots.txt
Redirect Domain www.vidaxl.com.au
Redirect Base vidaxl.com.au
Domain IPs 104.18.39.125, 172.64.148.131, 2606:4700:440d::6812:277d, 2a06:98c1:310b::ac40:9483
Redirect IPs 104.18.39.125, 172.64.148.131, 2606:4700:440d::6812:277d, 2a06:98c1:310b::ac40:9483
Response IP 172.64.148.131
Found Yes
Hash 95fd6474d121627f7db8f93186cccddf80e5b0f08faca8b3d996f896c2289d81
SimHash ec625274eb3a

Groups

*

Rule Path
Disallow /on/demandware.store/*
Disallow *Login-*
Disallow *Register-*
Disallow *Wishlist-*
Disallow *Cart-*
Disallow *Checkout-*
Disallow *Account-*
Disallow *Order-*
Disallow /account
Disallow /cart
Disallow /checkout
Disallow /wishlist
Disallow /write-review
Disallow /auction/*/*
Disallow /thankyou?ID=*
Disallow /product-recalls
Disallow /*null*
Disallow /*/passwordreset/
Disallow /*/account/
Disallow /*/orders/
Disallow /*/profile/
Allow /*?dwvar
Allow /*utm_medium%3Dmerchant
Disallow /*pmin*
Disallow /*pmax*
Disallow /*srule%3D*
Disallow /*?q=*
Disallow /*prefv*%3D*
Disallow /*prefn*%3D*
Disallow /*?attributes*
Disallow /*?order_by=*

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebookbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mj12bot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

slurp

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

uptimerobot

Rule Path
Disallow /

contentking

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.vidaxl.com.au/sitemap_index.xml

Comments

  • Transactional and Content
  • Search and Query Params
  • Slow crawling rate for some agents
  • Disallow aggressive agents
  • Sitemaps

Warnings

  • 2 invalid lines.