wespring.com
robots.txt

Robots Exclusion Standard data for wespring.com

Resource Scan

Scan Details

Site Domain wespring.com
Base Domain wespring.com
Scan Status Ok
Last Scan2024-08-31T23:31:26+00:00
Next Scan 2024-09-30T23:31:26+00:00

Last Scan

Scanned2024-08-31T23:31:26+00:00
URL https://wespring.com/robots.txt
Redirect https://www.wespring.com/robots.txt
Redirect Domain www.wespring.com
Redirect Base wespring.com
Domain IPs 76.76.21.21
Redirect IPs 76.76.21.164, 76.76.21.93
Response IP 76.76.21.61
Found Yes
Hash 83ca07a4681c69bbff03dc7b8b1e095f0526cbe89a33673765e94291598f7f83
SimHash 0c61d9d2e418

Groups

*

Rule Path
Disallow /

googlebot
googlebot smartphone
googlebot-image
googlebot-news
googlebot-video
mediapartners-google
adsbot-google
adsbot-google-mobile
adsbot-google-mobile-apps
bingbot
adidxbot
duckduckbot
qwantify
screaming frog seo spider
oncrawl
facebook
pinterest
ahrefsbot
semrushbot
barkrowler
chatgpt-user
gptbot

Rule Path
Allow /
Disallow /reviews?product=*
Disallow /pages/wildweb-copy-of-handle-checkout-once-app
Disallow /pages/lpa-lessive
Disallow /pages/handle-checkout-once-app
Disallow /collections/darty-recurring-orders-20
Allow /*.jpg************
Allow /*.jpeg************
Allow /*.png************
Allow /*.gif************
Allow /*.webp************
Allow /*.svg************
Allow /*.tif************
Allow /*.mp4***********

adsbot-google
adsbot-google-mobile
adsbot-google-mobile-apps

Rule Path
Allow /********************

Other Records

Field Value
sitemap https://www.wespring.com/sitemap.xml

Comments

  • Block access to all UA by default
  • Allow access to specific UA
  • Disallow crawling of specific pages
  • Allow images and special medias