Log In Sign Up

wiggle.com
robots.txt

Robots Exclusion Standard data for wiggle.com

Resource Scan

Scan Details

Site Domain wiggle.com
Base Domain wiggle.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-08-17T20:17:29+00:00
Next Scan 2024-11-15T20:17:29+00:00

Last Successful Scan

Scanned2023-04-19T05:57:06+00:00
URL https://www.wiggle.com/robots.txt
Domain IPs 80.69.132.17
Response IP 80.69.132.17
Found Yes
Hash a715ed402986131d81c56825c91133c10e308e52ef8ae806965d1852247d8919
SimHash 49165850680b

Groups

*

Rule Path Comment
Allow /*/*/black-friday?range=ec product-list
Allow /*/*/black-friday?range=ec -
Allow /*/black-friday?range=ec product-list
Allow /*/black-friday?range=ec -
Disallow /ajax/ -
Disallow /BasketDisplay.aspx -
Disallow /friend/ -
Disallow /AjaxEmailWhenInStock/ -
Disallow *%26referid%3D* -
Disallow *%26Referid%3D* -
Disallow *%26REFERID%3D* -
Disallow *%26ReferID%3D* -
Disallow *%26referID%3D* -
Disallow /?referid= -
Disallow /?Referid= -
Disallow /?REFERID= -
Disallow /?ReferID= -
Disallow /?referID= -
Disallow /de/ -
Disallow /fr/ -
Disallow /es/ -
Disallow /ja/ -
Disallow /nl/ -
Disallow /pt/ -
Disallow /it/ -
Disallow /ru/ -
Disallow /zh/ -
Disallow /h/option/ -
Disallow /basket/ -
Disallow /secure/ -
Disallow *?ea=* -
Disallow *%26ea%3D* -
Disallow *?s=* -
Disallow *%26s%3D* -
Disallow *?sr=* -
Disallow *%26sr%3D* -
Disallow *?ma=* -
Disallow *%26ma%3D* -
Disallow *?mi=* -
Disallow *%26mi%3D* -
Disallow *?oma=* -
Disallow *%26oma%3D* -
Disallow *?omi=* -
Disallow *%26omi%3D* -
Disallow *?layout=* -
Disallow *%26layout%3D* -
Disallow *?ris=* -
Disallow *%26ris%3D* -
Disallow *?o=* -
Disallow *%26o%3D* -
Disallow *?ps=* -
Disallow *%26ps%3D* -
Disallow *?redirectUrl=* -
Disallow *%26redirectUrl%3D* -
Disallow *?source=* -
Disallow *%26source%3D* -
Disallow *?ci_sku=* -
Disallow *%26ci_sku%3D* -
Disallow *?ci_src=* -
Disallow *%26ci_src%3D* -
Disallow *?Cat=* -
Disallow *%26Cat%3D* -
Disallow *?ProdID=* -
Disallow *%26ProdID%3D* -
Disallow *?prd=* -
Disallow *%26prd%3D* -
Disallow *?ReturnUrl=* -
Disallow *%26ReturnUrl%3D* -
Disallow *?returnUrl=* -
Disallow *%26returnUrl%3D* -
Disallow *?kpid=* -
Disallow *%26kpid%3D* -
Disallow *?pr=* -
Disallow *%26pr%3D* -
Disallow *?f=* -
Disallow *%26f%3D* -
Disallow *?ad_id_g=* -
Disallow *%26ad_id_g%3D* -
Disallow *?AD_ID_G=* -
Disallow *%26AD_ID_G%3D* -
Disallow *?adgroup_k=* -
Disallow *%26adgroup_k%3D* -
Disallow *?AdGROUP_K=* -
Disallow *%26AdGROUP_K%3D* -
Disallow *?at_xt=* -
Disallow *%26at_xt%3D* -
Disallow *?awc=* -
Disallow *%26awc%3D* -
Disallow *?campaign_k=* -
Disallow *%26campaign_k%3D* -
Disallow *?CAMPAIGN_K=* -
Disallow *%26CAMPAIGN_K%3D* -
Disallow *?channel_k=* -
Disallow *%26channel_k%3D* -
Disallow *?CHANNEL_K=* -
Disallow *%26CHANNEL_K%3D* -
Disallow *?keyword=* -
Disallow *%26keyword%3D* -
Disallow *?KEYWORD=* -
Disallow *%26KEYWORD%3D* -
Disallow *?keyword_k=* -
Disallow *%26keyword_k%3D* -
Disallow *?KEYWORD_K=* -
Disallow *%26KEYWORD_K%3D* -
Disallow *?matchtype=* -
Disallow *%26matchtype%3D* -
Disallow *?MATCHTYPE=* -
Disallow *%26MATCHTYPE%3D* -
Disallow *?tracking_id_k=* -
Disallow *%26tracking_id_k%3D* -
Disallow *?TRACKING_ID_K=* -
Disallow *%26TRACKING_ID_K%3D* -
Disallow *?utm_adgroup=* -
Disallow *%26utm_adgroup%3D* -
Disallow *?utm_agroup=* -
Disallow *%26utm_agroup%3D* -
Disallow *?prevDestCountryId=* -
Disallow *%26prevDestCountryId%3D* -
Disallow *?range=ec* -
Disallow *%26range%3Dec* -
Disallow *?range=ne* -
Disallow *%26range%3Dne* -

baiduspider
baiduspider-video
baiduspider-image
liebaofast
sogou web spider

Rule Path
Disallow /

ahrefsbot
applebot
qwantify
dotbot
semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

yandexbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.wiggle.com/sitemaps/wiggle_us_site_map_index.xml
Back to top

Comments

  • All robots will spider the domain
  • Disallow Chinese crawlers from all sites except .cn
  • Crawl delay list for all sites except .cn
  • Disallow Russian crawlers from all sites except .ru
  • Sitemaps Index
Back to top