Log In Sign Up

streakwave.com
robots.txt

Robots Exclusion Standard data for streakwave.com

Resource Scan

Scan Details

Site Domain streakwave.com
Base Domain streakwave.com
Scan Status Ok
Last Scan2025-12-27T18:48:19+00:00
Next Scan 2026-01-26T18:48:19+00:00

Last Scan

Scanned2025-12-27T18:48:19+00:00
URL https://www.streakwave.com/robots.txt
Domain IPs 172.66.40.226, 172.66.43.30, 2606:4700:3108::ac42:28e2, 2606:4700:3108::ac42:2b1e
Response IP 172.66.40.226
Found Yes
Hash c82b42f25a6ee4d7432a717d6ad14be9988aafcb6e491ae9102f468096bb17ce
SimHash 041e48b3e7df

Groups

googlebot

Rule Path
Disallow

bingbot

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

applebot

Rule Path
Disallow

facebookexternalhit

Rule Path
Disallow

twitterbot

Rule Path
Disallow

linkedinbot

Rule Path
Disallow

*

Rule Path
Disallow /cart/
Disallow /checkout/
Disallow /account/
Disallow /my-account/
Disallow /login
Disallow /admin/
Disallow /search

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

ccbot

Product Comment
ccbot CommonCrawl
Rule Path
Disallow /

bytespider

Product Comment
bytespider TikTok
Rule Path
Disallow /

petalbot

Product Comment
petalbot Huawei
Rule Path
Disallow /

yandex

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

sitecheckerbotcrawler

Rule Path
Disallow /

seobilitybot

Rule Path
Disallow /

*

Rule Path
Disallow /bin/
Disallow /files/
Disallow /files/exportimport/
Disallow /country/getstatesbycountryid
Disallow /setproductreviewhelpfulness
Disallow /*?*returnUrl=
Disallow /addproducttocart/catalog/
Disallow /addproducttocart/details/
Disallow /backinstocksubscriptions/manage
Disallow /cart$
Disallow /changecurrency
Disallow /changelanguage
Disallow /changetaxtype
Disallow /checkout
Disallow /checkout/billingaddress
Disallow /checkout/completed
Disallow /checkout/confirm
Disallow /checkout/shippingaddress
Disallow /checkout/shippingmethod
Disallow /checkout/paymentinfo
Disallow /checkout/paymentmethod
Disallow /clearcomparelist
Disallow /compareproducts
Disallow /compareproducts/add/*
Disallow /customer/activation
Disallow /customer/addresses
Disallow /customer/changepassword
Disallow /customer/checkusernameavailability
Disallow /customer/downloadableproducts
Disallow /customer/info
Disallow /customer/productreviews
Disallow /deletepm
Disallow /emailwishlist
Disallow /eucookielawaccept
Disallow /inboxupdate
Disallow /newsletter/subscriptionactivation
Disallow /onepagecheckout
Disallow /order/history
Disallow /orderdetails
Disallow /passwordrecovery/confirm
Disallow /poll/vote
Disallow /recentlyviewedproducts
Disallow /returnrequest
Disallow /returnrequest/history
Disallow /rewardpoints/history
Disallow /search?
Disallow /sentupdate
Disallow /shoppingcart/*
Disallow /storeclosed
Disallow /subscribenewsletter
Disallow /topic/authenticate
Disallow /uploadfilecheckoutattribute
Disallow /uploadfileproductattribute
Disallow /uploadfilereturnrequest
Disallow /wishlist

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.streakwave.com/sitemap.xml
sitemap https://www.streakwave.com/sitemap.xml
Back to top

Comments

  • robots.txt for https://www.streakwave.com
  • Allow major search engines. No crawl-delay.
  • Allow social/link preview fetchers so shares render nicely
  • Disallow low-value/private pages (adjust paths to your platform as needed)
  • (Optional) prevent tag/search result pages if applicable:
  • Disallow: /tag/
  • Disallow: /?s=
  • Block AI/data training & non-essential crawlers
  • Block common SEO scrapers / aggressive bots (not search engines)
  • Sitemap(s)
Back to top

Warnings

  • `host` is not a known field.
Back to top