Log In Sign Up

11teamsports.hr
robots.txt

Robots Exclusion Standard data for 11teamsports.hr

Resource Scan

Scan Details

Site Domain 11teamsports.hr
Base Domain 11teamsports.hr
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-05-26T11:51:14+00:00
Next Scan 2024-06-02T11:51:14+00:00

Last Successful Scan

Scanned2024-04-25T11:48:57+00:00
URL https://11teamsports.hr/robots.txt
Domain IPs 185.115.1.170
Response IP 185.115.1.170
Found Yes
Hash c2755afce2fdf6da579875da7629d5e3f78aab7832db06593cf13d0c6d896bf1
SimHash 6a0c9111a633

Groups

*

Rule Path
Disallow /c/*/sizes-*2B*
Disallow /c/*/asizes-*2B*
Disallow /c/*/as-*2B*
Disallow /c/*/g_price-
Disallow /c/*/colors-*%2B*
Disallow /c/*/producer-*%2B*
Disallow /c/*/type-*%2B*
Disallow /c/*/type2-*%2B*
Disallow /c/*/shoetype-*%2B*
Disallow /c/*/rada-*%2B*
Disallow /c/*/model-*%2B*
Disallow /page/event/*
Disallow /category-v2/*
Disallow /c/*?
Disallow /s/*?
Disallow /pg/*?
Disallow /brand/*?
Allow /brand/*?collections
Disallow /basket/*
Disallow /order/*
Disallow /user/*
Disallow /category/*
Disallow /product/gear-up*
Disallow /product/similar-products*

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

amazonbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

awariorssbot
timpibot
rainbot
awariosmartbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://11teamsports.hr/sitemap.xml
Back to top

Comments

  • see http://www.robotstxt.org/orig.html for documentation
  • disable filters in categories and subcategories
  • Disallow: /c/*/
  • Disallow: /c/*/st-
  • Disallow: /c/*/colors-
  • disable dynamic content and filters
  • Disallow: /*?
  • Disallow: /p/*?
  • special content
  • ajax loading
  • some delay
  • User-agent: AhrefsBot
  • Crawl-Delay: 5
  • User-agent: Pinterestbot
  • Crawl-Delay: 3
  • User-agent: bingbot
  • Crawl-Delay: 1
  • User-agent: YandexBot
  • Crawl-Delay: 10
  • User-agent: SeekportBot
  • Crawl-delay: 10
  • User-agent: Googlebot
  • Crawl-Delay: 1
  • User-agent: YandexBot
  • Crawl-Delay: 10
  • we don't use and causing non-trivial traffic
Back to top