11teamsports.fr
robots.txt

Robots Exclusion Standard data for 11teamsports.fr

Resource Scan

Scan Details

Site Domain 11teamsports.fr
Base Domain 11teamsports.fr
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-03T13:54:05+00:00
Next Scan 2024-07-02T13:54:05+00:00

Last Successful Scan

Scanned2024-02-11T05:30:19+00:00
URL https://11teamsports.fr/robots.txt
Domain IPs 185.115.1.170
Response IP 185.115.1.170
Found Yes
Hash 904e705573207dc044a1707972046aa911881f97133fcd583f514a11a34d494e
SimHash 6b0451518735

Groups

*

Rule Path
Disallow /c/*/sizes-*2B*
Disallow /c/*/as-*2B*
Disallow /c/*/g_price-
Disallow /c/*/colors-*%2B*
Disallow /c/*/producer-*%2B*
Disallow /c/*/type-*%2B*
Disallow /c/*/type2-*%2B*
Disallow /c/*/shoetype-*%2B*
Disallow /c/*/rada-*%2B*
Disallow /c/*/model-*%2B*
Disallow /page/event/*
Disallow /c/*?
Disallow /pg/*?
Disallow /brand/*?
Allow /brand/*?collections
Disallow /basket/*
Disallow /order/*
Disallow /user/*
Disallow /category/*
Disallow /product/gear-up*
Disallow /product/similar-products*

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

amazonbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

awariorssbot
timpibot
rainbot
awariosmartbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://11teamsports.fr/sitemap.xml

Comments

  • see http://www.robotstxt.org/orig.html for documentation
  • disable filters in categories and subcategories
  • Disallow: /c/*/
  • Disallow: /c/*/st-
  • Disallow: /c/*/colors-
  • disable dynamic content and filters
  • Disallow: /*?
  • Disallow: /p/*?
  • special content
  • ajax loading
  • some delay
  • User-agent: AhrefsBot
  • Crawl-Delay: 5
  • User-agent: Pinterestbot
  • Crawl-Delay: 3
  • User-agent: bingbot
  • Crawl-Delay: 1
  • User-agent: YandexBot
  • Crawl-Delay: 10
  • User-agent: SeekportBot
  • Crawl-delay: 10
  • User-agent: Googlebot
  • Crawl-Delay: 1
  • User-agent: YandexBot
  • Crawl-Delay: 10
  • we don't use and causing non-trivial traffic