Log In Sign Up

prettylittlething.fr
robots.txt

Robots Exclusion Standard data for prettylittlething.fr

Resource Scan

Scan Details

Site Domain prettylittlething.fr
Base Domain prettylittlething.fr
Scan Status Ok
Last Scan2026-01-31T20:01:49+00:00
Next Scan 2026-02-14T20:01:49+00:00

Last Scan

Scanned2026-01-31T20:01:49+00:00
URL https://prettylittlething.fr/robots.txt
Redirect https://www.prettylittlething.fr/robots.txt
Redirect Domain www.prettylittlething.fr
Redirect Base prettylittlething.fr
Domain IPs 23.41.38.63
Redirect IPs 23.41.38.63
Response IP 23.75.213.151
Found Yes
Hash c36d158b6939878127d963c2bbc02778ee4914b671e5aa1c47cc0e1a9f513b60
SimHash 7f0439c7e517

Groups

sogou spider

Rule Path
Disallow /

scrapy/1.0.4 (+http://scrapy.org)

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

shopstyle bot/1.0

Rule Path
Disallow /

python-grcrawler

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

axis/1.4

Rule Path
Disallow /

proximic

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

wget

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

check_http/v2.1.1 (monitoring-plugins 2.1.1)

Rule Path
Disallow /

mozilla/5.0 (compatible; productadsbot/1.0 https://sellercentral.amazon.com/gp/help/help.html?ie=utf8&itemid=200137680)

Rule Path
Disallow /

mozilla/5.0 (compatible; productadsbot/1.0 +https://sellercentral.amazon.com/gp/help/help.html?ie=utf8&itemid=200137680)

Rule Path
Disallow /

mozilla/5.0 (compatible; megaindex.ru/2.0; +http://megaindex.com/crawler)

Rule Path
Disallow /

adsbot-google-mobile
adsbot-google

Rule Path
Disallow /rb_bf12093eab
Disallow /nugg
Disallow /ruxitagentjs
Disallow /akam

*

Rule Path
Disallow /catalogsearch/
Disallow /search
Disallow /pltcatalog/stock/getStock
Disallow /catalog/product/
Disallow /catalog/category/
Disallow /checkout
Disallow /onestepcheckout
Disallow /customer/account
Disallow /wishlist/index/
Disallow /sales/order
Disallow /tracking/?order
Disallow /directory/currency/switch/currency/
Disallow *?SID=
Disallow *?sid=
Disallow *brands%3D
Disallow *colour_group%3D
Disallow *clothes_size_mapped%3D
Disallow *price_slider%3D
Disallow *product_category%3D
Disallow *style%3D
Disallow *?fit=
Disallow *?sleeves=
Disallow *clothes_size%3D
Disallow *?brand=
Disallow *?price=
Disallow *?itemsperpage=
Disallow *amp%3Bamp%3Bamp
Disallow *%3Daffiliate
Disallow *%3Dawin
Disallow *?awc
Disallow *navigation
Disallow /rb_bf12093eab
Disallow /nugg
Disallow /ruxitagentjs
Disallow /akam
Disallow /clicktale/
Disallow /orasso/
Disallow /merchandiser/
Disallow /itm/
Disallow /br/pathways/
Disallow /*/api/v1/
Disallow /*/api/v2/
Disallow /metrics/
Disallow /member/
Disallow /browse/
Disallow /redirect
Disallow /banner_conta.php
Disallow /go?*
Disallow /go.php
Disallow /goto
Disallow /*.ru
Disallow /open_is_url
Disallow /services/
Disallow *Partner.URL
Disallow *US_CENSUS_NAME
Back to top

Other Records

Field Value
sitemap https://www.prettylittlething.fr/sitemaps/siteindex/siteindex_7.xml
sitemap https://www.prettylittlething.fr/achat/nc/sitemap/keyword.xml
Back to top

Comments

  • ****************************************************************************
  • PrettyLittleThing.fr robots.txt
  • ****************************************************************************
  • Bots to block
  • AdsBot
  • ****************************************************************************
  • Do not index system settings
  • All other bots
  • ****************************************************************************
  • Do not index search pages
  • Do not index checkout and customer account section
  • Do not index currency switcher
  • Do not index parameters
  • Do not index broken feed parameters
  • Do not index afffilate or utm links
  • Do not index navigation tracking
  • Do not index system settings
  • Other sections to block
  • Bots
  • Broken Links
  • Sitemaps
Back to top