Log In Sign Up

prettylittlething.us
robots.txt

Robots Exclusion Standard data for prettylittlething.us

Resource Scan

Scan Details

Site Domain prettylittlething.us
Base Domain prettylittlething.us
Scan Status Ok
Last Scan2024-10-09T23:19:21+00:00
Next Scan 2024-10-23T23:19:21+00:00

Last Scan

Scanned2024-10-09T23:19:21+00:00
URL https://prettylittlething.us/robots.txt
Redirect https://www.prettylittlething.us/robots.txt
Redirect Domain www.prettylittlething.us
Redirect Base prettylittlething.us
Domain IPs 23.54.56.44
Redirect IPs 2600:1417:3f:789::3155, 2600:1417:3f:79b::3155, 96.16.116.50
Response IP 96.16.116.50
Found Yes
Hash 8608afb0bce08a1f3647963dcafc60709a7ea985769c506f796327b6167b02df
SimHash 1e0c59c3f50b

Groups

sogou spider

Rule Path
Disallow /

scrapy/1.0.4 (+http://scrapy.org)

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

shopstyle bot/1.0

Rule Path
Disallow /

python-grcrawler

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

axis/1.4

Rule Path
Disallow /

proximic

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

wget

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

check_http/v2.1.1 (monitoring-plugins 2.1.1)

Rule Path
Disallow /

mozilla/5.0 (compatible; productadsbot/1.0 https://sellercentral.amazon.com/gp/help/help.html?ie=utf8&itemid=200137680)

Rule Path
Disallow /

mozilla/5.0 (compatible; productadsbot/1.0 +https://sellercentral.amazon.com/gp/help/help.html?ie=utf8&itemid=200137680)

Rule Path
Disallow /

mozilla/5.0 (compatible; megaindex.ru/2.0; +http://megaindex.com/crawler)

Rule Path
Disallow /

adsbot-google-mobile
adsbot-google

Rule Path
Disallow /rb_bf12093eab
Disallow /nugg
Disallow /ruxitagentjs
Disallow /akam

*

Rule Path
Disallow /catalogsearch/
Disallow /search
Disallow /pltcatalog/stock/getStock
Disallow /catalog/product/
Disallow /catalog/category/
Disallow /checkout
Disallow /onestepcheckout
Disallow /customer/account
Disallow /wishlist/index/
Disallow /directory/currency/switch/currency/
Disallow /*?SID=
Disallow *brands%3D
Disallow *colour_group%3D
Disallow *clothes_size_mapped%3D
Disallow *price_slider%3D
Disallow *product_category%3D
Disallow *style%3D
Disallow *?fit=
Disallow *?sleeves=
Disallow *clothes_size%3D
Disallow *?brand=
Disallow *?price=
Disallow *amp%3Bamp%3Bamp
Disallow *%3Daffiliate
Disallow *%3Dawin
Disallow *?awc
Disallow *navigation
Disallow /rb_bf12093eab
Disallow /nugg
Disallow /ruxitagentjs
Disallow /akam
Disallow /clicktale/
Disallow /api/v1/
Disallow /api/v2/
Disallow /orasso/
Disallow /merchandiser/
Disallow /itm/
Disallow /member/
Disallow /browse/
Disallow /brands/
Disallow /redirect
Disallow /banner_conta.php
Disallow /go?*
Disallow /go.php
Disallow /goto
Disallow /*.ru
Disallow /open_is_url
Disallow /services/
Disallow *Partner.URL
Disallow *US_CENSUS_NAME
Back to top

Other Records

Field Value
sitemap https://www.prettylittlething.us/sitemaps/siteindex/siteindex_5.xml
sitemap https://www.prettylittlething.us/shop/nc/sitemap/keyword.xml
Back to top

Comments

  • ****************************************************************************
  • PrettyLittleThing.us robots.txt
  • ****************************************************************************
  • Bots to block
  • AdsBot
  • ****************************************************************************
  • Do not index system settings
  • All other bots
  • ****************************************************************************
  • Do not index search pages
  • Do not index checkout and customer account section
  • Do not index currency switcher
  • Do not index parameters
  • Do not index broken feed parameters
  • Do not index afffilate or utm links
  • Do not index navigation tracking
  • Do not index system settings
  • Other sections to block
  • Bots
  • Broken Links
  • Sitemaps
Back to top