avril-beaute.fr
robots.txt

Robots Exclusion Standard data for avril-beaute.fr

Resource Scan

Scan Details

Site Domain avril-beaute.fr
Base Domain avril-beaute.fr
Scan Status Ok
Last Scan2024-10-25T20:40:11+00:00
Next Scan 2024-11-24T20:40:11+00:00

Last Scan

Scanned2024-10-25T20:40:11+00:00
URL https://avril-beaute.fr/robots.txt
Domain IPs 46.105.150.113
Response IP 46.105.150.113
Found Yes
Hash f1e16a504a3b935aa474819a1239c8efbbd3969bed9a16f1342d9aa36d9001d3
SimHash 9d14f3d20f87

Groups

*

Rule Path
Disallow
Disallow /lib/
Disallow /*.php$
Disallow /pkginfo/
Disallow /report/
Disallow /var/
Disallow /catalog/
Disallow /customer/
Disallow /sendfriend/
Disallow /wishlist/
Disallow /review/
Disallow /*SID%3D
Disallow /*?
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /catalog/product/gallery/
Disallow /newsletter/
Disallow /amlocator/
Disallow /app/
Disallow /bin/
Disallow /dev/
Disallow /lib/
Disallow /phpserver/
Disallow /pub/
Disallow /static/
Disallow /tag/
Disallow /review/
Disallow /composer.json
Disallow /composer.lock
Disallow /CONTRIBUTING.md
Disallow /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow /COPYING.txt
Disallow /Gruntfile.js
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /LICENSE.html
Disallow /nginx.conf.sample
Disallow /package.json
Disallow /php.ini.sample
Disallow /RELEASE_NOTES.txt
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /install.php
Disallow /STATUS.txt
Disallow /*?*product_list_mode=
Disallow /*?*product_list_order=
Disallow /*?*product_list_limit=
Disallow /*?*product_list_dir=
Disallow /*page%3D
Disallow /*sort%3D
Disallow /*.git
Disallow /*.CVS
Disallow /*.Zip$
Disallow /*.Svn$
Disallow /*.Idea$
Disallow /*.Sql$
Disallow /*.Tgz$

adequat

Rule Path
Disallow /

admantx bot

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

corporama

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotmic dotbot

Rule Path
Disallow /

ellisphere

Rule Path
Disallow /

etaospider

Rule Path
Disallow /

fetch

Rule Path
Disallow /

grapeshot bot

Rule Path
Disallow /

houzzbot

Rule Path
Disallow /

infoseek

Rule Path
Disallow /

jetbot

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

leadbox

Rule Path
Disallow /

linkfluence

Rule Path
Disallow /

linko

Rule Path
Disallow /

mail.ru bot

Rule Path
Disallow /

meltwater

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

npbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

opinion-tracker

Rule Path
Disallow /

proxem

Rule Path
Disallow /

proximic bot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

spotter

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

teleport

Rule Path
Disallow /

updownerbot

Rule Path
Disallow /

verticalsearch

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wget

Rule Path
Disallow /

winello

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

orangebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

yandexbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

archive.org_bot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

ahrefsbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

soso spider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

sogou spider

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

net veille

Rule Path
Disallow /

paarly

Rule Path
Disallow /

spycommerce

Rule Path
Disallow /

pricing assistant

Rule Path
Disallow /

price observatory

Rule Path
Disallow /

workit

Rule Path
Disallow /

keox

Rule Path
Disallow /

pricejet

Rule Path
Disallow /

prixan

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.avril-beaute.fr/sitemaps/fr/sitemap.xml

Comments

  • Disable checkout & customer account
  • Disable Search pages
  • Paths (clean URLs)
  • Disable common folders
  • Disable Tag & Review (Avoid duplicate content)
  • Common files
  • Disable sorting (Avoid duplicate content)
  • Disable version control folders and others
  • Bloquage Robots indesirables
  • Google Image Crawler Setup
  • Ralentissement de robots moins utiles
  • Bloquage des outils de veille de prix