Log In Sign Up

brandalley.co.uk
robots.txt

Robots Exclusion Standard data for brandalley.co.uk

Resource Scan

Scan Details

Site Domain brandalley.co.uk
Base Domain brandalley.co.uk
Scan Status Ok
Last Scan2024-10-21T04:51:08+00:00
Next Scan 2024-11-20T04:51:08+00:00

Last Scan

Scanned2024-10-21T04:51:08+00:00
URL https://brandalley.co.uk/robots.txt
Redirect https://www.brandalley.co.uk/robots.txt
Redirect Domain www.brandalley.co.uk
Redirect Base brandalley.co.uk
Domain IPs 104.18.0.52, 104.18.1.52, 2606:4700::6812:134, 2606:4700::6812:34
Redirect IPs 104.18.0.52, 104.18.1.52, 2606:4700::6812:134, 2606:4700::6812:34
Response IP 104.18.1.52
Found Yes
Hash d048cc61adcbb8af6ef9d11897f9c91bda9f73c615c70d5126ca289a8b4a9bcb
SimHash a512f1538ab3

Groups

*

Rule Path
Allow /blog/
Disallow /CVS
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$
Disallow /admin/
Disallow /c6th2gbAALm99/
Disallow /404/
Disallow /app/
Disallow /cgi-bin/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /media/
Disallow /media/captcha/
Disallow /media/customer/
Disallow /media/dhl/
Disallow /media/downloadable/
Disallow /media/import/
Disallow /media/pdf/
Disallow /media/sales/
Disallow /media/tmp/
Disallow /media/wysiwyg/
Disallow /media/xmlconnect/
Disallow /pkginfo/
Disallow /reports/
Disallow /scripts/
Disallow /shell/
Disallow /stats/
Disallow /var/
Disallow /maintenance_page/
Disallow /firewall_error_page/
Disallow /documentation/
Disallow /ogone/
Disallow /*?SID=
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /achicalanding/incoming/
Disallow /index.php/
Disallow /catalog/product/gallery/
Disallow /control/
Disallow /contacts/
Disallow /customize/
Disallow /newsletter/
Disallow /poll/
Disallow /review/
Disallow /sendfriend/
Disallow /tag/
Disallow /wishlist/
Disallow /rm-returns/
Disallow /api.php
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /install.php
Disallow /get.php
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow /STATUS.txt
Disallow /mage
Disallow /chanel.html
Disallow /checkSession.php
Disallow /gitdeploymentversion.txt
Disallow /maintenance.php
Disallow /scheduler_cron.sh
Disallow /timeonserver.php
Disallow /MAGENTO-520-patch.patch
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php
Disallow /*?dir=*
Disallow /*%26dir%3D*
Disallow /*?mode=*
Disallow /*%26mode%3D*
Disallow /*?order=*
Disallow /*%26order%3D*
Disallow /*?is_ajax=*
Disallow /*%26is_ajax%3D*
Disallow /*?limit=*
Disallow /*%26limit%3D*
Disallow /*?product=*
Disallow /*%26product%3D*
Disallow /*?portfolioID=*
Disallow /*%26portfolioID%3D*
Disallow /*?___store=*
Disallow /rss*
Disallow /*PHPSESSID
Disallow /blog/print/
Allow /blog/?display=wide%2F
Allow /blog/wp-content/uploads/
Allow /blog/wp-json/posts
Disallow /blog/readme.html
Disallow /blog/wp-config.php
Disallow /blog/refer/
Disallow /blog/wp-admin/
Disallow /blog/wp-content/mu-plugins/
Disallow /blog/wp-content/upgrade/
Disallow /register-awin/
Disallow /register-to-win-cosmopolitan/
Disallow /register-to-win-marie-claire-2/
Disallow /register-to-win-marie-claire/
Disallow /register-to-win-weekend-hamper/
Disallow /voucher-competition/
Disallow /7fam-sales
Disallow /activewear-sale
Disallow /all-saints
Disallow /allsaints-sale
Disallow /ascot-sales
Disallow /bally-sales
Disallow /bedlinen
Disallow /boss
Disallow /brandalley_kids
Disallow /browser-not-supported
Disallow /careers
Disallow /cashmere
Disallow /castaner-sales
Disallow /christmas-advent-calendar
Disallow /christmas-shop
Disallow /dieselsales
Disallow /dkny-sales
Disallow /dune
Disallow /enable-cookies
Disallow /gant
Disallow /gant-sales
Disallow /geographical-norway
Disallow /guess
Disallow /hackettsales
Disallow /halston-heritage-sales
Disallow /heidi-klum-sales
Disallow /hobbs-sales
Disallow /hobbsandfitflop
Disallow /hunter-sales
Disallow /j-brand
Disallow /j-brand-sale
Disallow /jackwills
Disallow /jaeger
Disallow /jaeger-sales
Disallow /job-opportunities
Disallow /juicy-couture-sale
Disallow /karen-millen
Disallow /kidswear-sale
Disallow /leon-max-sales
Disallow /lkbennett
Disallow /lombok-sales
Disallow /mango
Disallow /michael-kors
Disallow /mulberrysales
Disallow /musto-sale
Disallow /newbalance
Disallow /no-route
Disallow /oliver-sweeney
Disallow /oliver-sweeney-sale
Disallow /oneill
Disallow /orla-kiely-sale
Disallow /petit-bateau
Disallow /privacy-policy-cookie-restriction-mode
Disallow /rag-and-bone-sales
Disallow /regatta
Disallow /reiss
Disallow /reissmen
Disallow /reisswomen
Disallow /replaysales
Disallow /reward-points
Disallow /service-unavailable
Disallow /slipper-boutique
Disallow /soak-sleep-sale
Disallow /spyder
Disallow /stella-mccartney-sales
Disallow /superdry
Disallow /ted-baker
Disallow /ted-baker-footwear
Disallow /test-pages
Disallow /thanks-for-entering
Disallow /thanks-for-registering
Disallow /timberland-sales
Disallow /ugg
Disallow /18075664.html
Disallow /18075662.html
Disallow /18075660.html
Disallow /18075658.html
Disallow /18075656.html
Disallow /18029505.html
Disallow /18029503.html
Disallow /18029501.html
Disallow /18029499.html
Disallow /18029497.html
Disallow /18029495.html
Disallow /18005529.html
Disallow /18005527.html
Disallow /18005525.html
Disallow /18005523.html
Disallow /18005521.html
Disallow /17969634.html
Disallow /17969632.html
Disallow /17969630.html
Disallow /17969628.html
Disallow /17969626.html
Disallow /17969622.html
Disallow /17969620.html
Disallow /17969618.html
Disallow /17969616.html
Disallow /17969614.html
Disallow /17969612.html
Disallow /17969610.html
Disallow /17969608.html
Disallow /17969606.html
Disallow /17969604.html
Disallow /17928148.html
Disallow /17928146.html
Disallow /17928144.html
Disallow /17928142.html
Disallow /17928140.html
Disallow /17882412.html
Disallow /17882410.html
Disallow /17882408.html
Disallow /17882406.html
Disallow /17882404.html
Disallow /17882402.html
Disallow /17882400.html
Disallow /17882398.html
Disallow /17882396.html
Disallow /17882394.html
Disallow /17882392.html
Disallow /17882390.html
Disallow /17882388.html
Disallow /17882386.html
Disallow /17882384.html
Disallow /17882382.html
Disallow /17882380.html
Disallow /17882378.html
Disallow /17882376.html
Disallow /17882374.html
Disallow /17882372.html
Disallow /17143646.html
Disallow /17143644.html
Disallow /17143642.html
Disallow /17143640.html
Disallow /17143638.html
Disallow /17143636.html
Disallow /17143634.html
Disallow /17143632.html
Disallow /18075665.html
Disallow /18029506.html
Disallow /18029504.html
Disallow /18029500.html
Disallow /18029498.html
Disallow /18005528.html
Disallow /18005524.html
Disallow /18005522.html
Disallow /17969635.html
Disallow /17969633.html
Disallow /17969631.html
Disallow /17969629.html
Disallow /17969627.html
Disallow /17969623.html
Disallow /17969619.html
Disallow /17969617.html
Disallow /17969605.html
Disallow /17882411.html
Disallow /17882407.html
Disallow /17882403.html
Disallow /17882399.html
Disallow /17882397.html
Disallow /17882395.html
Disallow /17882391.html
Disallow /17882389.html
Disallow /17882385.html
Disallow /17882383.html
Disallow /17882377.html
Disallow /17882375.html
Disallow /17143647.html
Disallow /17143645.html
Disallow /17143643.html
Disallow /17143641.html
Disallow /17143637.html
Disallow /17143635.html
Disallow /17143633.html

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

nutch

Rule Path
Disallow /

spock

Rule Path
Disallow /

omniexplorer_bot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

geniebot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

sbider/nutch

Rule Path
Disallow /

sbider

Rule Path
Disallow /

jyxobot

Rule Path
Disallow /

magent

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

huasai

Rule Path
Disallow /

datacha0s

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

atomic_email_hunter

Rule Path
Disallow /

mp3bot

Rule Path
Disallow /

winhttp

Rule Path
Disallow /

betabot

Rule Path
Disallow /

core-project

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

java

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

orthogaffe

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

webreaper

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.brandalley.co.uk/sitemap.xml
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these â€Ã
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://www.brandalley.co.uk/robots.txt
  • Ignored: http:// www.brandalley.co.uk/site/robots.txt
  • Website Sitemap
  • Crawlers Setup
  • Allowable Index for blog
  • Do not index the page Magento admin
  • Do not index the general technical Magento directory
  • Disallow: /js/
  • Disallow: /media/catalog/
  • Disallow: /media/css/
  • Disallow: /media/css_secure/
  • Disallow: /media/js/
  • Disallow: /skin/
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl seach pages and not-SEO optimized catalog links
  • Paths (clean URLs) & Do not index the page checkout and user account
  • Do not crawl common Magento files
  • Do not crawl as per current development files
  • Magento 1.5+
  • Do not crawl sub category pages that are sorted or filtered & QueryStrings
  • previewAction
  • wordpress-robots
  • WP-Print plugin
  • Disallow: /blog/*?
  • Disallow: /blog/wp-content/plugins/
  • Disallow: /feed/
  • Disallow: */feed/
  • Disallow: /comments/
  • Disallow: */comments/
  • Disallow: /trackback/
  • Disallow: */trackback/
  • Disallow: /blog/wp-includes/
  • Disallow: /blog/wp-content/themes/
  • Disallow: /blog/wp-content/plugins/
  • Crawlers Competition page
  • Do not index CMS pages
  • Disallow: /black-friday
  • Products removed from Google search
  • Some bots are known to be trouble, particularly those designed to copy
  • entire sites or download them for offline viewing.
  • Xenu Crawler
  • W3C Crawler
  • LookSmart Crawler
  • Hits many times per second, not acceptable
  • Crawlers that are kind enough to obey, but which we'd rather not have
  • unless they're feeding search engines.
  • A capture bot, downloads gazillions of pages with no public benefit
Back to top

Warnings

  • 3 invalid lines.
Back to top