Log In Sign Up

tfnclondon.com
robots.txt

Robots Exclusion Standard data for tfnclondon.com

Resource Scan

Scan Details

Site Domain tfnclondon.com
Base Domain tfnclondon.com
Scan Status Ok
Last Scan2025-04-16T15:48:36+00:00
Next Scan 2025-05-16T15:48:36+00:00

Last Scan

Scanned2025-04-16T15:48:36+00:00
URL https://tfnclondon.com/robots.txt
Redirect https://www.tfnclondon.com/robots.txt
Redirect Domain www.tfnclondon.com
Redirect Base tfnclondon.com
Domain IPs 104.21.68.47, 172.67.186.168, 2606:4700:3034::6815:442f, 2606:4700:3035::ac43:baa8
Redirect IPs 104.21.68.47, 172.67.186.168, 2606:4700:3034::6815:442f, 2606:4700:3035::ac43:baa8
Response IP 104.21.68.47
Found Yes
Hash 911f6403d50eaa833442a61956dfb442abd8a89c487c92370d5adf8adb1d7b0e
SimHash be16d8098ff1

Groups

updownerbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

*

Rule Path Comment
Allow /*?p= -
Allow /index.php/blog/ -
Allow /catalog/seo_sitemap/category/ -
Allow /media/catalog/ -
Disallow /404/ -
Disallow /app/ -
Disallow /cgi-bin/ -
Disallow /downloader/ -
Disallow /errors/ -
Disallow /includes/ -
Disallow /magento/ -
Disallow /pkginfo/ -
Disallow /report/ -
Disallow /scripts/ -
Disallow /shell/ -
Disallow /stats/ -
Disallow /var/ -
Disallow /sitemap/ctl/ -
Disallow /sitemap/hs360/ -
Disallow /sitemap/miabode/ -
Disallow /sitemap/robotsctl/ -
Disallow /sitemap/robotshs360/ -
Disallow /sitemap/robotsmiabode/ -
Disallow /index.php/ -
Disallow /catalog/product_compare/ -
Disallow /catalog/category/view/ -
Disallow /catalog/product/view/ -
Disallow /catalog/product/gallery/ -
Disallow /catalogsearch/ -
Disallow /checkout/ -
Disallow /control/ -
Disallow /contacts/ -
Disallow /*customer/ -
Disallow /customize/ -
Disallow /newsletter/ -
Disallow /poll/ -
Disallow /review/ -
Disallow /sendfriend/ -
Disallow /tag/ -
Disallow /wishlist/ -
Disallow /cron.php -
Disallow /cron.sh -
Disallow /error_log -
Disallow /install.php -
Disallow /LICENSE.html -
Disallow /LICENSE.txt -
Disallow /LICENSE_AFL.txt -
Disallow /STATUS.txt -
Disallow /get.php Magento 1.5+
Disallow /*?p=* -
Disallow /*%26p%3D* -
Disallow /*?price=* -
Disallow /*%26price%3D* -
Disallow /*?color=* -
Disallow /*%26color%3D* -
Disallow /*?c2c_colour=* -
Disallow /*%26c2c_colour%3D* -
Disallow /*?size=* -
Disallow /*%26size%3D* -
Disallow /*?dressshape=* -
Disallow /*%26dressshape%3D* -
Disallow /*?dresslength=* -
Disallow /*%26dresslength%3D* -
Disallow /*?limit=* -
Disallow /*%26limit%3D* -
Disallow /*?mode=* -
Disallow /*%26mode%3D* -
Disallow /*?order=* -
Disallow /*%26order%3D* -
Disallow /*?dir=* -
Disallow /*%26dir%3D* -
Disallow /*.php$ -
Disallow /*?SID= -
Disallow /*%26SID%3D* -
Disallow /?___from_store= -
Disallow /*___from_store%3D -
Disallow /rss* -
Disallow /*PHPSESSID -
Back to top

Other Records

Field Value
sitemap https://www.tfnclondon.com/sitemap.xml
Back to top

Comments

  • $Id: robots.txt,v magento-specific 2010/28/01Â 18:24:19 goba Exp $
  • Edited: 2013/08/14 ECIT Added Line 51 - Disallow: /*shopby
  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these “robots” where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/wc/robots.html
  • For syntax checking, see:
  • http://www.sxw.org.uk/computing/robots/check.html
  • Website Sitemap
  • Crawlers Setup
  • Allowable Index
  • Mind that Allow is not an official standard
  • Allow: /catalogsearch/result/
  • Directories
  • Disallow: /js/
  • Disallow: /lib/
  • Disallow: /media/
  • Disallow: /skin/
  • Paths (clean URLs)
  • Files
  • Paths (no clean URLs)
  • Disallow: /*.js$ So that Google Webmaster Tools can render page effectively 24032015
  • Disallow: /*.css$ So that Google Webmaster Tools can render page effectively 24032015
Back to top