Log In Sign Up

titaneyeplus.com
robots.txt

Robots Exclusion Standard data for titaneyeplus.com

Resource Scan

Scan Details

Site Domain titaneyeplus.com
Base Domain titaneyeplus.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-06-23T13:22:31+00:00
Next Scan 2024-07-07T13:22:31+00:00

Last Successful Scan

Scanned2023-01-08T06:42:54+00:00
URL https://titaneyeplus.com/robots.txt
Redirect https://www.titaneyeplus.com/robots.txt
Redirect Domain www.titaneyeplus.com
Redirect Base titaneyeplus.com
Domain IPs 125.56.228.18, 2600:1413:1:986::2942, 2600:1413:1:992::2942
Redirect IPs 125.56.228.18, 2600:1413:b000:381::2942, 2600:1413:b000:394::2942
Response IP 104.69.162.70
Found Yes
Hash 3208eeaaa7ff8374f347033bf0271c0443bf8e0b0914f9894d802b9475243430
SimHash ac06bf7366f1

Groups

googlebot-image

Rule Path
Disallow
Allow /*?p=
Disallow /*?p=*&
Disallow /*?

*

Rule Path
Allow /*?p=
Disallow /offers.html
Disallow /404/
Disallow /app/
Disallow /cgi-bin/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /magento/
Disallow /media/captcha/
Disallow /media/customer/
Disallow /media/dhl/
Disallow /media/downloadable/
Disallow /media/import/
Disallow /media/pdf/
Disallow /media/sales/
Disallow /media/tmp/
Disallow /media/xmlconnect/
Disallow /pkginfo/
Disallow /report/
Disallow /scripts/
Disallow /shell/
Disallow /stats/
Disallow /var/
Disallow /search
Disallow */index.php/
Disallow */catalog/product_compare/
Disallow */catalog/category/view/
Disallow */catalog/product/view/
Disallow */catalog/product/gallery/
Disallow */catalogsearch/
Disallow */control/
Disallow */contacts/
Disallow */customer/
Disallow */customize/
Disallow */newsletter/
Disallow */poll/
Disallow */review/
Disallow */sendfriend/
Disallow */tag/
Disallow */wishlist/
Disallow */checkout/
Disallow */onestepcheckout/
Disallow */page/*
Disallow */show/*
Disallow */sort-by/*
Disallow */front-color/*
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /STATUS.txt
Disallow /*?dir*
Disallow /*?limit*
Disallow /*?mode*
Disallow /*?___from_store=*
Disallow /*?___store=*
Disallow /*?cat=*
Disallow /*?q=*
Disallow /*?price=*
Disallow /*?availability=*
Disallow /*?brand=*
Disallow *?lens_technology_sunglasses=*
Disallow *?frame_material=*
Disallow *?collection_type=*
Disallow *tag%3D*
Disallow /*?p=*&
Disallow /*.php$
Disallow /*?SID=
Back to top

Other Records

Field Value
sitemap https://www.titaneyeplus.com/sitemap.xml
Back to top

Comments

  • sitemaps
  • Google Image Crawler Setup
  • Problem is mostly related to layered nav and query params, allow only paging
  • Crawlers Setup
  • Allow paging (unless paging inside a listing with more params, as disallowed below)
  • Directories
  • Disallow: /media/
  • Disallow: /media/catalog/
  • Disallow: /media/wysiwyg/
  • Disallow: /skin/
  • Paths (if using shop id in URL must prefix with * or copy for each)
  • Files
  • Do not crawl sub category pages that are sorted or filtered.
  • This would be very broad, could hurt (incl. SEO).
  • Disallow: /*?*
  • These are more specific, pick what you need - and do not forget to add your custom filters!
  • Paths that can be safely ignored (no clean URLs)
Back to top