Log In Sign Up

aptekastore.com
robots.txt

Robots Exclusion Standard data for aptekastore.com

Resource Scan

Scan Details

Site Domain aptekastore.com
Base Domain aptekastore.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-26T16:48:40+00:00
Next Scan 2024-10-10T16:48:40+00:00

Last Successful Scan

Scanned2024-08-19T16:45:50+00:00
URL https://aptekastore.com/robots.txt
Redirect https://www.aptekastore.com/robots.txt
Redirect Domain www.aptekastore.com
Redirect Base aptekastore.com
Domain IPs 104.21.96.42, 172.67.172.253, 2606:4700:3034::ac43:acfd, 2606:4700:3036::6815:602a
Redirect IPs 104.21.96.42, 172.67.172.253, 2606:4700:3034::ac43:acfd, 2606:4700:3036::6815:602a
Response IP 172.67.172.253
Found Yes
Hash 93c353bd5ac1177250f77fbb9207dca5fca814f39a6215d8d0cd385a318caa06
SimHash 2f46db164661

Groups

*

Rule Path
Disallow *utm
Disallow *openstat
Disallow /*.js$
Disallow /*.css$
Disallow /*.php$
Disallow *?p=1$

blexbot

Rule Path
Disallow /

baiduspider
baiduspider-video
baiduspider-image

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

swebot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

twengabot-discover

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

bender

Rule Path
Disallow /

discobot

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

searchwebengine.net

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

speedy

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

nerdbynature.bot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sindicebot

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

findfiles.net

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

goodzer

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

lemurwebcrawler

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

fast enterprise crawler 6

Rule Path
Disallow /

sensis.com.au web crawler

Rule Path
Disallow /

worio bot heritrix

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /
Disallow /CVS
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$
Disallow /admin/
Disallow /app/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /pkginfo/
Disallow /shell/
Disallow /var/
Disallow /api.php
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /get.php
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow /*?dir*
Disallow /*?mode*
Disallow /*?order*
Disallow /*?limit*
Disallow /index.php/
Disallow /*?SID=*
Disallow /checkout/*
Disallow /onestepcheckout/
Disallow /customer/
Disallow /wishlist/*
Disallow /sendfriend/*
Disallow /customer/account/*
Disallow /catalogsearch/*
Disallow /catalog/product_compare/*
Disallow /catalog/category/view/*
Disallow /catalog/product/view/*
Disallow /quickview/*
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php
Back to top

Other Records

Field Value
sitemap https://www.aptekastore.com/sitemap.xml
sitemap https://www.aptekastore.com/sitemap/sitemap.xml
Back to top

Comments

  • added by SD on 12/30/04 bot causing a lot of CF errors to occur
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • GENERAL MAGENTO SETTINGS
  • Do not crawl Magento admin page
  • Do not crawl common Magento technical folders
  • Do not crawl common Magento files
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl seach pages and not-SEO optimized catalog links
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
  • IMAGE CRAWLERS SETTINGS
  • Extra: Uncomment if you do not wish Google and Bing to index your images
  • User-agent: Googlebot-Image
  • Disallow: /
  • User-agent: msnbot-media
  • Disallow: /
Back to top

Warnings

  • 1 invalid line.
Back to top