gipermarket-gorod-dverej.ua
robots.txt

Robots Exclusion Standard data for gipermarket-gorod-dverej.ua

Resource Scan

Scan Details

Site Domain gipermarket-gorod-dverej.ua
Base Domain gipermarket-gorod-dverej.ua
Scan Status Ok
Last Scan2024-11-11T11:46:14+00:00
Next Scan 2024-12-11T11:46:14+00:00

Last Scan

Scanned2024-11-11T11:46:14+00:00
URL https://gipermarket-gorod-dverej.ua/robots.txt
Domain IPs 91.238.103.16
Response IP 91.238.103.16
Found Yes
Hash f2eacaf6ecb314ab290ae30c025adbfb4234f2fc985f94afca89d80a342a7059
SimHash c727b9794399

Groups

*

Rule Path Comment
Allow /index.php/blog/ -
Allow /catalog/seo_sitemap/category/ -
Allow /media/catalog/ -
Disallow /testimonials/* -
Disallow /testimonials/ -
Disallow /testimonials -
Disallow /404/ -
Disallow /app/ -
Disallow /cgi-bin/ -
Disallow /downloader/ -
Disallow /errors/ -
Disallow /includes/ -
Disallow /lib/ -
Disallow /magento/ -
Disallow /media/captcha/ -
Disallow /media/css_secure/ -
Disallow /media/customer/ -
Disallow /media/dhl/ -
Disallow /media/downloadable/ -
Disallow /media/import/ -
Disallow /media/js/ -
Disallow /media/pdf/ -
Disallow /media/sales/ -
Disallow /media/tmp/ -
Disallow /media/xmlconnect/ -
Disallow /pkginfo/ -
Disallow /report/ -
Disallow /scripts/ -
Disallow /shell/ -
Disallow /stats/ -
Disallow /var/ -
Disallow /index.php/ -
Disallow /catalog/product_compare/ -
Disallow /catalog/category/view/ -
Disallow /catalog/product/view/ -
Disallow /catalog/product/gallery/ -
Disallow /catalogsearch/* -
Disallow /catalogsearch/result/* -
Disallow /checkout/ -
Disallow /control/ -
Disallow /customer/ -
Disallow /customize/ -
Disallow /newsletter/ -
Disallow /poll/ -
Disallow /review/ -
Disallow /testimonials/ -
Disallow /sendfriend/ -
Disallow /tag/ -
Disallow /wishlist/ -
Disallow /cron.php -
Disallow /cron.sh -
Disallow /error_log -
Disallow /install.php -
Disallow /LICENSE.html -
Disallow /LICENSE.txt -
Disallow /LICENSE_AFL.txt -
Disallow /STATUS.txt -
Disallow /get.php Magento 1.5+
Disallow /*?p=* -
Disallow /*%26p%3D* -
Disallow /*?price=* -
Disallow /*%26price%3D* -
Disallow /*?color=* -
Disallow /*%26color%3D* -
Disallow /*?limit=* -
Disallow /*%26limit%3D* -
Disallow /*?mode=* -
Disallow /*%26mode%3D* -
Disallow /*?order=* -
Disallow /*%26order%3D* -
Disallow /*?dir=* -
Disallow /*%26dir%3D* -
Disallow /*?cat=* -
Disallow /*%26cat%3D* -
Disallow /*?construct* -
Disallow /*%26construct* -
Disallow /*.php$ -
Disallow /*?SID= -
Disallow /?___from_store= -
Disallow /*___from_store%3D -
Disallow /*?gclid=* -
Disallow /rss* -
Disallow /*PHPSESSID -
Disallow /credit?* -
Disallow */limit/* -
Disallow */order/* -
Disallow */dir/* -
Disallow */mode/* -

adsbot-google

Rule Path
Allow /*?gclid=*

adsbot-google-mobile

Rule Path
Allow /*?gclid=*

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

exabot

Rule Path
Disallow /

idbot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-news

Rule Path
Disallow /

baiduspider-favo

Rule Path
Disallow /

baiduspider-cpro

Rule Path
Disallow /

baiduspider-ads

Rule Path
Disallow /

baidu

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

flipboardproxy

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://gipermarket-gorod-dverej.ua/sitemap.xml

Comments

  • ~~ Crawlers Setup
  • Allowable Index
  • Mind that Allow is not an official standard
  • Allow: /*?p=
  • Directories
  • Disallow: /js/
  • Disallow: /media/
  • Disallow: /media/catalog/
  • Disallow: /media/css/
  • Disallow: /skin/
  • Paths (clean URLs)
  • Files
  • Paths (no clean URLs)
  • Disallow: /*?manufacturer*
  • Disallow: /*&manufacturer*
  • Disallow: /*?typedoor*
  • Disallow: /*&typedoor*
  • Disallow: /*?material*
  • Disallow: /*&material*
  • Disallow: /filter?*
  • Disallow: /*.js$
  • Disallow: /*.css$
  • Disallow: */p/*
  • Disallow: /ua/content/
  • Disallow: /ua/

Warnings

  • `host` is not a known field.