Log In Sign Up

advanier.com
robots.txt

Robots Exclusion Standard data for advanier.com

Resource Scan

Scan Details

Site Domain advanier.com
Base Domain advanier.com
Scan Status Ok
Last Scan2024-10-21T17:36:35+00:00
Next Scan 2024-11-20T17:36:35+00:00

Last Scan

Scanned2024-10-21T17:36:35+00:00
URL https://advanier.com/robots.txt
Domain IPs 104.21.90.201, 172.67.161.21, 2606:4700:3031::6815:5ac9, 2606:4700:3031::ac43:a115
Response IP 172.67.161.21
Found Yes
Hash abe0f6e560670d194bf8955b84424923e6b4a6ded0edb7d9176bb5b4a5fe5837
SimHash 8e2669d0c6e2

Groups

googlebot

Rule Path
Disallow *?
Disallow */currency

Other Records

Field Value
crawl-delay 57

*
facebookexternalhit/1.1

Rule Path
Disallow /

bingbot

Rule Path
Disallow *?
Disallow */currency

Other Records

Field Value
crawl-delay 57

meta-externalagent

Rule Path
Disallow /

geedoproductsearch

Rule Path
Disallow /

amazon-kendra-web-crawler-*

Product Comment
amazon-kendra-web-crawler-* all customers of Amazon Kendra's web crawler
Rule Path Comment
Disallow / disallow everything

thesis-research-bot

Rule Path
Disallow /

geedobot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ninjabot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /
Disallow /CVS
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$

baiduspider

Rule Path
Disallow /

baiduspider-imagem

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-news

Rule Path
Disallow /

baiduspider-favo

Rule Path
Disallow /

baiduspider-cpro

Rule Path
Disallow /

baiduspider-anãƒâºncios

Rule Path
Disallow /

baidu

Rule Path
Disallow /

majestic12

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yandex

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /
Disallow /admin/
Disallow /404/
Disallow /app/
Disallow /cgi-bin/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /magento/
Disallow /media/captcha/
Disallow /media/customer/
Disallow /media/dhl/
Disallow /media/downloadable/
Disallow /media/import/
Disallow /media/pdf/
Disallow /media/tmp/
Disallow /media/xmlconnect/
Disallow /pkginfo/
Disallow /report/
Disallow /scripts/
Disallow /shell/
Disallow /stats/
Disallow /var/
Disallow /index.php/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /catalog/product/gallery/
Disallow /catalogsearch/
Disallow /checkout/
Disallow /control/
Disallow /contacts/
Disallow /customer/
Disallow /customize/
Disallow /newsletter/
Disallow /poll/
Disallow /review/
Disallow /sendfriend/
Disallow /tag/
Disallow /wishlist/
Disallow /directory/currency/switch/currency/
Disallow /front-page-products.html
Disallow /customer-service
Disallow /enable-cookies
Disallow /buyshop_404
Disallow /newsletter
Disallow /buyshop_home_advanced
Disallow /privacy-policy
Disallow /buyshop_html
Disallow /product-not-available
Disallow /api.php
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /get.php
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /*?___from_store=*
Disallow /*?cat=*
Disallow /*?q=*
Disallow /*?SID=
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /catalog/product/view/
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php
Disallow /findorders.php
Disallow /*.php$
Disallow /rss*
Disallow /*PHPSESSID
Disallow /nrk/
Disallow /123/
Disallow /123moveis/
Disallow /casasonline/
Disallow /contactlensguide/
Disallow /domainsforsale/
Disallow /labels/
Disallow /logodesign/
Disallow /opticaiberica/
Disallow /opticaiberica_copy/
Disallow /upload/
Disallow /viaturas_copy/
Disallow /*.ico$
Back to top

Comments

  • Crawlers Setup
  • User-agent: *
  • start AI crawler block
  • end AI crawler block
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • Baiduspider
  • GENERAL MAGENTO SETTINGS
  • Do not crawl Magento admin page
  • Directories
  • Disallow: /js/
  • Disallow: /media/
  • Disallow: /media/catalog/
  • Disallow: /media/css/
  • Disallow: /media/css_secure/
  • Disallow: /media/js/
  • Disallow: /media/sales/
  • Disallow: /media/wysiwyg/
  • Disallow: /skin/
  • Paths (clean URLs)
  • NILS
  • Do not crawl common Magento files
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Disallow: /index.php/
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl seach pages and not-SEO optimized catalog links
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
  • IMAGE CRAWLERS SETTINGS
  • Extra: Uncomment if you do not wish Google and Bing to index your images
  • User-agent: Googlebot-Image
  • Disallow: /
  • User-agent: msnbot-media
  • Disallow: /
  • Paths (no clean URLs)
  • Disallow: /*.js$
  • Disallow: /*.css$
  • Nils
Back to top