Log In Sign Up

162.31.192.35.bc.googleusercontent.com
robots.txt

Robots Exclusion Standard data for 162.31.192.35.bc.googleusercontent.com

Resource Scan

Scan Details

Site Domain 162.31.192.35.bc.googleusercontent.com
Base Domain googleusercontent.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonRequest timed out.
Last Scan2024-09-16T12:16:34+00:00
Next Scan 2024-12-15T12:16:34+00:00

Last Successful Scan

Scanned2023-05-01T04:19:19+00:00
URL http://162.31.192.35.bc.googleusercontent.com/robots.txt
Domain IPs 35.192.31.162
Response IP 35.192.31.162
Found Yes
Hash fe47f7786892be30985eb5da6809ba65a9267dd6c8805d3d5ec5c6e3239c121d
SimHash 888eaf7366f1

Groups

yandexbot

Rule Path
Allow /*?p=
Disallow /*?p=*&
Disallow /*?

mediapartners-google

Rule Path
Disallow /

adsbot-google

Rule Path
Disallow /

googlebot

Rule Path
Disallow /nogooglebot/
Allow /*?p=
Disallow /files/
Disallow /links/
Disallow /samples/
Disallow /*.pdf$
Disallow /404/
Disallow /app/
Disallow /api/
Disallow /api
Disallow /cgi-bin/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /magento/
Disallow /media/
Disallow /downloadable/
Disallow /media/captcha/
Disallow /media/catalog/
Disallow /media/customer/
Disallow /media/dhl/
Disallow /media/downloadable/
Disallow /media/import/
Disallow /media/pdf/
Disallow /media/sales/
Disallow /media/tmp/
Disallow /media/wysiwyg/
Disallow /media/xmlconnect/
Disallow /pkginfo/
Disallow /report/
Disallow /scripts/
Disallow /shell/
Disallow /stats/
Disallow /var/
Disallow */index.php/
Disallow */catalog/product_compare/
Disallow */catalog/category/view/
Disallow */catalog/product/view/
Disallow */catalog/product/gallery/
Disallow */catalogsearch/
Disallow */control/
Disallow */contacts/
Disallow */customer/
Disallow */media/
Disallow */downloadable/
Disallow */links/
Disallow */customize/
Disallow */newsletter/
Disallow */poll/
Disallow */review/
Disallow */sendfriend/
Disallow */tag/
Disallow */wishlist/
Disallow */checkout/
Disallow */onestepcheckout/
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /STATUS.txt
Disallow /*?dir*
Disallow /*?limit*
Disallow /*?mode*
Disallow /*?___from_store=*
Disallow /*?___store=*
Disallow /*?cat=*
Disallow /*?q=*
Disallow /*?price=*
Disallow /*?availability=*
Disallow /*?brand=*
Disallow /phpinfo.php
Disallow /README.txt
Disallow /api.php
Disallow /*?p=*&
Disallow /*.php$
Disallow /*.pdf$
Disallow /*?SID=
Disallow /rss*
Disallow /*PHPSESSID
Disallow /*js$
Disallow /*.css$
Disallow /*.pdf$
Back to top

Other Records

Field Value
sitemap https://zeero-s.com/sitemap.xml
Back to top

Comments

  • Sitemap: https://zeero-s.com/sitemap.xml
  • Google Image Crawler Setup - having crawler-specific sections makes it ignore generic e.g *
  • User-agent: Googlebot-Image
  • Disallow:
  • User-agent: Googlebot
  • Disallow:
  • Yandex tends to be rather aggressive, may be worth keeping them at arms lenght
  • Crawl-delay: 20
  • Problem is mostly related to layered nav and query params, allow only paging
  • Crawlers Setup
  • User-agent: *
  • User-agent: Googlebot
  • Disallow: /
  • Allow paging (unless paging inside a listing with more params, as disallowed below)
  • Directories
  • Disallow: /skin/
  • Paths (if using shop id in URL must prefix with * or copy for each)
  • Files
  • Do not crawl sub category pages that are sorted or filtered.
  • This would be very broad, could hurt (incl. SEO).
  • Disallow: /*?*
  • These are more specific, pick what you need - and do not forget to add your custom filters!
  • Paths that can be safely ignored (no clean URLs)
Back to top