Log In Sign Up

jerrysartarama.com
robots.txt

Robots Exclusion Standard data for jerrysartarama.com

Resource Scan

Scan Details

Site Domain jerrysartarama.com
Base Domain jerrysartarama.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-02T13:23:52+00:00
Next Scan 2024-12-31T13:23:52+00:00

Last Successful Scan

Scanned2023-02-18T10:52:40+00:00
URL https://jerrysartarama.com/robots.txt
Redirect https://www.jerrysartarama.com/robots.txt
Redirect Domain www.jerrysartarama.com
Redirect Base jerrysartarama.com
Domain IPs 104.17.198.86, 104.17.199.86, 2606:4700::6811:c656, 2606:4700::6811:c756
Redirect IPs 104.17.198.86, 104.17.199.86, 2606:4700::6811:c656, 2606:4700::6811:c756
Response IP 104.17.199.86
Found Yes
Hash 0977195a310f78c83cb5948edce71e7b788cd19b606984aa065c34ea56962022
SimHash a70ca15b87e9

Groups

*
googlebot

Rule Path
Disallow /catalogindex/

googlebot-image

Rule Path
Disallow

adsbot-google

Rule Path Comment
Disallow -
Disallow /search/go -
Disallow /art/*? -
Allow /media/catalog/ -
Allow /media/ -
Disallow /catalogindex/ -
Disallow /catalog/ -
Disallow /catalogindex/search/ -
Disallow /catalog/product_compare/ -
Disallow /catalog/category/view/ -
Disallow /catalog/product/view/ -
Disallow /catalog/product/gallery/ -
Disallow /catalogsearch/ -
Disallow /control/ -
Disallow /coupon-codes/ -
Disallow /special-offers/ -
Disallow /super-sales/ -
Disallow /super-sale/ -
Disallow /surveys/ -
Disallow /index.php/ -
Disallow /quickview/* -
Disallow /quickview/catalog_product/* -
Disallow /jerrysapplications/ -
Disallow /Jerrys-Videos/ -
Disallow /Comm/ -
Disallow /PopUp/ -
Disallow /mobile/ -
Disallow /m/ -
Disallow /coupon-eligible/ -
Disallow /*?* -
Disallow /*?cat -
Disallow /*?dir* -
Disallow /*?dir=desc -
Disallow /*?dir=asc -
Disallow /*?limit=all -
Disallow /*?mode* -
Disallow /*/browse-by -
Disallow /*?size -
Disallow /*?casters -
Disallow /*?is_proprietary -
Disallow /*?ferrule -
Disallow /*?handle -
Disallow /*?hair_type -
Disallow /*?brand_name -
Disallow /*?type -
Disallow /*?shape -
Disallow /*?texture -
Disallow /*?material -
Disallow /*?price -
Disallow /*?limit -
Disallow /*?base_color -
Disallow /*?viewAll -
Disallow /*?p -
Disallow /*?bvrrp* -
Disallow /*?format -
Disallow /*?primer -
Disallow /checkout/* -
Disallow /checkout/index/ -
Disallow /customer/ -
Disallow /customer/account/ -
Disallow /customer/account/login/referer/* -
Disallow /customer/account/login/ -
Disallow /wishlist/ -
Disallow /newsletter/ -
Disallow /poll/ -
Disallow /review/ -
Disallow /write-a-review/ -
Disallow /reviews/ -
Disallow /sendfriend/* -
Disallow /productalert/ -
Disallow /invitation/ -
Disallow /tag/ -
Disallow /giftregistry/ -
Disallow /contest/profile/ -
Disallow /order-status/ -
Disallow /sales/ -
Disallow /giftcard/* -
Disallow /giftcard/customer/ -
Disallow /cybersource/manage/ -
Disallow /favorites/list/ -
Disallow /teacherwishlist/ -
Disallow /ajax/ -
Disallow /amlist/* -
Disallow .csv -
Disallow .svn -
Disallow .idea -
Disallow .sql -
Disallow .tgz -
Disallow .git -
Disallow /admin/ -
Disallow /dev/ -
Disallow /cleanup.php -
Disallow /info.php -
Disallow /404/ -
Disallow /app/ -
Disallow /cgi-bin/ -
Disallow /bin/ -
Disallow /downloader/ -
Disallow /errors/ -
Disallow /includes/ -
Disallow /lib/ -
Disallow /phpserver/ -
Disallow /magento/ -
Disallow /tools/ -
Disallow /*? -
Disallow /?* -
Disallow /catalog/ -
Disallow /media/captcha/ -
Disallow /media/customer/ -
Disallow /media/dhl/ -
Disallow /media/downloadable/ -
Disallow /media/import/ -
Disallow /media/sales/ -
Disallow /media/tmp/ -
Disallow /media/xmlconnect/ -
Disallow /pkginfo/ -
Disallow /report/ -
Disallow /scripts/ -
Disallow /shell/ -
Disallow /stats/ -
Disallow /var/ -
Disallow /cron.php -
Disallow /health_check.php -
Disallow /static.php -
Disallow /info3.php -
Disallow /error_log -
Disallow /install.php -
Disallow /LICENSE.html -
Disallow /LICENSE.txt -
Disallow /LICENSE_AFL.txt -
Disallow /STATUS.txt -
Disallow /get.php Magento 1.5+
Disallow /*.php$ -
Disallow /*?SID= -
Disallow /rss* -
Disallow /*PHPSESSID -

yandexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

bingbot

Rule Path
Disallow /catalogindex/

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

screaming frog seo spider

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.jerrysartarama.com/sitemap_index.xml
sitemap https://www.jerrysartarama.com/fal_sitemap.xml
sitemap https://www.jerrysartarama.com/blog/sitemap_index.xml
sitemap https://www.jerrysartarama.com/sitemap_index.xml
Back to top

Comments

  • Crawlers Setup
  • SLI Search
  • Allowable Index
  • Mind that Allow is not an official standard
  • Do not crawl CMS pages and not-SEO optimized catalog links
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl qty or box options
  • Disallow: /*?opt
  • Do not crawl checkout and user account pages
  • Do not crawl development files and folders: csv, svn directories and dump files
  • Do not crawl admin and dev page
  • Do not crawl common server technical folders and files and directories
  • Disallow: /js/ ##added to fix issues with structured data retrieve by google bot
  • Images and themes- removed media wysiwyg
  • Disallow: /media/css/
  • Disallow: /media/css_secure/
  • Disallow: /media/js/
  • Files
  • Paths (no clean URLs)
  • Disallow: /*.js$
  • Disallow: /*.css$
  • Seo bots
Back to top

Warnings

  • `noindex` is not a known field.
Back to top