nycdiamonddistrict.com
robots.txt

Robots Exclusion Standard data for nycdiamonddistrict.com

Resource Scan

Scanned	2024-06-19T12:20:04+00:00
URL	https://nycdiamonddistrict.com/robots.txt
Domain IPs	18.154.132.13, 18.154.132.38, 18.154.132.51, 18.154.132.96
Response IP	18.165.171.51
Found	Yes
Hash	c752a2aca4ccd5a92f4f3ef2e39c7929761f2c338ecfecc9d997e3e72b9278d2
SimHash	61117371c7b8

No rules defined. All paths allowed.

Back to top

Robots.txt for Magento Community and Enterprise
GENERAL SETTINGS
Enables robots.txt rules for all crawlers
Crawl-delay parameter: the number of seconds you want to wait between successful requests to the same server.
Set a crawl rate, if your server's traffic problems. Please note that Google ignore crawl-delay setting in Robots.txt. You can set up this in Google Webmaster tool
Crawl-delay: 30
Magento sitemap: URL to your sitemap file in Magento
Sitemap: http://www.mystore.com/sitemap/sitemap.xml
Settings that relate to the UNDER CONSTRUCTION
Do not allow indexing files and folders that are required during development: CVS, SVN directory and dump files
GENERAL SETTINGS For MAGENTO
Do not index the page Magento admin
Do not index the general technical Magento directory
Do not index the shared files Magento
MAGENTO SEA IMPROVEMENT
Do not index the page subcategories that are sorted or filtered.
Do not index the second copy of the home page (example.com / index.php /). Un-comment only if you have activated Magento SEO URLs.
Disallow: / index.php /
Do not index the link from the session ID
Do not index the page checkout and user account
Do not index the search page and CEO, non-optimized link categories
Server Settings
Do not index the general technical directories and files on a server
SETTINGS Image indexing
Optional: If you do not want to Google and Bing to index your images
User-agent: Googlebot-Image
Disallow: /
User-agent: msnbot-media
Disallow: /

Back to top

Back to top