Log In Sign Up

wallingtonplumbingsupply.com
robots.txt

Robots Exclusion Standard data for wallingtonplumbingsupply.com

Resource Scan

Scan Details

Site Domain wallingtonplumbingsupply.com
Base Domain wallingtonplumbingsupply.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-20T07:53:58+00:00
Next Scan 2024-11-03T07:53:58+00:00

Last Successful Scan

Scanned2024-07-12T07:51:46+00:00
URL https://wallingtonplumbingsupply.com/robots.txt
Redirect https://www.wallingtonplumbingsupply.com/robots.txt
Redirect Domain www.wallingtonplumbingsupply.com
Redirect Base wallingtonplumbingsupply.com
Domain IPs 143.204.215.129, 143.204.215.23, 143.204.215.77, 143.204.215.9
Redirect IPs 54.230.112.126, 54.230.112.26, 54.230.112.31, 54.230.112.59
Response IP 3.160.246.92
Found Yes
Hash 6e758937ea7898c5ae80594aef38511451a3ca6c45da966669f2506918169a73
SimHash ad30771746f1

Groups

siteauditbot

Rule Path
Allow /

semrushbot

Rule Path
Allow /

semrushbot-sa

Rule Path
Allow /

semrushbot-si

Rule Path
Allow /

semrushbot-ba

Rule Path
Allow /

semrushbot-swa

Rule Path
Allow /

semrushbot-ct

Rule Path
Allow /

semrushbot-bm

Rule Path
Allow /

semrushbot-seoab

Rule Path
Allow /

siteauditbot

Rule Path
Allow /

rogerbot

Rule Path
Disallow /CVS
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$
Disallow /wpsadm/
Disallow /app/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /pkginfo/
Disallow /shell/
Disallow /var/
Disallow /api.php
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /get.php
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /*?SID=
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php

googlebot-image
msnbot-media
googlebot

Rule Path
Disallow /*.php$
Disallow /report/
Disallow /catalog/
Disallow /sendfriend/
Disallow /review/
Disallow /control/
Disallow /contacts/
Disallow /customize/
Disallow /newsletter/
Disallow /poll/
Disallow /tag/
Disallow /wishlist/
Disallow /composer.json
Disallow /composer.lock
Disallow /CONTRIBUTING.md
Disallow /CONTRIBUTOR_LICENSE_AGREEMENT.html
Disallow /COPYING.txt
Disallow /Gruntfile.js
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /nginx.conf.sample
Disallow /package.json
Disallow /php.ini.sample
Disallow /RELEASE_NOTES.txt
Disallow /*?*product_list_mode=
Disallow /*?*product_list_order=
Disallow /*?*product_list_limit=
Disallow /*?*product_list_dir=

*

Rule Path
Disallow /CVS
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$
Disallow /app/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /pkginfo/
Disallow /shell/
Disallow /var/
Disallow /api.php
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /get.php
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /*?SID=
Disallow /checkout/
Disallow /onestepcheckout/
Disallow /customer/
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /catalogsearch/
Disallow /catalog/product_compare/
Disallow /catalog/category/view/
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php

googlebot-mobile

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://www.wallingtonplumbingsupply.com/sitemap/sitemap.xml
sitemap https://www.wallingtonplumbingsupply.com/pub/sitemap/sitemap-1-1.xml
sitemap https://www.wallingtonplumbingsupply.com/pub/sitemap/sitemap-1-2.xml
sitemap https://www.wallingtonplumbingsupply.com/sitemap/sitemap.xml
sitemap https://www.wallingtonplumbingsupply.com/pub/sitemap/sitemap-1-1.xml
sitemap https://www.wallingtonplumbingsupply.com/pub/sitemap/sitemap-1-2.xml
sitemap https://www.wallingtonplumbingsupply.com/pub/sitemap/sitemap.xml
Back to top

Comments

  • robots.txt for Magento Community and Enterprise
  • GENERAL SETTINGS
  • Enable robots.txt rules for all crawlers
  • Disallow: /
  • Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
  • Set a custom crawl rate if you're experiencing traffic problems with your server.
  • Crawl-delay: 30
  • Magento sitemap: uncomment and replace the URL to your Magento sitemap file
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • GENERAL MAGENTO SETTINGS
  • Do not crawl Magento admin page
  • Do not crawl common Magento technical folders
  • Do not crawl common Magento files
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Disallow: /index.php/
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl seach pages and not-SEO optimized catalog links
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
  • IMAGE CRAWLERS SETTINGS
  • Extra: Uncomment if you do not wish Google and Bing to index your images
  • Disallow: /
  • Disallow: /
  • Google Merchanct Center Bots
  • Disallow: /
  • Files
  • Do not index pages that are sorted or filtered.
  • Disallow: /
  • Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
  • Set a custom crawl rate if you're experiencing traffic problems with your server.
  • Crawl-delay: 30
  • Magento sitemap: uncomment and replace the URL to your Magento sitemap file
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • GENERAL MAGENTO SETTINGS
  • Do not crawl common Magento technical folders
  • Do not crawl common Magento files
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Disallow: /index.php/
  • Do not crawl links with session IDs
  • Do not crawl checkout and user account pages
  • Do not crawl search pages and not-SEO optimized catalog links
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
Back to top