Log In Sign Up

metroshoes.net
robots.txt

Robots Exclusion Standard data for metroshoes.net

Resource Scan

Scan Details

Site Domain metroshoes.net
Base Domain metroshoes.net
Scan Status Ok
Last Scan2025-03-24T07:17:58+00:00
Next Scan 2025-04-23T07:17:58+00:00

Last Scan

Scanned2025-03-24T07:17:58+00:00
URL https://metroshoes.net/robots.txt
Redirect https://www.metroshoes.com/robots.txt
Redirect Domain www.metroshoes.com
Redirect Base metroshoes.com
Domain IPs 104.26.10.75, 104.26.11.75, 172.67.70.177, 2606:4700:20::681a:a4b, 2606:4700:20::681a:b4b, 2606:4700:20::ac43:46b1
Redirect IPs 104.26.0.244, 104.26.1.244, 172.67.68.136, 2606:4700:20::681a:1f4, 2606:4700:20::681a:f4, 2606:4700:20::ac43:4488
Response IP 104.26.0.244
Found Yes
Hash 11a3706b6c385d37b39770a6e15fb7f0771d0d358100b9252d7cf74f3158e455
SimHash 636173534673

Groups

*

Rule Path
Allow /
Disallow /*?q=
Disallow /*?filter=
Disallow /CVS
Disallow /*.svn$
Disallow /*.idea$
Disallow /*.sql$
Disallow /*.tgz$
Disallow /admin/
Disallow /metro-admin/
Disallow /metroadmin/
Disallow /app/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /lib/
Disallow /pkginfo/
Disallow /shell/
Disallow /var/
Disallow /api.php
Disallow /cron.php
Disallow /cron.sh
Disallow /error_log
Disallow /get.php
Disallow /install.php
Disallow /LICENSE.html
Disallow /LICENSE.txt
Disallow /LICENSE_AFL.txt
Disallow /README.txt
Disallow /RELEASE_NOTES.txt
Disallow /*?dir*
Disallow /*?dir=desc
Disallow /*?dir=asc
Disallow /*?limit=all
Disallow /*?mode*
Disallow /*?filter*
Disallow /*?SID=
Disallow /*search
Disallow /*searchtap
Disallow /search-results*
Disallow */checkout*
Disallow /onestepcheckout/
Disallow /customer*
Disallow /customer/account/
Disallow /customer/account/login/
Disallow /onepagecheckout
Disallow */cart*
Disallow /cgi-bin/
Disallow /cleanup.php
Disallow /apc.php
Disallow /memcache.php
Disallow /phpinfo.php

googlebot-image

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

blackwidow

Rule Path
Disallow /

chinaclaw

Rule Path
Disallow /

custo

Rule Path
Disallow /

discobot

Rule Path
Disallow /

disco

Rule Path
Disallow /

download\ demon

Rule Path
Disallow /

ecatch

Rule Path
Disallow /

eirgrabber

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

express\ webpictures

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

eyenetie

Rule Path
Disallow /

flashget

Rule Path
Disallow /

getright

Rule Path
Disallow /

getweb\!

Rule Path
Disallow /

go\!zilla

Rule Path
Disallow /

go\-ahead\-got\-it

Rule Path
Disallow /

grabnet

Rule Path
Disallow /

grafula

Rule Path
Disallow /

harvest

Rule Path
Disallow /

httrack

Rule Path
Disallow /

image\ stripper

Rule Path
Disallow /

image\ sucker

Rule Path
Disallow /

indy\ library

Rule Path
Disallow /

interget

Rule Path
Disallow /

jetcar

Rule Path
Disallow /

leechftp

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

lwp\-trivial

Rule Path
Disallow /

midown\ tool

Rule Path
Disallow /

mister\ pix

Rule Path
Disallow /

navroad

Rule Path
Disallow /

nearsite

Rule Path
Disallow /

netants

Rule Path
Disallow /

netspider

Rule Path
Disallow /

netzip

Rule Path
Disallow /

nutch

Rule Path
Disallow /

octopus

Rule Path
Disallow /

pagegrabber

Rule Path
Disallow /

pavuk

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

r6_feedfetcher

Rule Path
Disallow /

realdownload

Rule Path
Disallow /

reget

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

smartdownload

Rule Path
Disallow /

superbot

Rule Path
Disallow /

superhttp

Rule Path
Disallow /

surfbot

Rule Path
Disallow /

takeout

Rule Path
Disallow /

teleport\ pro

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

voideye

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webfetch

Rule Path
Disallow /

webgo\ is

Rule Path
Disallow /

webleacher d

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

websauger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webwhacker

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wget

Rule Path
Disallow /

widow

Rule Path
Disallow /

wwwoffle

Rule Path
Disallow /

xaldon\ webspider

Rule Path
Disallow /

zeus

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

bot-traffic

Rule Path
Disallow /

bottraffic

Rule Path
Disallow /

trafficbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.metroshoes.com/sitemap.xml
Back to top

Comments

  • robots.txt for Magento Community and Enterprise
  • GENERAL SETTINGS
  • Enable robots.txt rules for all crawlers
  • Crawl-delay parameter: number of seconds to wait between successive requests to the same server.
  • Set a custom crawl rate if you're experiencing traffic problems with your server.
  • Crawl-delay: 30
  • Magento sitemap: uncomment and replace the URL to your Magento sitemap file
  • DEVELOPMENT RELATED SETTINGS
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • GENERAL MAGENTO SETTINGS
  • Do not crawl Magento admin page
  • Do not crawl common Magento technical folders
  • Do not crawl common Magento files
  • MAGENTO SEO IMPROVEMENTS
  • Do not crawl sub category pages that are sorted or filtered.
  • Do not crawl 2-nd home page copy (example.com/index.php/). Uncomment it only if you activated Magento SEO URLs.
  • Disallow: /index.php/
  • Do not crawl links with session IDs
  • Do not crawl links with search term
  • Do not crawl checkout and user account pages
  • Do not crawl seach pages and not-SEO optimized catalog links
  • SERVER SETTINGS
  • Do not crawl common server technical folders and files
  • IMAGE CRAWLERS SETTINGS
  • User-agent: msnbot-media
  • Disallow: /
  • BOTS SETTINGS
Back to top