Log In Sign Up

metalbandt-shirt.com
robots.txt

Robots Exclusion Standard data for metalbandt-shirt.com

Resource Scan

Scan Details

Site Domain metalbandt-shirt.com
Base Domain metalbandt-shirt.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-10-01T14:34:41+00:00
Next Scan 2024-12-30T14:34:41+00:00

Last Successful Scan

Scanned2024-02-12T13:18:33+00:00
URL https://metalbandt-shirt.com/robots.txt
Domain IPs 188.132.220.19
Response IP 188.132.220.19
Found Yes
Hash 26295e53e15a56cb9d6290977f159c72e19e2556d110b6f70a984182a1aecd5c
SimHash 5394f5526aa2

Groups

*

Rule Path
Allow .js
Allow .css
Allow .jpg
Disallow /search/
Disallow /wp-login.php
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /my-account/lost-password/
Disallow /my-account/edit-account/

Other Records

Field Value
crawl-delay 20

yandex

Rule Path
Allow /

yandexbot

Rule Path
Allow /

yandexmobilebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot

Rule Path
Allow /

googlebot-image/1.0

Rule Path
Allow /

bingbot

Rule Path
Allow /

www.80legs.com

Rule Path
Disallow /

acoonbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider/2.0

Rule Path
Disallow /

barkrowler/0.7 (+http://www.exensa.com/crawl)

Rule Path
Disallow /

beetlebot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

blexbot/1.0

Rule Path
Disallow /

booglebot

Rule Path
Disallow /

booglebot2

Rule Path
Disallow /

booglebot 2.0

Rule Path
Disallow /

bubing

Rule Path
Disallow /

businessdbbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

ccbot/2.0

Rule Path
Disallow /

centurybot9@gmail.com

Rule Path
Disallow /

comodospider

Rule Path
Disallow /

comodospider/nutch-1.2

Rule Path
Disallow /

crawler

Rule Path
Disallow /

crawler.feedback+wc@gmail.com

Rule Path
Disallow /

crawler (crawler.feedback@gmail.com)

Rule Path
Disallow /

cyotekwebcopy/1.0

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

ebibot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

flipboard

Rule Path
Disallow /

flipboardproxy

Rule Path
Disallow /

go-http-client/1.1

Rule Path
Disallow /

icarus

Rule Path
Disallow /

icarus

Rule Path
Disallow /

icarus6

Rule Path
Disallow /

icarus6

Rule Path
Disallow /

icarus6j

Rule Path
Disallow /

influencebo

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

istellabot/t.1

Rule Path
Disallow /

istellabot/t.1.13

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

knelson

Rule Path
Disallow /

knelson/0.9

Rule Path
Disallow /

linguee

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

lcc

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mj12bot/v1.0.5

Rule Path
Disallow /

nerdbynature.bot

Rule Path
Disallow /

niki-bot

Rule Path
Disallow /

paracrawl

Rule Path
Disallow /

pcore-http

Rule Path
Disallow /

pcore-http/v0.24.5

Rule Path
Disallow /

psbot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

rogerbot/1.0

Rule Path
Disallow /

scalaj-http/1.0

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

scrapy/1.5.0

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot/2~bl

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

sougou+web+spider

Rule Path
Disallow /

spbot

Rule Path
Disallow /

superfeedr

Rule Path
Disallow /

swebot

Rule Path
Disallow /

swebot

Rule Path
Disallow /

test crawl

Rule Path
Disallow /

testitest1

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

vegi bot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

velenpublicwebcrawler (velen.io)

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

wotbox/2.01

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /
Back to top

Comments

  • Allow Directives
  • Private pages
  • Directories
  • Files
Back to top

Warnings

  • 12 invalid lines.
Back to top