Log In Sign Up

sidmouth.play-cricket.com
robots.txt

Robots Exclusion Standard data for sidmouth.play-cricket.com

Resource Scan

Scan Details

Site Domain sidmouth.play-cricket.com
Base Domain play-cricket.com
Scan Status Ok
Last Scan2024-10-21T21:45:29+00:00
Next Scan 2024-11-20T21:45:29+00:00

Last Scan

Scanned2024-10-21T21:45:29+00:00
URL https://sidmouth.play-cricket.com/robots.txt
Domain IPs 107.162.141.113
Response IP 107.162.141.113
Found Yes
Hash dc07151730e8b604250d2f5d37ad06eaee03137d5351eb8bba1b0642f0365886
SimHash 073ff3c367d5

Groups

*

Rule Path
Disallow /website/player_stats_widget/batting_stats/
Disallow /website/player_stats_widget/bowling_stats/
Disallow /website/player_stats_widget/fielding_stats/
Disallow /website/player_stats_for_season/
Disallow /website/batting_stats_widget/render_html/
Disallow /website/bowling_stats_widget/render_html/
Disallow /wp-admin/
Disallow /*.php$

mediapartners-google

Rule Path
Disallow /

googlebot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 15

adsbot-google

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow /

googlebot-mobile

Rule Path
Disallow

Other Records

Field Value
crawl-delay 15

msnbot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

bingbot/2.0

Rule Path
Disallow /

slurp

Rule Path
Disallow /

yahoo! slurp

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

acoon

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

asterias

Rule Path
Disallow /

backdoorbot/1.0

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

binlar_2.6.3

Rule Path
Disallow /

black hole

Rule Path
Disallow /

botalot

Rule Path
Disallow /

builtbottough

Rule Path
Disallow /

bullseye

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

cegbfeieh

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

crescent

Rule Path
Disallow /

dcpbot/1.2

Rule Path
Disallow /

discobot

Rule Path
Disallow /

disco pump 3.1

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

garlikcrawler/1.2

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

erocrawler

Rule Path
Disallow /

exabot

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

hailoo news crawler

Rule Path
Disallow /

garlikcrawler/1.2

Rule Path
Disallow /

harvest/1.5

Rule Path
Disallow /

humanlinks

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

infonavirobot

Rule Path
Disallow /

jennybot

Rule Path
Disallow /

kenjin spider

Rule Path
Disallow /

lexibot

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

ltbot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

mata hari

Rule Path
Disallow /

miixpc

Rule Path
Disallow /

mister pix

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

moget

Rule Path
Disallow /

nerdbynature.bot

Rule Path
Disallow /

netants

Rule Path
Disallow /

netattache

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

netseer crawler/2.0

Rule Path
Disallow /

openindexshallowspider

Rule Path
Disallow /

openindexspider

Rule Path
Disallow /

propowerbot/2.14

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

quickobot/quickobot-1

Rule Path
Disallow /

repomonkey

Rule Path
Disallow /

rexyobot

Rule Path
Disallow /

seexie.com_bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

spankbot

Rule Path
Disallow /

spanner

Rule Path
Disallow /

spbot

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

superbot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

swebot

Rule Path
Disallow /

szukacz/1.4

Rule Path
Disallow /

tbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /

gwadoobot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

timpibot/0.9 (+http://www.timpi.io)

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

zoominfobot (zoominfobot at zoominfo dot com)

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

serpstatbot/2.1

Rule Path
Disallow /

*

Rule Path
Disallow /
Back to top

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • Allow only major search spiders
Back to top