Log In Sign Up

shermanctweather.org
robots.txt

Robots Exclusion Standard data for shermanctweather.org

Resource Scan

Scan Details

Site Domain shermanctweather.org
Base Domain shermanctweather.org
Scan Status Ok
Last Scan2024-05-12T07:07:32+00:00
Next Scan 2024-06-11T07:07:32+00:00

Last Scan

Scanned2024-05-12T07:07:32+00:00
URL https://shermanctweather.org/robots.txt
Domain IPs 70.32.23.118
Response IP 70.32.23.118
Found Yes
Hash 321160e327ea5f2c80d4331260cb833b567fcb083845f5fbddb67c149be8c84b
SimHash 759f4270d557

Groups

googlebot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 30

applebot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

bingbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Disallow /

qwantify/2.2w

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

privatesearch/0.1.0

Rule Path
Disallow /

speedyspider

Rule Path
Disallow /

semrushbot/1.1~bl

Rule Path
Disallow /

seoscanners.net/1

Rule Path
Disallow /

facebookexternalhit/1.1

Rule Path
Disallow /

semrushbot/1~bl

Rule Path
Disallow /

wotbox/2.01

Rule Path
Disallow /

experibot_v1

Rule Path
Disallow /

ahc/1.0

Rule Path
Disallow /

wonderbot/js 1.0

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

obot/2.3.1

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

spiderbot/nutch-1.7

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

metacommentbot

Rule Path
Disallow /

msnbot-media/1.1

Rule Path
Disallow /

laserlikebot/0.1

Rule Path
Disallow /

www-mechanize/1.74

Rule Path
Disallow /

turn it in

Rule Path
Disallow /

yahoo slurp

Rule Path
Disallow /

yahoo! slurp

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

java

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

y!j yahoo japan

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

livelapbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

archive.org

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

twitterbot/1.0

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

openhosebot

Rule Path
Disallow /

tweetmemebot

Rule Path
Disallow /

google-http-java-client/1.17.0-rc (gzip)

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

facebook bot

Rule Path
Disallow /

duckduckgo-favicons-bot/1.0

Rule Path
Disallow /

mail.ru_bot/2.0

Rule Path
Disallow /

exabot/3.0

Rule Path
Disallow /

riddler

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

safedns search bot/nutch-1.9

Rule Path
Disallow /

omgili/0.5

Rule Path
Disallow /

lipperhey-kaus-australis/5.0

Rule Path
Disallow /

ccbot/2.0

Rule Path
Disallow /

metauri api/2.0

Rule Path
Disallow /

princetonbot/1.0

Rule Path
Disallow /

newsharecounts.com/1.0

Rule Path
Disallow /

veoozbot/1.0

Rule Path
Disallow /

uptimebot/1.0

Rule Path
Disallow /

domainstatsbot/1.0

Rule Path
Disallow /
Back to top

Warnings

  • 2 invalid lines.
Back to top