Log In Sign Up

diseasesdatabase.com
robots.txt

Robots Exclusion Standard data for diseasesdatabase.com

Resource Scan

Scan Details

Site Domain diseasesdatabase.com
Base Domain diseasesdatabase.com
Scan Status Ok
Last Scan2024-09-14T07:35:53+00:00
Next Scan 2024-10-14T07:35:53+00:00

Last Scan

Scanned2024-09-14T07:35:53+00:00
URL http://diseasesdatabase.com/robots.txt
Domain IPs 51.155.213.230
Response IP 51.155.213.230
Found Yes
Hash c3e974476ed3a4cd181ea59015179813bf6c5ff0841a5c49fdb8c95ec777738d
SimHash 3314d4f08c81

Groups

mediapartners-google*

Rule Path
Disallow /

acoon

Rule Path
Disallow /

acontbot

Rule Path
Disallow /

accelobot

Rule Path
Disallow /

aipbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

boitho.com-dc

Rule Path
Disallow /

bubing

Rule Path
Disallow /

camontspider

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

charlotte

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

deusu

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

freefind

Rule Path
Disallow /

fyberspider

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

gingercrawler

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

kalooga

Rule Path
Disallow /

lcc

Rule Path
Disallow /

mqbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

omniexplorer_bot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

nutch

Rule Path
Disallow /

psbot

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

sdcresearchlabs-testbot

Rule Path
Disallow /

seekbot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

spbot

Rule Path
Disallow /

speedy

Rule Path
Disallow /

semanticdiscovery

Rule Path
Disallow /

electricmonk

Rule Path
Disallow /

suggybot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

wget

Rule Path
Disallow /

voilabot

Rule Path
Disallow /

voyager/1.0

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yetibot

Rule Path
Disallow /

msnbot

Rule Path
Disallow /madin.asp

Other Records

Field Value
crawl-delay 10

teoma

Rule Path
Disallow /madin.asp

Other Records

Field Value
crawl-delay 10

slurp

Rule Path
Disallow /madin.asp

Other Records

Field Value
crawl-delay 10

googlebot

Rule Path
Disallow /madin.asp

*

Rule Path
Disallow /madin.asp
Disallow /snomed/
Disallow /oppass/
Disallow /sieve/
Back to top

Comments

  • Please also comply with robots meta tag directions on the pages.
  • Thank you http://www.diseasesdatabase.com 20240724a
Back to top

Warnings

  • 4 invalid lines.
Back to top