Log In Sign Up

biosciencewriters.com
robots.txt

Robots Exclusion Standard data for biosciencewriters.com

Resource Scan

Scan Details

Site Domain biosciencewriters.com
Base Domain biosciencewriters.com
Scan Status Ok
Last Scan2024-11-03T19:59:53+00:00
Next Scan 2024-12-03T19:59:53+00:00

Last Scan

Scanned2024-11-03T19:59:53+00:00
URL https://biosciencewriters.com/robots.txt
Domain IPs 72.32.4.128
Response IP 72.32.4.128
Found Yes
Hash c7243c52ea39f562eb0b57fde6ab9ece3ec326589649f7226d52751e2f1e86a2
SimHash 69c4407c6a64

Groups

baiduspider

Rule Path
Disallow /WebResource.axd
Disallow /ScriptResource.axd
Disallow /combinescriptshandler.axd
Disallow /AccessDenied.axpx
Disallow /EditingAccount.axpx
Disallow /EditingErrors.axpx
Disallow /FileNotFound.axpx
Disallow /MobileAccess.aspx
Disallow /*Javascript%3A
Disallow /*.css$
Disallow /*.docx$
Disallow /*.doc$
Disallow /*.js$
Disallow /*.php$
Disallow /*.xlsx$
Disallow /*.xls$

Other Records

Field Value
crawl-delay 10

bingbot

Rule Path
Disallow /WebResource.axd
Disallow /ScriptResource.axd
Disallow /combinescriptshandler.axd
Disallow /images/editors/
Disallow /AccessDenied.axpx
Disallow /MobileAccess.aspx
Disallow /EditingAccount.axpx
Disallow /FileNotFound.axpx
Disallow /MobileAccess.aspx
Disallow /*Javascript%3A
Disallow /*.css$
Disallow /*.docx$
Disallow /*.doc$
Disallow /*.js$
Disallow /*.php$
Disallow /*.xlsx$
Disallow /*.xls$

Other Records

Field Value
crawl-delay 10

duckduckbot/1.1

Rule Path
Disallow /WebResource.axd
Disallow /ScriptResource.axd
Disallow /combinescriptshandler.axd
Disallow /AccessDenied.axpx
Disallow /EditingAccount.axpx
Disallow /EditingErrors.axpx
Disallow /FileNotFound.axpx
Disallow /MobileAccess.aspx
Disallow /*Javascript%3A
Disallow /*.css$
Disallow /*.docx$
Disallow /*.doc$
Disallow /*.js$
Disallow /*.php$
Disallow /*.xlsx$
Disallow /*.xls$

Other Records

Field Value
crawl-delay 10

googlebot/2.1

Rule Path
Disallow /WebResource.axd
Disallow /ScriptResource.axd
Disallow /combinescriptshandler.axd
Disallow /images/editors/
Disallow /AccessDenied.axpx
Disallow /MobileAccess.aspx
Disallow /EditingAccount.axpx
Disallow /FileNotFound.axpx
Disallow /MobileAccess.aspx
Disallow /*Javascript%3A
Disallow /*.css$
Disallow /*.docx$
Disallow /*.doc$
Disallow /*.js$
Disallow /*.php$
Disallow /*.xlsx$
Disallow /*.xls$

Other Records

Field Value
crawl-delay 10

sogou web spider/4.0

Rule Path
Disallow /WebResource.axd
Disallow /ScriptResource.axd
Disallow /combinescriptshandler.axd
Disallow /AccessDenied.axpx
Disallow /EditingAccount.axpx
Disallow /EditingErrors.axpx
Disallow /FileNotFound.axpx
Disallow /MobileAccess.aspx
Disallow /*Javascript%3A
Disallow /*.css$
Disallow /*.docx$
Disallow /*.doc$
Disallow /*.js$
Disallow /*.php$
Disallow /*.xlsx$
Disallow /*.xls$

Other Records

Field Value
crawl-delay 10

yahoo! slurp

Rule Path
Disallow /WebResource.axd
Disallow /ScriptResource.axd
Disallow /combinescriptshandler.axd
Disallow /AccessDenied.axpx
Disallow /EditingAccount.axpx
Disallow /EditingErrors.axpx
Disallow /FileNotFound.axpx
Disallow /MobileAccess.aspx
Disallow /*Javascript%3A
Disallow /*.css$
Disallow /*.docx$
Disallow /*.doc$
Disallow /*.js$
Disallow /*.php$
Disallow /*.xlsx$
Disallow /*.xls$

Other Records

Field Value
crawl-delay 10

yeti

Rule Path
Disallow /WebResource.axd
Disallow /ScriptResource.axd
Disallow /combinescriptshandler.axd
Disallow /AccessDenied.axpx
Disallow /EditingAccount.axpx
Disallow /EditingErrors.axpx
Disallow /FileNotFound.axpx
Disallow /MobileAccess.aspx
Disallow /*Javascript%3A
Disallow /*.css$
Disallow /*.docx$
Disallow /*.doc$
Disallow /*.js$
Disallow /*.php$
Disallow /*.xlsx$
Disallow /*.xls$

Other Records

Field Value
crawl-delay 10

adbeat

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

adsbot-google

Rule Path
Disallow /

adsbot-google-mobile

Rule Path
Disallow /

adobeuxtechc4-async/3.0.12

Rule Path
Disallow /

adscanner

Rule Path
Disallow /

adsrvrbot

Rule Path
Disallow /

adstxtcrawlertp/1.2

Rule Path
Disallow /

ahc/2.1

Rule Path
Disallow /

ahc/2.0

Rule Path
Disallow /

ahrefsbot/6.1

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aihitbot/2.9

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

aiohttp/3.6.2

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

alittle client

Rule Path
Disallow /

alittle+client

Rule Path
Disallow /

alphabot

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

amazonadbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

anarchy99

Rule Path
Disallow /

anchorage crawler

Rule Path
Disallow /

anderspinkbot/1.1

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

apache-httpasyncclient

Rule Path
Disallow /

apache-httpclient

Rule Path
Disallow /

apis-google

Rule Path
Disallow /

appengine-google

Rule Path
Disallow /

aqua_products

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

ares/5.0

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

asana/1.4.0

Rule Path
Disallow /

asterias

Rule Path
Disallow /

aspider/3.2.1

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

atomseobot/2.0

Rule Path
Disallow /

awariosmartbot/1.0

Rule Path
Disallow /

axios/1.6.4

Rule Path
Disallow /

axios/0.21.4

Rule Path
Disallow /

axios/0.19.0

Rule Path
Disallow /

axios/0.18.0

Rule Path
Disallow /

axios/0.17.1

Rule Path
Disallow /

axios

Rule Path
Disallow /

b2w/0.1

Rule Path
Disallow /

backdoorbot/1.0

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

barcrawler/0.9

Rule Path
Disallow /

barkrowler/0.9

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

bbot

Rule Path
Disallow /

bdfetch

Rule Path
Disallow /

bidswitchbot

Rule Path
Disallow /

birilbot

Rule Path
Disallow /

bitlybot

Rule Path
Disallow /

bitsightbot/1.0

Rule Path
Disallow /

blackboard safeassign

Rule Path
Disallow /

blackboardally

Rule Path
Disallow /

blexbot/1.0

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

blogbeatbot/2.5

Rule Path
Disallow /

blowfish/1.0

Rule Path
Disallow /

bnf.fr_bot

Rule Path
Disallow /

bookmark search tool

Rule Path
Disallow /

borneobot

Rule Path
Disallow /

botalot

Rule Path
Disallow /

botrighthere

Rule Path
Disallow /

bpimagewalker/2.0

Rule Path
Disallow /

brands-bot-logo/0.1

Rule Path
Disallow /

brandverity/1.0

Rule Path
Disallow /

brightedge crawler/1.0

Rule Path
Disallow /

bubing

Rule Path
Disallow /

buck/2.3.2

Rule Path
Disallow /

buck/2.2

Rule Path
Disallow /

buck

Rule Path
Disallow /

builtbottough

Rule Path
Disallow /

builtwith

Rule Path
Disallow /

bullseye/1.0

Rule Path
Disallow /

bunnyslippers

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

bytiye /nutch-2.3

Rule Path
Disallow /

cakephp

Rule Path
Disallow /

catexplorador/1.0beta

Rule Path
Disallow /

catexplorador

Rule Path
Disallow /

ccbot/2.0

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

censysinspect/1.1

Rule Path
Disallow /

centro ads.txt crawler/1.0

Rule Path
Disallow /

cf-uc user agent v.1d.73037

Rule Path
Disallow /

cf-uc user agent v.1d.71250

Rule Path
Disallow /

cf-uc user agent v.1d.70038

Rule Path
Disallow /

cf-uc user agent v.1d.117536

Rule Path
Disallow /

chatglm-spider/1.0

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

check_http/v1.4.15-61-g4d527

Rule Path
Disallow /

checkmarknetwork/1.0

Rule Path
Disallow /

checkmarknetwork

Rule Path
Disallow /

cherrypickerelite/1.0

Rule Path
Disallow /

cherrypickerse/1.0

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

cincraw/1.0

Rule Path
Disallow /

cipacrawler

Rule Path
Disallow /

ciscowebexmeetings-win/40.4.12

Rule Path
Disallow /

clarabot/1.4

Rule Path
Disallow /

claritybot/9.0

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

clcrawler/1.0

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

cliqzbot/3.0

Rule Path
Disallow /

cliqzbot/2.0

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

cloud mapping experiment

Rule Path
Disallow /

cloudinary/1.0

Rule Path
Disallow /

coccocbot-image/1.0

Rule Path
Disallow /

coccocbot-web/1.0

Rule Path
Disallow /

coccocbot

Rule Path
Disallow /

cocolyzebot/1.0

Rule Path
Disallow /

colly

Rule Path
Disallow /

contxbot

Rule Path
Disallow /

copernic

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

cortex/1.0

Rule Path
Disallow /

cosmos

Rule Path
Disallow /

cowbot/1.0

Rule Path
Disallow /

cpython/2.7.16

Rule Path
Disallow /

cpython/2.7.5

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

crawlson/1.0

Rule Path
Disallow /

crescent internet toolpak http ole control v.1.0

Rule Path
Disallow /

crescent

Rule Path
Disallow /

crowdtanglebot/1.0

Rule Path
Disallow /

crowdtanglebot

Rule Path
Disallow /

curl/7.65.1

Rule Path
Disallow /

curl/7.58.0

Rule Path
Disallow /

curl/7.29.0

Rule Path
Disallow /

curl/7.16.4

Rule Path
Disallow /

custom-asynchttpclient

Rule Path
Disallow /

cyotekhttp/2.0

Rule Path
Disallow /

cyotekwebcopy/1.6

Rule Path
Disallow /

dataaccessd/1.0

Rule Path
Disallow /

dataforseobot/1.0

Rule Path
Disallow /

datanyze

Rule Path
Disallow /

datagnionbot

Rule Path
Disallow /

dataprovider

Rule Path
Disallow /

ds9 2.001.ec2

Rule Path
Disallow /

ds9 2.000.ec2

Rule Path
Disallow /

dcrawl/1.0

Rule Path
Disallow /

diffbot/0.1

Rule Path
Disallow /

discordbot/2.0

Rule Path
Disallow /

dispatch/1.0.0-m1

Rule Path
Disallow /

dittospyder

Rule Path
Disallow /

docbase crawler/1.0

Rule Path
Disallow /

domaincheck.io

Rule Path
Disallow /

domaincrawler/1.0

Rule Path
Disallow /

domainstatsbot/1.0

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

domcopbot

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

drupal

Rule Path
Disallow /

duckassistbot/1.1

Rule Path
Disallow /

dyno mapper crawler 1.0

Rule Path
Disallow /

e.ventures investment crawler

Rule Path
Disallow /

easybib autocite

Rule Path
Disallow /

easybib+autocite+

Rule Path
Disallow /

ecairn-grabber/1.0

Rule Path
Disallow /

econtext

Rule Path
Disallow /

electronic frontier foundation's do not track verifier

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

embarcadero uri client/1.0

Rule Path
Disallow /

embedly/0.2

Rule Path
Disallow /

erocrawler

Rule Path
Disallow /

evc-batch/2.0

Rule Path
Disallow /

eventmachine httpclient

Rule Path
Disallow /

every-lang-seo

Rule Path
Disallow /

exabot/3.0

Rule Path
Disallow /

extractorpro

Rule Path
Disallow /

facebookexternalhit/1.1

Rule Path
Disallow /

facebot

Rule Path
Disallow /

fairad client

Rule Path
Disallow /

fantastic_search_engine_crawler/2.0

Rule Path
Disallow /

faraday v0.15.2

Rule Path
Disallow /

fasthttp

Rule Path
Disallow /

feedburner/1.0

Rule Path
Disallow /

feedfetcher-google

Rule Path
Disallow /

feedly/1.0

Rule Path
Disallow /

feedlyapp/1.0

Rule Path
Disallow /

femtosearchbot/1.0

Rule Path
Disallow /

fetch libfetch/2.0

Rule Path
Disallow /

fetcher/0.01

Rule Path
Disallow /

finbot

Rule Path
Disallow /

flaming attackbot

Rule Path
Disallow /

fluid/0.0

Rule Path
Disallow /

foobot

Rule Path
Disallow /

foregenix webscan security auditor

Rule Path
Disallow /

friendlycrawler/nutch-1.20-snapshot

Rule Path
Disallow /

futurelearn linkchecker

Rule Path
Disallow /

fuzz faster u fool v1.1.0

Rule Path
Disallow /

harvest/1.5

Rule Path
Disallow /

gabanzabot/1.1

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

garlikcrawler/1.2

Rule Path
Disallow /

gensparkbot

Rule Path
Disallow /

getright/4.2

Rule Path
Disallow /

gg peekbot 2.0

Rule Path
Disallow /

gigablastopensource/1.0

Rule Path
Disallow /

globalwebsearchx

Rule Path
Disallow /

gluten free crawler/1.0

Rule Path
Disallow /

go-http-client/1.1

Rule Path
Disallow /

go-http-client/2.0

Rule Path
Disallow /

go-resty/2.0.0

Rule Path
Disallow /

gocrawl/0.4

Rule Path
Disallow /

google favicon

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

googlebot-image/1.0

Rule Path
Disallow /

googlebot-news

Rule Path
Disallow /

googleother

Rule Path
Disallow /

googleproducer

Rule Path
Disallow /

googlebot-video/1.0

Rule Path
Disallow /

goose/3.1.6

Rule Path
Disallow /

got

Rule Path
Disallow /

gptbot/1.0

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

grammarly/1.0

Rule Path
Disallow /

grapeshotcrawler/2.0

Rule Path
Disallow /

gregarius/0.5.2

Rule Path
Disallow /

grequests/0.10

Rule Path
Disallow /

grouphigh/1.0

Rule Path
Disallow /

grouphigh/1.1

Rule Path
Disallow /

gulper web bot 0.2.4

Rule Path
Disallow /

guzzlehttp/6.5.1

Rule Path
Disallow /

hatena-favicon/2

Rule Path
Disallow /

haxermen

Rule Path
Disallow /

headerinspector bot

Rule Path
Disallow /

headlesschrome

Rule Path
Disallow /

healthcheckbot/0.2

Rule Path
Disallow /

hexometer

Rule Path
Disallow /

hloader

Rule Path
Disallow /

hello, world

Rule Path
Disallow /

hello,+world

Rule Path
Disallow /

hgfalphaxcrawl/0.1

Rule Path
Disallow /

httpie/1.0.2

Rule Path
Disallow /

httplib

Rule Path
Disallow /

httrack 3.0

Rule Path
Disallow /

httpunit/1.x

Rule Path
Disallow /

hubspot crawler

Rule Path
Disallow /

humanlinks

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ia_archiver/1.0

Rule Path
Disallow /

ia_archiver-web.archive.org

Rule Path
Disallow /

ias-va/3.1

Rule Path
Disallow /

ias crawler

Rule Path
Disallow /

iaskbot/1.0

Rule Path
Disallow /

icc-crawler/2.0

Rule Path
Disallow /

ichiro/3.0

Rule Path
Disallow /

idbte4m code87

Rule Path
Disallow /

iframely/1.3.1

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

impact radius compliance bot

Rule Path
Disallow /

indeedbot

Rule Path
Disallow /

ineturl:/1.0

Rule Path
Disallow /

infonavirobot

Rule Path
Disallow /

infotigerbot/1.9

Rule Path
Disallow /

instapaper/4.0

Rule Path
Disallow /

internet-census

Rule Path
Disallow /

internetmeasurement/1.0

Rule Path
Disallow /

ioncrawl

Rule Path
Disallow /

ips-agent

Rule Path
Disallow /

iron33/1.0.2

Rule Path
Disallow /

ivre-masscan/1.3

Rule Path
Disallow /

isscyberriskcrawler/1.1.6

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

jakarta commons-httpclient/3.1

Rule Path
Disallow /

jaunt/1.5

Rule Path
Disallow /

java/1.8.0_261

Rule Path
Disallow /

java/1.8.0_232

Rule Path
Disallow /

java/1.8.0_212

Rule Path
Disallow /

java/1.8.0_201

Rule Path
Disallow /

java

Rule Path
Disallow /

jennybot

Rule Path
Disallow /

jersey/2.25.1

Rule Path
Disallow /

jetty/9.4.11

Rule Path
Disallow /

jetty/9.4.42.v20210604

Rule Path
Disallow /

jetty

Rule Path
Disallow /

jsimplepiefactory

Rule Path
Disallow /

keys-so-bot

Rule Path
Disallow /

kenjin spider

Rule Path
Disallow /

keyword density/0.9

Rule Path
Disallow /

kocmohabt

Rule Path
Disallow /

komodiabot/1.0

Rule Path
Disallow /

kstandbot/1.0

Rule Path
Disallow /

kubectl/v1.12.0

Rule Path
Disallow /

larbin_2.6.2

Rule Path
Disallow /

leap

Rule Path
Disallow /

lexibot

Rule Path
Disallow /

libweb/clshttp

Rule Path
Disallow /

libwww-perl/6.05

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

lightspeedsystemscrawler

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

linkreader/1.0

Rule Path
Disallow /

linqia-discovery/1.0

Rule Path
Disallow /

linkpadbot/2.3

Rule Path
Disallow /

linkextractorpro

Rule Path
Disallow /

linkscan/8.1a unix

Rule Path
Disallow /

linkwalker/3.0

Rule Path
Disallow /

linkwalker/2.0

Rule Path
Disallow /

linkwalker

Rule Path
Disallow /

linuxgeturl/2.0

Rule Path
Disallow /

livelapbot/0.2

Rule Path
Disallow /

lnspiderguy

Rule Path
Disallow /

looid.com search/0.1

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

lua-resty-http/0.10

Rule Path
Disallow /

lunabot

Rule Path
Disallow /

lwp-trivial/1.34

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

mail.ru_bot/2.0

Rule Path
Disallow /

mail.ru_bot/img/2.0

Rule Path
Disallow /

mail.ru_bot/robots/2.0

Rule Path
Disallow /

marketingminer bot v1.0

Rule Path
Disallow /

masscan

Rule Path
Disallow /

mata hari

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mbcrawler/1.0

Rule Path
Disallow /

mechanize/2.8.1

Rule Path
Disallow /

mechanize/2.7.6

Rule Path
Disallow /

mediapartners-google

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

metadata-downloader-bot/1.0

Rule Path
Disallow /

metainspector/5.6.0

Rule Path
Disallow /

masscan-ng/1.3

Rule Path
Disallow /

mediapartners-google/2.1;

Rule Path
Disallow /

metainspector/4.7.2

Rule Path
Disallow /

microsoft url control - 5.01.4511

Rule Path
Disallow /

microsoft url control - 6.00.8169

Rule Path
Disallow /

microsoft url control

Rule Path
Disallow /

miixpc/4.2

Rule Path
Disallow /

miixpc

Rule Path
Disallow /

mister pix

Rule Path
Disallow /

mixnodecache/1.8

Rule Path
Disallow /

mixrankbot

Rule Path
Disallow /

mj12bot/v1.4.8

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

moget/2.1

Rule Path
Disallow /

moget

Rule Path
Disallow /

mojeekbot/0.11

Rule Path
Disallow /

mojeekbot/0.6

Rule Path
Disallow /

mojolicious

Rule Path
Disallow /

monsidobot/2.2

Rule Path
Disallow /

monsidobot

Rule Path
Disallow /

moodlebot/4.1

Rule Path
Disallow /

moodlebot/4.0

Rule Path
Disallow /

mothra/jul-10-17:33:30-edt-2006

Rule Path
Disallow /

mozilla/4.0 (compatible; bullseye; windows 95)

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

mybot/nutch-1.15-snapshot

Rule Path
Disallow /

myseosnapshot/1.0

Rule Path
Disallow /

naaraa/nutch-1.17

Rule Path
Disallow /

nadesiko

Rule Path
Disallow /

neevabot/1.0

Rule Path
Disallow /

netants

Rule Path
Disallow /

netapi

Rule Path
Disallow /

netcraftsurveyagent/1.0

Rule Path
Disallow /

netcraftsurveyagent

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

netpeakcheckerbot/3.2

Rule Path
Disallow /

newspaper/0.2.8

Rule Path
Disallow /

newspaper/0.2.5

Rule Path
Disallow /

newspaper/0.2.2

Rule Path
Disallow /

nicerspro

Rule Path
Disallow /

nimbostratus-bot/v1.3.2

Rule Path
Disallow /

nimbostratus

Rule Path
Disallow /

ninjabot/2.0

Rule Path
Disallow /

nmap scripting engine

Rule Path
Disallow /

node-fetch/1.0

Rule Path
Disallow /

notion/1.0

Rule Path
Disallow /

novichenkobot

Rule Path
Disallow /

nsrbot

Rule Path
Disallow /

nuclei - open-source project

Rule Path
Disallow /

nutch-1.18

Rule Path
Disallow /

oai-searchbot/1.0

Rule Path
Disallow /

obot/2.3.1

Rule Path
Disallow /

obot

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

omgili/0.5

Rule Path
Disallow /

omni+cms+(link+check)

Rule Path
Disallow /

onespot-scraperbot/1.0

Rule Path
Disallow /

openbot

Rule Path
Disallow /

openfind data gatherer

Rule Path
Disallow /

openfind

Rule Path
Disallow /

oracle ultra search

Rule Path
Disallow /

orbbot/1.1

Rule Path
Disallow /

outclicksbot/4

Rule Path
Disallow /

pagepeeker/3.0

Rule Path
Disallow /

pandalytics/1.0

Rule Path
Disallow /

pandalytics

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

paperlibot/2.1

Rule Path
Disallow /

peach/1.01

Rule Path
Disallow /

penthouse critical path css generator

Rule Path
Disallow /

perman

Rule Path
Disallow /

perplexity

Rule Path
Disallow /

perplexitybot/1.0

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

photon/1.0

Rule Path
Disallow /

phpcrawl

Rule Path
Disallow /

phpwebsite/5.3+

Rule Path
Disallow /

pimeyes.com crawler

Rule Path
Disallow /

pinterestbot/1.0

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /

please name your robot

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

pocketimagecache/1.0

Rule Path
Disallow /

pocketparser/2.0

Rule Path
Disallow /

poddirectory/2.1

Rule Path
Disallow /

polaris/6.01

Rule Path
Disallow /

postmanruntime/7.19.0

Rule Path
Disallow /

postmanruntime

Rule Path
Disallow /

prlog/1.0

Rule Path
Disallow /

project 25499 scanner

Rule Path
Disallow /

project+25499+scanner

Rule Path
Disallow /

probethenet

Rule Path
Disallow /

propowerbot/2.14

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

proximic

Rule Path
Disallow /

https://www.proxychecker.ai/proxy-checker

Rule Path
Disallow /

psbot

Rule Path
Disallow /

punkspider/5.0

Rule Path
Disallow /

pycurl/7.43.0.6

Rule Path
Disallow /

python/3.7

Rule Path
Disallow /

python-requests/2.8.1

Rule Path
Disallow /

python-requests/2.6.0

Rule Path
Disallow /

python-requests/2.22.0

Rule Path
Disallow /

python-requests/2.21.0

Rule Path
Disallow /

python-requests/2.19.1

Rule Path
Disallow /

python-requests/2.18.4

Rule Path
Disallow /

python-requests/2.13.0

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

python-urllib/2.7

Rule Path
Disallow /

python-urllib/1.17

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

queryn metasearch

Rule Path
Disallow /

quora-bot/1.0

Rule Path
Disallow /

quora

Rule Path
Disallow /

qwantify-dev2035/1.0

Rule Path
Disallow /

rabbot/nutch-1.17

Rule Path
Disallow /

radiation retriever 1.1

Rule Path
Disallow /

rankingbot2

Rule Path
Disallow /

repomonkey bait & tackle/v1.01

Rule Path
Disallow /

repomonkey

Rule Path
Disallow /

researchscan.comsys

Rule Path
Disallow /

researchscan/http

Rule Path
Disallow /

researchscan/t1 3rl

Rule Path
Disallow /

researchscan/t1 3rl

Rule Path
Disallow /

researchscan/t13-0rtt

Rule Path
Disallow /

researchscan/t12sns

Rule Path
Disallow /

researchscan/t12ca

Rule Path
Disallow /

rest-client/2.1.0

Rule Path
Disallow /

rest-client/2.0.2

Rule Path
Disallow /

riddler

Rule Path
Disallow /

rma

Rule Path
Disallow /

robotstxtfetcher/1.3

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

rssingbot

Rule Path
Disallow /

ruby

Rule Path
Disallow /

safarifetcherd/604.1

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

sakai java/1.8.0_212

Rule Path
Disallow /

scalaj-http/2.4.2

Rule Path
Disallow /

scrapeherobot/1.0

Rule Path
Disallow /

scrapy/2.10.1

Rule Path
Disallow /

scrapy/2.8.0

Rule Path
Disallow /

scrapy/1.5.2

Rule Path
Disallow /

scrapy/1.5.1

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

screaming frog seo spider/20.2

Rule Path
Disallow /

screaming frog seo spider/19.4

Rule Path
Disallow /

screaming frog seo spider/14.3

Rule Path
Disallow /

screaming frog seo spider/12.4

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

searchatlas.com seo crawler

Rule Path
Disallow /

searchatlas

Rule Path
Disallow /

searchpreview

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semaltbot/0.1

Rule Path
Disallow /

semanticbot

Rule Path
Disallow /

semanticscholarbot

Rule Path
Disallow /

semrushbot/6~bl

Rule Path
Disallow /

semrushbot/3~bl

Rule Path
Disallow /

semrushbot/1.0~bm

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

seobilitybot

Rule Path
Disallow /

seochecker/1.1

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

serendeputybot

Rule Path
Disallow /

serpstatbot/2.1

Rule Path
Disallow /

serpstatbot/1.0

Rule Path
Disallow /

seznambot/4.0

Rule Path
Disallow /

seznambot/3.2

Rule Path
Disallow /

seznambot/3.2-test1

Rule Path
Disallow /

sidetrade indexer bot

Rule Path
Disallow /

simulator/0.7

Rule Path
Disallow /

siteanalyzerbot/2.0

Rule Path
Disallow /

sitechecker/1.0

Rule Path
Disallow /

siteimprove

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

sjuupbot

Rule Path
Disallow /

skyworkspider

Rule Path
Disallow /

slackbot-linkexpanding 1.0

Rule Path
Disallow /

slack-imgproxy

Rule Path
Disallow /

slick-proxy

Rule Path
Disallow /

smtbot/1.0

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

snap url preview service

Rule Path
Disallow /

snapchat-proxy

Rule Path
Disallow /

spankbot

Rule Path
Disallow /

spanner

Rule Path
Disallow /

spark desktop.exe

Rule Path
Disallow /

spawning-ai

Rule Path
Disallow /

special_archiver

Rule Path
Disallow /

special_archiver/3.1.1

Rule Path
Disallow /

spiderling

Rule Path
Disallow /

sslyze/2.1.3

Rule Path
Disallow /

startmebot/1.0

Rule Path
Disallow /

startpageprivateimageproxy/2.0

Rule Path
Disallow /

storebot-google/1.0

Rule Path
Disallow /

statically-images/2.0

Rule Path
Disallow /

stractbot/0.1

Rule Path
Disallow /

streamline3bot/1.0

Rule Path
Disallow /

surdotlybot/1.0

Rule Path
Disallow /

suzuran

Rule Path
Disallow /

symfony browserkit

Rule Path
Disallow /

szukacz/1.4

Rule Path
Disallow /

t3versionsbot/1.0

Rule Path
Disallow /

telegrambot

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

telesoft

Rule Path
Disallow /

tenmilliondomainsbot/1.0

Rule Path
Disallow /

testbot/0.1

Rule Path
Disallow /

the intraformant

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

thenomad

Rule Path
Disallow /

tighttwatbot

Rule Path
Disallow /

tineye-bot/1.31

Rule Path
Disallow /

tineye-bot

Rule Path
Disallow /

tocrawl/urldispatcher

Rule Path
Disallow /

tomthepeeper/0.1

Rule Path
Disallow /

tpradstxtcrawler/1.0

Rule Path
Disallow /

trafilatura/1.8.0

Rule Path
Disallow /

trafilatura/1.2.0

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

trendsmapresolver/0.1

Rule Path
Disallow /

trensant

Rule Path
Disallow /

triplecheckerrobot/1.0

Rule Path
Disallow /

true_robot/1.0

Rule Path
Disallow /

true_robot

Rule Path
Disallow /

ttd-content

Rule Path
Disallow /

turingos

Rule Path
Disallow /

turnitinbot/1.5

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

tweetmemebot/4.0

Rule Path
Disallow /

twingly recon-klondike

Rule Path
Disallow /

twingly recon

Rule Path
Disallow /

twitterbot/1.0

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

ucmore crawler app

Rule Path
Disallow /

ultimate_sitemap_parser/0.5

Rule Path
Disallow /

um-ln/1.0

Rule Path
Disallow /

unfurlist

Rule Path
Disallow /

unirest-java/3.1.00

Rule Path
Disallow /

universalfeedparser/5.2.1

Rule Path
Disallow /

uptimebot/1.0

Rule Path
Disallow /

url control

Rule Path
Disallow /

url_spider_pro

Rule Path
Disallow /

urllib gold

Rule Path
Disallow /

urly warning

Rule Path
Disallow /

valve+steam+client

Rule Path
Disallow /

vebidoobot/1.0

Rule Path
Disallow /

velenpublicwebcrawler/1.0

Rule Path
Disallow /

velenpublicwebcrawler/0.1

Rule Path
Disallow /

velenpublicwebcrawler (velen.io)

Rule Path
Disallow /

velenpublicwebdownloader/1.0

Rule Path
Disallow /

vci webviewer vci webviewer win32

Rule Path
Disallow /

vci

Rule Path
Disallow /

vuhuvbot/1.0

Rule Path
Disallow /

vulnnr-win!10

Rule Path
Disallow /

w3c-checklink/4.5

Rule Path
Disallow /

wappalyzer

Rule Path
Disallow /

wbsrch

Rule Path
Disallow /

web image collector

Rule Path
Disallow /

webauto

Rule Path
Disallow /

webbandit/3.50

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webcapture 2.0

Rule Path
Disallow /

webcopier v.2.2

Rule Path
Disallow /

webcopier v.2.2

Rule Path
Disallow /

webcopier v3.2a

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webdatastats/1.0

Rule Path
Disallow /

webenhancer

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /

webprosbot/2.0

Rule Path
Disallow /

websauger

Rule Path
Disallow /

website quester

Rule Path
Disallow /

webster pro

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webzip

Rule Path
Disallow /

webzip/4.0

Rule Path
Disallow /

webzip/4.21

Rule Path
Disallow /

webzip/5.0

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wellknownbot/0.1

Rule Path
Disallow /

wellknownbot

Rule Path
Disallow /

wget/1.5.3

Rule Path
Disallow /

wget/1.6

Rule Path
Disallow /

wget

Rule Path
Disallow /

whatsapp/2.23.23.79

Rule Path
Disallow /

whatweb

Rule Path
Disallow /

who.is bot

Rule Path
Disallow /

whoislinkingbot

Rule Path
Disallow /

widget server

Rule Path
Disallow /

winhttp

Rule Path
Disallow /

wordpress/5.8.2

Rule Path
Disallow /

wordpress/5.8.1

Rule Path
Disallow /

wordpress/5.7.4

Rule Path
Disallow /

wordpress/5.2.11

Rule Path
Disallow /

workona-favicon-service/1.0.0

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

wpbot

Rule Path
Disallow /

www.ru

Rule Path
Disallow /

www-collector-e

Rule Path
Disallow /

xenu link sleuth/1.3.8

Rule Path
Disallow /

xenu's link sleuth 1.1c

Rule Path
Disallow /

xenu'scakephp

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yak/1.0

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximages/3.0

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /

yodaobot/1.0

Rule Path
Disallow /

yomins.com

Rule Path
Disallow /

youbot

Rule Path
Disallow /

zeus link scout

Rule Path
Disallow /

zeus 32297 webster pro v2.9 win32

Rule Path
Disallow /

zeus

Rule Path
Disallow /

zgrab/0.x

Rule Path
Disallow /

zgrab

Rule Path
Disallow /

zmeu

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

zumbot/1.0

Rule Path
Disallow /

*

Rule Path
Disallow /WebResource.axd
Disallow /ScriptResource.axd
Disallow /combinescriptshandler.axd
Disallow /images/editors/
Disallow /AccessDenied.axpx
Disallow /EditingAccount.axpx
Disallow /EditingErrors.axpx
Disallow /FileNotFound.axpx
Disallow /MobileAccess.aspx
Disallow /*Javascript%3A
Disallow /*.css$
Disallow /*.docx$
Disallow /*.doc$
Disallow /*.js$
Disallow /*.php$
Disallow /*.xlsx$
Disallow /*.xls$

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://www.biosciencewriters.com/sitemap.xml
Back to top

Comments

  • robots.txt for https://www.biosciencewriters.com/
  • China search engine
  • Baiduspider is attempting to download axd scripts and causing errors, and it ignores these dissallows unless their bot is specified explicitly.
  • bingbot is attempting to download axd scripts and causing errors, and it ignores these dissallows unless their bot is specified explicitly.
  • China search engine
  • Korea search engine
Back to top

Warnings

  • 10 invalid lines.
Back to top