*
| Rule |
Path |
| Disallow |
/log/ |
| Disallow |
/mailers/ |
| Disallow |
/uploads/ |
| Disallow |
/home/rssfeedsnew/ |
| Disallow |
/home/rssfeeds/ |
| Disallow |
/Scripts/ |
| Disallow |
/images/ |
| Disallow |
/gadgets/ |
| Disallow |
/cgi-bin/ |
| Disallow |
/bin/ |
mob2bw
| Rule |
Path |
| Allow |
/portal/site/exxonmobil/ |
| Allow |
/portal/site/exxonmobilchemical/ |
scoutjet
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
yahoo! slurp
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
bingbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
googlebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
yandexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
twitterbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
googlebot-image
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
ahrefsbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
spinn3r
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
omgilibot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
msnbot-newsblogs
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
addthis.com robot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
yandexnews
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
livelapbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
msnbot-udiscovery
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
paperlibot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
trendictionbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
newslebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
yandeximages
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
diffbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
tweetmemebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
mail.ru_bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
rediffnewsbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
linkedinbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
yandeximageresizer
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
mj12bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
everyonesocialbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
hubspot page fetching bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
konqueror
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
msnbot-media
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
zumbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
businesswirebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
picturebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
interfaxbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
interfaxscanbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
kraken
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
flamingo_searchengine
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
linkdexbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
nachtschatten, neofonie search:robot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
vocusbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
umbot-ln
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
yeti
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
umbot-fc
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
openhosebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
googlebot-mobile
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
wikioimagesbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
scooperbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
kemvibot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
xintellibot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
archive.org_bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
semrushbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
showyoubot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
bitlybot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
veooz
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
zitebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
tineye-bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
exabot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
tweetedtimes bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
rogerbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
james bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
newsbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
blekkobot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
contextad bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
queryseekerspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
squirrobot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
marketbrewbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
mixbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
abot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
ms search 6.0 robot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
heritrix
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
dotbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
dow jones searchbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
mediajam-bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
slackbot-linkexpanding
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
landau-media-spider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
surveybot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
socialshare
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
ldrbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
wasalive-bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
gozaikbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
marketwirebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
seznambot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
g2reader-bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
gyffu_bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
academicbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
bufferbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
ftrf: friendly robot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
yolinkbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
rightintelbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
claritybot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
cxensebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
yandex.gazeta bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
newslookup-bot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
sreleasebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
nerdybot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
skimbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
datagnionbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
archiveteam archivebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
focuseekbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
slurp
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
linkpadbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
mojeekbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
yioopbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
sinaweibobot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
spiderbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
swayy api
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
iisbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
msnbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
robot leapit.com
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
gsa-crawler
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
vegebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
apis-google
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
mediapartners-google
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
adsbot-google-mobile
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
adsbot-google
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
googlebot-news
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
googlebot-video
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
duckduckbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
baiduspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
baiduspider-image
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
baiduspider-video
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
baiduspider-news
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
baiduspider-favo
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
baiduspider-cpro
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
baiduspider-ads
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
baiduspider
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
omgilibot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
facebot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
applenewsbot
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
expo9
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
nuzzel
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
newspaper
No rules defined. All paths allowed.
Other Records
| Field |
Value |
| crawl-delay |
10 |
Comments