Log In Sign Up

navbug.com
robots.txt

Robots Exclusion Standard data for navbug.com

Resource Scan

Scan Details

Site Domain navbug.com
Base Domain navbug.com
Scan Status Ok
Last Scan2024-06-24T00:59:38+00:00
Next Scan 2024-07-01T00:59:38+00:00

Last Scan

Scanned2024-06-24T00:59:38+00:00
URL https://navbug.com/robots.txt
Domain IPs 23.235.200.89
Response IP 23.235.200.89
Found Yes
Hash dea6b8998bb0a13c3b78ae26d84eb6d442cbc532b0f6caf4105b742bb1981b90
SimHash ab6712028cd3

Groups

mediapartners-google

Rule Path
Disallow

googlebot

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

googlebot-mobile

Rule Path
Disallow

msnbot

Rule Path
Disallow

slurp

Rule Path
Disallow

teoma

Rule Path
Disallow

gigabot

Rule Path
Disallow

robozilla

Rule Path
Disallow

nutch

Rule Path
Disallow

ia_archiver

Rule Path
Disallow

baiduspider

Rule Path
Disallow

naverbot

Rule Path
Disallow

yeti

Rule Path
Disallow

yahoo-mmcrawler

Rule Path
Disallow

psbot

Rule Path
Disallow

yahoo-blogs/v3.9

Rule Path
Disallow

*

Rule Path
Disallow
Disallow /cgi-bin/

baiduspider

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

twitmunin

Rule Path
Disallow /

a1 website download

Rule Path
Disallow /

accelobot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

mediapartners-google

Rule Path
Disallow

accelovation

Rule Path
Disallow /

archive.org

Rule Path
Disallow /

archive.orgbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

atraxbot

Rule Path
Disallow /

attrakt

Rule Path
Disallow /

attributor

Rule Path
Disallow /

cloudacl

Rule Path
Disallow /

creative autoupdate

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

ccbot/1.0

Rule Path
Disallow /

charlotte.betaspider

Rule Path
Disallow /

cogent

Rule Path
Disallow /

comodospider

Rule Path
Disallow /

crawler.archive.org

Rule Path
Disallow /

discobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotnetdotcom

Rule Path
Disallow /

dropdowndeals

Rule Path
Disallow /

findfiles.net/0.98

Rule Path
Disallow /

getleft 1.2

Rule Path
Disallow /

gootkit auto-rooter scanner

Rule Path
Disallow /

heratrix

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

heretix

Rule Path
Disallow /

heratix

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

ichiro/2.0

Rule Path
Disallow /

ilial

Rule Path
Disallow /

influencebot/0.9

Rule Path
Disallow /

itsapic.com_crawler

Rule Path
Disallow /

kimsufi.com

Rule Path
Disallow /

lemurproject nutch spider

Rule Path
Disallow /

litefinder

Rule Path
Disallow /

lssbot.com

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

msr-isrccrawler

Rule Path
Disallow /

msrputnik

Rule Path
Disallow /

mvk-it.com

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

openx

Rule Path
Disallow /

pagenest

Rule Path
Disallow /

proxyway.com

Rule Path
Disallow /

ptd-crawler

Rule Path
Disallow /

purebot

Rule Path
Disallow /

qbikspider

Rule Path
Disallow /

ripper

Rule Path
Disallow /

sbider/sbider-0.8.2-dev

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

semrushbot/0.9

Rule Path
Disallow /

shablastbot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sixtrix

Rule Path
Disallow /

softbytelabs.com

Rule Path
Disallow /

sogou

Rule Path
Disallow /

spider06.yandex.ru

Rule Path
Disallow /

squid-prefetch

Rule Path
Disallow /

superbot

Rule Path
Disallow /

swish

Rule Path
Disallow /

swish-e

Rule Path
Disallow /

twtelecom.net

Rule Path
Disallow /

webalta

Rule Path
Disallow /

webpix

Rule Path
Disallow /

webster pro v3.4

Rule Path
Disallow /

wells search ii

Rule Path
Disallow /

wep

Rule Path
Disallow /

wget

Rule Path
Disallow /

wget/1.9

Rule Path
Disallow /

wget/1.10 devel

Rule Path
Disallow /

wget/1.10.2

Rule Path
Disallow /

wget/1.12

Rule Path
Disallow /

winhttp

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

wow64

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

yanga worldsearch bot v1.1/beta

Rule Path
Disallow /

yeti

Rule Path
Disallow /

your-server.de

Rule Path
Disallow /

zermelo

Rule Path
Disallow /

charlotte

Rule Path
Disallow /

sosospider

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.navbug.com/sitemap.xml
Back to top

Comments

  • User-agent: amazon
  • Disallow: /
  • User-agent: amazonaws.com
  • Disallow: /
Back to top

Warnings

  • 6 invalid lines.
Back to top