Log In Sign Up

bigbluebook.org
robots.txt

Robots Exclusion Standard data for bigbluebook.org

Resource Scan

Scan Details

Site Domain bigbluebook.org
Base Domain bigbluebook.org
Scan Status Ok
Last Scan2024-09-23T02:04:56+00:00
Next Scan 2024-10-07T02:04:56+00:00

Last Scan

Scanned2024-09-23T02:04:56+00:00
URL https://bigbluebook.org/robots.txt
Domain IPs 45.56.96.130
Response IP 45.56.96.130
Found Yes
Hash 0b23c1b233a80b86c4dcde97dd3bb37a83af1af0428d197cea01b3f484ecb0b9
SimHash 589f75466530

Groups

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

Rule Path
Disallow /p/

Other Records

Field Value
crawl-delay 10

ia_archiver

Rule Path
Disallow /p/

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /p/

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Disallow /p/
Disallow /feedback/
Disallow /es/feedback/
Disallow /fr/feedback/
Disallow /de/feedback/
Disallow /it/feedback/
Disallow /ru/feedback/
Disallow /pt/feedback/
Disallow /pl/feedback/
Disallow /nl/feedback/
Disallow /hu/feedback/
Disallow /sv/feedback/
Disallow /fi/feedback/
Disallow /ko/feedback/

Other Records

Field Value
crawl-delay 1

adsbot

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

rankboteu

Rule Path
Disallow /

ultimate_sitemap_parser

Rule Path
Disallow /

daum

Rule Path
Disallow /

paracrawl

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

semanticscholarbot

Rule Path
Disallow /

indeedbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow *

mauibot

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

swiftbot

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

hubspot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

nutch

Rule Path
Disallow /

nutch

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

thunderstone

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

voltron

Rule Path
Disallow /

ips-agent

Rule Path
Disallow /

lssrocketcrawler/1.0 lightspeedsystems

Rule Path
Disallow /

lssrocketcrawler

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

nerdybot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

wget

Rule Path
Disallow /

wget

Rule Path
Disallow /

litefinder

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

vscooter

Rule Path
Disallow /

psbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

taptubot

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

infopath

Rule Path
Disallow /

infopath.2

Rule Path
Disallow /

swebot

Rule Path
Disallow /

ec2linkfinder

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

searchmetericsbot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

ip-web-crawler.com

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

linkpadbot

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

riddler

Rule Path
Disallow /

loadtimebot

Rule Path
Disallow /

obot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

memorybot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

advbot

Rule Path
Disallow /
Back to top

Warnings

  • 2 invalid lines.
Back to top