bvresources.com
robots.txt
Robots Exclusion Standard data for bvresources.com
Resource Scan
Scan Details
| Site Domain | bvresources.com |
| Base Domain | bvresources.com |
| Scan Status | Ok |
| Last Scan | 2025-03-16T11:32:07+00:00 |
| Next Scan | 2025-04-15T11:32:07+00:00 |
Last Scan
| Scanned | 2025-03-16T11:32:07+00:00 |
| URL | https://bvresources.com/robots.txt |
| Redirect | https://www.bvresources.com/robots.txt |
| Redirect Domain | www.bvresources.com |
| Redirect Base | bvresources.com |
| Domain IPs | 104.26.8.176, 104.26.9.176, 172.67.71.240, 2606:4700:20::681a:8b0, 2606:4700:20::681a:9b0, 2606:4700:20::ac43:47f0 |
| Redirect IPs | 104.26.8.176, 104.26.9.176, 172.67.71.240, 2606:4700:20::681a:8b0, 2606:4700:20::681a:9b0, 2606:4700:20::ac43:47f0 |
| Response IP | 172.67.71.240 |
| Found | Yes |
| Hash | 5b90a6f6ff4df007cb801e34dc421912cb687b8fc1db310a7cb566be1f72bbd4 |
| SimHash | 5c0619179ed7 |
Groups
ahrefsbot
baiduspider
barkrowler
blexbot
buck
coccocbot
dotbot
mauibot
seznambot
sogou web spider
sogou inst spider
yandex
zoominfobot
| Rule | Path |
|---|---|
| Disallow | / |
agent gpt
agentgpt
ai article writer
ai content detector
ai dungeon
ai search engine
ai seo crawler
ai writer
ai21 labs
ai2bot
aibot
aisearchbot
alexatm
alpha ai
alphaai
amazon bedrock
amazon lex
amazonbot
amelia
anthropic-ai
anthropicai
anypicker
anyword
applebot
articoolo
autogpt
automated writer
awariorssbot
awariosmartbot
bingai
brave leo ai
bytespider
catboost
cc-crawler
ccbot
chatgpt
chinchilla
claude-web
claudebot
clearscope
cohere-ai
cohere-training-data-crawler
common crawl
commoncrawl
content harmony
content king
content optimizer
content samurai
contentatscale
contentbot
contentedge
conversion ai
copyai
copymatic
copyscape
crawlq ai
crawlspace
crew ai
crewai
dall-e
dataforseobot
deepai
deepl
deepmind
deepseek
depolarizinggpt
dialogpt
diffbot
duckassistbot
facebookbot
firecrawl
flyriver
frase ai
friendlycrawler
gemini
gemma
genai
google bard ai
google-cloudvertexbot
googleother
gpt-2
gpt-3
gpt-4
gptbot
gptzero
grammarly
grok
hemingway editor
hugging face
hypotenuse ai
iaskspider
icc-crawler
imagesiftbot
img2dataset
ink editor
inkforall
intelliseek.ai
inferkit
isscyberriskcrawler
jasperai
kafkai
kangaroo
keyword density ai
leftwinggpt
llama
magpie-crawler
marketmuse
meltwater
meta ai
meta llama
meta.ai
meta-ai
meta-externalagent
meta-externalfetcher
metaai
metatagbot
mistral
narrative device
neural text
neuralseo
oai-searchbot
oai searchbot
omgili
omnigpt
open ai
openai
opentext ai
outwrite
page analyzer ai
pangubot
paraphraser.io
peer39_crawler
perplexitybot
petalbot
prowritingaid
quillbot
rightwinggpt
robotspider
rytr
saplingai
scalenut
scrapy
scriptbook
searchgpt
semrushbot
seo content machine
seo robot
sidetrade
simplified ai
slickwrite
spin rewriter
spinbot
stability
sudowrite
surfer ai
text blaze
textcortex
the knowledge ai
timpibot
velenpublicwebcrawler
vidnami ai
webchatgpt
webzio
whisper
wordai
wordtune
writecream
writerzen
writescope
writesonic
x.ai
xai
youbot
zero gtp
zimmwriter
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /pdf/ |
| Disallow | /pdfs/ |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.bvresources.com/sitemap/sitemap-index.xml |