Log In Sign Up

blog.vepo.dev
robots.txt

Robots Exclusion Standard data for blog.vepo.dev

Resource Scan

Scan Details

Site Domain blog.vepo.dev
Base Domain vepo.dev
Scan Status Ok
Last Scan2025-10-04T10:58:21+00:00
Next Scan 2025-11-03T10:58:21+00:00

Last Scan

Scanned2025-10-04T10:58:21+00:00
URL https://blog.vepo.dev/robots.txt
Domain IPs 185.199.108.153, 185.199.109.153, 185.199.110.153, 185.199.111.153, 2606:50c0:8000::153, 2606:50c0:8001::153, 2606:50c0:8002::153, 2606:50c0:8003::153
Response IP 185.199.111.153
Found Yes
Hash 2b1c4b1ca3d6f0344caf028615b1affca500b4ba9bc5505639d6e0ef8ecfff87
SimHash 31c45051a467

Groups

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

*

Rule Path
Allow /
Allow /_/
Allow /posts/_/
Allow /tutorial/_/
Allow /en/_/
Allow /computacao-critica/_/
Disallow /assets/

awariobot
embedly
facebookexternalhit
google-safety
google web preview
google-read-aloud
linkedinbot
mediapartners-google
mediatoolkitbot
muckrack
netvibes
pinterestbot
twitterbot
turnitin
twingly
bitlybot

Rule Path
Disallow /

brightedge crawler
dataforseobot
pandalytics
proximic
screaming frog seo spider
semrushbot*
zoominfobot
zoombot

Rule Path
Disallow /

admantx
barkrowler
buck
coccocbot-web
dataprovider.com
easybib autocite
newspaper
scrapy
trafilatura
um-ln

Rule Path
Disallow /

ai2bot*
aihitbot
amazonbot
andibot
anthropic-ai
applebot*
bedrockbot
brightbot*
bytespider
ccbot
chatgpt-user
claude*
cohere*
cotoyogi
crawlspace
diffbot
duckassistbot
facebookbot
factset_spyderbot
firecrawlagent
friendlycrawler
google-cloudvertexbot
google-extended
googleother*
gptbot
iaskspider*
icc-crawler
imagesiftbot
img2dataset
isscyberriskcrawler
kangaroo bot
meta-external*
mistralai*
novaact
oai-searchbot
omgili*
operator
pangubot
panscient*
perplexity*
petalbot
phindbot
qualifiedbot
quillbot*
sbintuitionsbot
sidetrade indexer bot
tiktokspider
timpibot
velenpublicwebcrawler
webzio-extended
wpbot
yandexadditional*
youbot

Rule Path
Disallow /

adsbot-google-mobile

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://blog.vepo.dev/sitemap.xml
Back to top

Comments

  • www.robotstxt.org/
  • www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449
  • Disabling AI bots
  • SOCIAL MEDIA E MEDIA CRAWLERS
  • SEO E ANALYTICS TOOLS
  • CONTENT AGGREGATORS E SCRAPERS
  • AI BOTS E LANGUAGE MODELS
  • MOBILE ADS BOTS
Back to top

Warnings

  • 1 invalid line.
Back to top