Log In Sign Up

bol.pt
robots.txt

Robots Exclusion Standard data for bol.pt

Resource Scan

Scan Details

Site Domain bol.pt
Base Domain bol.pt
Scan Status Ok
Last Scan2024-11-13T21:19:14+00:00
Next Scan 2024-11-20T21:19:14+00:00

Last Scan

Scanned2024-11-13T21:19:14+00:00
URL https://bol.pt/robots.txt
Domain IPs 104.26.12.138, 104.26.13.138, 172.67.72.198, 2606:4700:20::681a:c8a, 2606:4700:20::681a:d8a, 2606:4700:20::ac43:48c6
Response IP 104.26.12.138
Found Yes
Hash d9fd9a0a025573972e1ec13ac0ec19f75d5218c1f9b769401d3a7291c93e965c
SimHash d21c432a479b

Groups

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

proximic

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

crowsnest

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

httrack

Rule Path
Disallow /

identity

Rule Path
Disallow /

yandex

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

moget

Rule Path
Disallow /

ichiro

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

exabot

Rule Path
Disallow /

flipboard

Rule Path
Disallow /

flipboardproxy

Rule Path
Disallow /

unwindfetchor

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

page2rss

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

discoverybot

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

embedly

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

acoon

Rule Path
Disallow /

backlink rastreador

Rule Path
Disallow /

www.integromedb.org/crawler

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

bdcbot

Rule Path
Disallow /

abonti

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

rankactivelinkbot

Rule Path
Disallow /

gear5

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

criteobot/0.1

Rule Path
Disallow /

*

Rule Path
Disallow /Ajuda/PrivacidadeSeguranca
Disallow /Publicidade
Allow /
Back to top

Other Records

Field Value
sitemap http://www.bol.pt/sitemap.xml
Back to top