Log In Sign Up

interbenavente.es
robots.txt

Robots Exclusion Standard data for interbenavente.es

Resource Scan

Scan Details

Site Domain interbenavente.es
Base Domain interbenavente.es
Scan Status Ok
Last Scan2024-11-14T19:47:13+00:00
Next Scan 2024-11-21T19:47:13+00:00

Last Scan

Scanned2024-11-14T19:47:13+00:00
URL https://interbenavente.es/robots.txt
Domain IPs 104.21.8.138, 172.67.139.131, 2606:4700:3031::6815:88a, 2606:4700:3034::ac43:8b83
Response IP 172.67.139.131
Found Yes
Hash dc2bce01ab9f12dfde838c41a659443ca47eafc4688173851fcfa6c8c1b3e723
SimHash 1306516184e8

Groups

neticle crawler

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /

telegrambot

Rule Path
Disallow /

amazonadbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

criteobot

Rule Path
Disallow /

sirdatabot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

buck

Rule Path
Disallow /

twitterbot

Rule Path
Disallow *
Allow /upload

moreover

Rule Path
Disallow /

test-bot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yahoo! slurp

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

linkdexbot/2.1

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

moodlebot/1.0

Rule Path
Disallow /

moodlebot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

applebot/0.1

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

baiduspider+

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

nutch

Rule Path
Disallow /

dow jones searchbot

Rule Path
Disallow /

linkdex

Rule Path
Disallow /

linkdex.com

Rule Path
Disallow /

linkdex.com/v2.0

Rule Path
Disallow /

flamingo_searchengine+(+http://www.flamingosearch.com/bot)

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

owlin bot v. 3.0

Rule Path
Disallow /

owlin bot v3

Rule Path
Disallow /

owlin bot

Rule Path
Disallow /

owlin

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

panscient.com

Rule Path
Disallow /

domainappender

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

aboundexbot

Rule Path
Disallow /

linguee bot

Rule Path
Disallow /

xovibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

fyberspider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

niki-bot

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

jobs.de-robot

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

seodiver

Rule Path
Disallow /

spbot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

backlinkcrawler

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

obot

Rule Path
Disallow /

fr-crawler

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

cloudservermarketspider

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

careerbot

Rule Path
Disallow /

lipperhey-kaus-australis

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

metajobbot

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

linkstats

Rule Path
Disallow /

jobboersebot

Rule Path
Disallow /

iccrawler

Rule Path
Disallow /

plista

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

um-ic

Rule Path
Disallow /

mindupbot

Rule Path
Disallow /

sg-orbiter

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

kraken

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

haosouspider

Rule Path
Disallow /

openhosebot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

thumbsniper

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

implisensebot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

alphaseobot

Rule Path
Disallow /

alphaseobot-sa

Rule Path
Disallow /

indeedbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

*

Rule Path
Disallow /app
Disallow /timeout

yoozbot

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

domaincrawler

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

*

Rule Path
Disallow

Other Records

Field Value
crawl-delay 120
Back to top

Warnings

  • 10 invalid lines.
Back to top