subastas.io
robots.txt

Robots Exclusion Standard data for subastas.io

Resource Scan

Scan Details

Site Domain subastas.io
Base Domain subastas.io
Scan Status Ok
Last Scan2025-10-08T11:16:29+00:00
Next Scan 2025-11-07T11:16:29+00:00

Last Scan

Scanned2025-10-08T11:16:29+00:00
URL https://subastas.io/robots.txt
Domain IPs 104.21.94.152, 172.67.137.104, 2606:4700:3034::6815:5e98, 2606:4700:3037::ac43:8968
Response IP 104.21.94.152
Found Yes
Hash 9a49193691897cf7d7073deb6c860e2bfaa690786517f904f644a7607e1c1f84
SimHash e5135d92c6f1

Groups

*

Rule Path
Disallow /acceder
Disallow /acceder-con-enlace
Disallow /login
Disallow /magiclink
Disallow /registro
Disallow /recuperar
Disallow /favoritos
Disallow /busquedas
Disallow /usuario
Disallow /precios
Disallow /contacto
Disallow /finca/*
Disallow /bien/*
Disallow /subasta/*
Disallow /*?
Disallow /?s=
Disallow /cdn-cgi/
Disallow /detalleSubasta.php
Allow /assets/*.js
Allow /assets/*.css

noxtrumbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

ia_archiver

Rule Path
Disallow

msiecrawler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

libwww

Rule Path
Disallow /

orthogaffe

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

cutestat

Rule Path
Disallow /

Other Records

Field Value
sitemap https://subastas.io/sitemap.xml

Comments

  • Block crawlers