Log In Sign Up

abogados-barcelona.eu
robots.txt

Robots Exclusion Standard data for abogados-barcelona.eu

Resource Scan

Scan Details

Site Domain abogados-barcelona.eu
Base Domain abogados-barcelona.eu
Scan Status Ok
Last Scan2024-10-05T08:14:55+00:00
Next Scan 2024-11-04T08:14:55+00:00

Last Scan

Scanned2024-10-05T08:14:55+00:00
URL https://abogados-barcelona.eu/robots.txt
Domain IPs 2a00:1d70:c01c::171:151, 5.145.171.151
Response IP 5.145.171.151
Found Yes
Hash cf0e88e3f84f4f2ec9ba85564e94d80cadf967493db6e3635af8ecba68e3e7db
SimHash 32b27959ccf4

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Allow /
Allow /*.js$
Allow /*.css$
Allow /maps/api/js/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /wp-admin/
Disallow /wp-includes/
Allow /wp-includes/*.js$
Allow /wp-includes/*.css$
Disallow /author/
Disallow /comments/
Disallow /avisos-legales/
Disallow /politica-de-privacidad/
Disallow /politica-de-cookies/

googlebot

Rule Path
Allow /*.css$
Allow /*.js$

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

lemurwebcrawler

Rule Path
Disallow /*/booking.ashx
Disallow /*/*/booking.ashx
Disallow /*/*/*/booking.ashx

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

google-hoteladsverifier

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

berlin-fu-cow

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

semager

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

orangebot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

sputnikbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

seoscanners.net

Rule Path
Disallow /

deusu

Rule Path
Disallow /

meds-online24.com

Rule Path
Disallow /

seoscanners

Rule Path
Disallow /

pingdom.com

Rule Path
Disallow /

prtgcloudbot

Rule Path
Disallow /

linguee

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

bubing

Rule Path
Disallow /

lcc

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mauibot+

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.abogados-barcelona.eu/sitemap.xml
sitemap https://www.abogados-barcelona.eu/images_sitemap.xml
Back to top

Comments

  • This robot is from a research project. A bug in the crawler makes it# try to download non-existent pages. The following rules try to fix
  • its behaviour
  • Some bots are known to be trouble, particularly those designed to copy
  • entire sites. Please obey robots.txt.
  • Hits many times per second, not acceptable
  • http://www.nameprotect.com/botinfo.html
  • A capture bot, downloads gazillions of pages with no public benefit
  • http://www.webreaper.net/
  • COW - Proyecto universitario que escanea webs. De todas maneras no va a obtener mucha informacion de nuestra web
  • http://hpsg.fu-berlin.de/cow/
  • JikeSpider - Crawler general de uso privado. No nos reporta ningun beneficio
  • http://shoulu.jike.com/spider.html
  • Semager - Crawler para una web semantica
  • http://www.semager.de/blog/semager-bots/
  • BLEXBot - Crawler para una web semantica
  • http://webmeup-crawler.com/
  • MJ12bot - Crawler
  • http://www.majestic12.co.uk/
  • DotBot- Crawler
  • https://moz.com/researchtools/ose/dotbot
  • AhrefsBot - Crawler
  • http://ahrefs.com/robot/
  • OrangeBot - Crawler
  • support.orangebot@orange.com
  • Screaming Frog SEO Spider
  • https://www.screamingfrog.co.uk/seo-spider/
  • Megaindex
  • https://megaindex.com/crawler
  • Block MegaIndex.ru
  • Seokicks
  • https://en.seokicks.de/robot.html
  • LTX71
  • http://ltx71.com/
  • Sputnik
  • http://corp.sputnik.ru/webmaster
  • Open Link Profiler
  • http://openlinkprofiler.org/bot
  • spider@seoscanners.net
  • spider@seoscanners.net
  • Linguee
  • http://www.linguee.com/bot
  • SemrushBot
  • Sitemap
  • By David Zapata 22/12/2020 www.seo-madrid.com
  • __ _ __
  • ________ ____ ____ ___ ____ _____/ /____(_)___/ /_________ ____ ___
  • / ___/ _ \/ __ \______/ __ `__ \/ __ `/ __ / ___/ / __ // ___/ __ \/ __ `__ \
  • (__ ) __/ /_/ /_____/ / / / / / /_/ / /_/ / / / / /_/ // /__/ /_/ / / / / / /
  • /____/\___/\____/ /_/ /_/ /_/\__,_/\__,_/_/ /_/\__,_(_)___/\____/_/ /_/ /_/
Back to top