Log In Sign Up

librotea.com
robots.txt

Robots Exclusion Standard data for librotea.com

Resource Scan

Scan Details

Site Domain librotea.com
Base Domain librotea.com
Scan Status Ok
Last Scan2024-05-27T11:38:53+00:00
Next Scan 2024-06-03T11:38:53+00:00

Last Scan

Scanned2024-05-27T11:38:53+00:00
URL https://librotea.com/robots.txt
Domain IPs 13.33.88.116, 13.33.88.123, 13.33.88.33, 13.33.88.64
Response IP 13.33.88.116
Found Yes
Hash e899958d80017f3ca5d920fd5a1217dc37da837d76847292dba43005a8e7c0dc
SimHash 38921f13e564

Groups

*

Rule Path
Allow /
Allow /wp-sitemap.xml
Allow /estanterias/*
Allow /inspiradores/*
Allow /articulos/*
Allow /libros/*
Allow /*.php$
Allow /*.js$
Allow /*.inc$
Allow /*.css$
Allow /*.xhtml$
Allow /*.gif$
Allow /*.png$
Allow /*.jpeg$
Allow /*.jpg$
Allow /*.ico$
Disallow /buscar
Disallow /restablecer-contrasena
Disallow /wp-content/
Disallow /bundles/
Disallow /css/
Disallow /fonts/
Disallow /sites/
Disallow /images/
Disallow /js/
Disallow /admin/

genio

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

scooperbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

flamingo_searchengine

Rule Path
Disallow /

facebot

Rule Path
Disallow /

luminatebot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

r6_commentreader

Rule Path
Disallow /

yeti

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

showyoubot

Rule Path
Disallow /

gozaikbot

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

queryseekerspider

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

apache-httpclient

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

buck

Rule Path
Disallow /

wikido

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

zend_http_client

Rule Path
Disallow /

robots

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

bidswitchbot

Rule Path
Disallow /

g-i-g-a-b-o-t

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

caam

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

clickagy intelligence bot

Rule Path
Disallow /

jersey

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

omgili

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://images.librotea.com/sitemap/ame-sitemap-articles-index.xml
sitemap https://images.librotea.com/sitemap/ame-sitemap-bks-index.xml
sitemap https://images.librotea.com/sitemap/ame-sitemap-bnw-index.xml
sitemap https://images.librotea.com/sitemap/ame-sitemap-edt-index.xml
sitemap https://images.librotea.com/sitemap/ame-sitemap-ent-index.xml
sitemap https://images.librotea.com/sitemap/ame-sitemap-google-news-index.xml
sitemap https://images.librotea.com/sitemap/ame-sitemap-images-index.xml
sitemap https://images.librotea.com/sitemap/ame-sitemap-ntv-index.xml
sitemap https://images.librotea.com/sitemap/ame-sitemap-nws-index.xml
sitemap https://images.librotea.com/sitemap/ame-sitemap-shv-index.xml
sitemap https://images.librotea.com/sitemap/ame-sitemap-tags-index.xml
sitemap https://images.librotea.com/sitemap/ame-sitemap-videos-index.xml
Back to top

Comments

  • Name: robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/wc/robots.html
  • For syntax checking, see:
  • http://www.sxw.org.uk/computing/robots/check.html
  • No incluir la página de búsqueda.
  • No incluir la página de restablecer contraseña.
  • Excluir rutas de wordpress
  • No carpetas de recursos estáticos.
  • Site maps
  • Block admin section
  • Otros bots
Back to top