hermanodeleche.com
robots.txt

Robots Exclusion Standard data for hermanodeleche.com

Resource Scan

Scan Details

Site Domain hermanodeleche.com
Base Domain hermanodeleche.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-04-22T05:42:51+00:00
Next Scan 2024-07-21T05:42:51+00:00

Last Successful Scan

Scanned2023-12-02T05:41:11+00:00
URL https://hermanodeleche.com/robots.txt
Domain IPs 104.21.233.219, 104.21.233.220
Response IP 104.21.233.220
Found Yes
Hash 943ba3f4670d8481ed99b43b425055530a3389481d92a46b64405fbe3e2303e0
SimHash b5b04982ce8e

Groups

*

Rule Path
Disallow /startTopic/
Disallow /*?do=add
Disallow /*?do=submit
Disallow /discover/unread/
Disallow /markallread/
Disallow /staff/
Disallow /online/
Disallow /discover/
Disallow /leaderboard/
Disallow /search/
Disallow /*?advancedSearchForm=
Disallow /register/
Disallow /lostpassword/
Disallow /login/
Disallow /*?sortby=
Disallow /*?filter=
Disallow /*?tab=comments
Disallow /*?do=email
Disallow /*?do=findComment
Disallow /*?do=getLastComment
Disallow /*?do=getNewComment
Disallow /profile/

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

wget

Rule Path
Disallow /

grub-client

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.hermanodeleche.com/sitemap.php

Comments

  • Rules for Invision Community (https://invisioncommunity.com)
  • Block pages with no unique content
  • Block faceted pages and 301 redirect pages
  • Block profile pages as these have little unique value, consume a lot of crawl time and contain hundreds of 301 links
  • Sitemap URL
  • elcomercio.pe robots.txt
  • la mayoria de veces causa problemas