idus.us.es
robots.txt

Groups

*

mediapartners-google*

ubicrawler

doc

zao

facebookexternalhit

bytespider

bytedance

semrushbot

siteauditbot

semrushbot-ba

semrushbot-si

semrushbot-swa

semrushbot-ct

splitsignalbot

semrushbot-coub

sitecheck.internetseer.com

zealbot

msiecrawler

sitesnagger

webstripper

webcopier

fetch

offline explorer

teleport

teleportpro

webzip

linko

httrack

microsoft.url.control

xenu

larbin

libwww

zyborg

download ninja

petalbot

fast

wget

grub-client

k2spider

npbot

webreaper

Other Records

Comments

• The FULL URL to the DSpace sitemaps
• The https://idus.us.es will be auto-filled with the value in dspace.cfg
• XML sitemap is listed first as it is preferred by most search engines
• Default Access Group
• (NOTE: blank lines are not allowable in a group record)
• Disable access to Discovery search and filters
• Optionally uncomment the following line ONLY if sitemaps are working
• and you have verified that your site is being indexed correctly.
• Disallow: /browse
• Disallow: /handle/11441/*/browse
• If you have configured DSpace (Solr-based) Statistics to be publicly
• accessible, then you may not want this content to be indexed
• You also may wish to disallow access to the following paths, in order
• to stop web spiders from accessing user-based content
• Section for misbehaving bots
• The following directives to block specific robots were borrowed from Wikipedia's robots.txt
• advertising-related bots:
• Crawlers that are kind enough to obey, but which we'd rather not have
• unless they're feeding search engines.
• Facebook crawler
• To block Bytespider from crawling:
• To block Bytedance from crawling:
• Blocked SemrushBot since it is creating lot of requests
• To block SemrushBot from crawling your site for different SEO and technical issues:
• To block SemrushBot from crawling your site for Backlink Audit tool:
• To block SemrushBot from crawling your site for On Page SEO Checker tool and similar tools:
• To block SemrushBot from checking URLs on your site for SWA tool:
• To block SemrushBot from crawling your site for Content Analyzer and Post Tracking tools:
• To block SplitSignalBot from crawling your site for SplitSignal tool:
• To block SemrushBot-COUB from crawling your site for Content Outline Builder tool:
• Some bots are known to be trouble, particularly those designed to copy
• entire sites. Please obey robots.txt.
• Misbehaving: requests much too fast:
• If your DSpace is going down because of someone using recursive wget,
• you can activate the following rule.
• If your own faculty is bringing down your dspace with recursive wget,
• you can advise them to use the --wait option to set the delay between hits.
• The 'grub' distributed client has been *very* poorly behaved.
• Doesn't follow robots.txt anyway, but...
• Hits many times per second, not acceptable
• http://www.nameprotect.com/botinfo.html
• A capture bot, downloads gazillions of pages with no public benefit
• http://www.webreaper.net/