foreign.mingluji.com
robots.txt

Groups

mediapartners-google

baiduspider
yisouspider
sogou web spider
sogoubot
bytespider
haosouspider
yodaobot
bingbot
googlebot
msnbot
*

amazonbot

semrushbot

claudebot

gptbot

chatgpt

Other Records

ubicrawler

doc

zao

sitecheck.internetseer.com

zealbot

msiecrawler

sitesnagger

webstripper

webcopier

fetch

offline explorer

teleport

teleportpro

webzip

linko

httrack

microsoft.url.control

xenu

larbin

libwww

zyborg

download ninja

wget

grub-client

k2spider

npbot

webreaper

Other Records

Comments

• jamesqi 2014-11-27 14:27
• foreign.mingluji.com
• Add Start
• NOTICE: The collection of content and other data on this site through automated means, including any device, tool,or process designed to data mine or scrape content, is prohibited except (1) for the purpose of search engine indexing or artificial intelligence retrieval augmented generation or (2) with express written permission from this site’s operator.
• To request permission to license our intellectual property and/or other materials, please contact this site’s operator directly.
• BEGIN Cloudflare Managed content
• Disallow: /Talk
• 2018-12-18 comment below line because google webmaster tools can not get resouce of load.php
• Disallow: /load.php
• Disallow: /images/
• Directory refused to crawl.
• END Cloudflare Managed Content
• sitemap start
• sitemap end
• Crawl-delay: 300 # set to 300 seconds to wait between successive requests to the same server for Yahoo Slurp
• Request-rate: 1/10 # maximum rate is one page every 5 seconds
• Visit-time: 0000-0800
• Request-rate: 1/20s 1020-1200 # between 10:20 to 12:00, 1 visit in 20s
• Add End
• Crawlers that are kind enough to obey, but which we'd rather not have
• unless they're feeding search engines.
• Some bots are known to be trouble, particularly those designed to copy
• entire sites. Please obey robots.txt.
• Sorry, wget in its recursive mode is a frequent problem.
• Please read the man page and use it properly; there is a
• --wait option you can use to set the delay between hits,
• for instance.
• The 'grub' distributed client has been *very* poorly behaved.
• Doesn't follow robots.txt anyway, but...
• Hits many times per second, not acceptable
• http://www.nameprotect.com/botinfo.html
• A capture bot, downloads gazillions of pages with no public benefit
• http://www.webreaper.net/
• Don't allow the wayback-maschine to index user-pages
• User-agent: ia_archiver
• Disallow: /wiki/User
• Disallow: /wiki/Benutzer
• Friendly, low-speed bots are welcome viewing article pages, but not
• dynamically-generated pages please.
• Inktomi's "Slurp" can read a minimum delay between hits; if your
• bot supports such a thing using the 'Crawl-delay' or another
• instruction, please let us know.

Warnings

• 1 invalid line.