Log In Sign Up

lymenet.org
robots.txt

Robots Exclusion Standard data for lymenet.org

Resource Scan

Scan Details

Site Domain lymenet.org
Base Domain lymenet.org
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-10-07T09:51:21+00:00
Next Scan 2026-01-05T09:51:21+00:00

Last Successful Scan

Scanned2023-11-25T07:37:03+00:00
URL https://lymenet.org/robots.txt
Domain IPs 104.21.15.213, 172.67.164.125, 2606:4700:3033::ac43:a47d, 2606:4700:3037::6815:fd5
Response IP 172.67.164.125
Found Yes
Hash 8a01b15cfe20dab6a0c6549b5761ad3de4bf33c6cad550a83a6aeddee4690b39
SimHash 6f34590bcb70

Groups

*

Rule Path
Disallow /icons/
Disallow /ubb/Forum1/HTML/
Disallow /ubb/Forum2/HTML/
Disallow /ubb/Forum3/HTML/
Disallow /ubb/Forum4/HTML/
Disallow /ubb/Forum5/HTML/
Disallow /ubb/Forum6/HTML/
Disallow /ubb/Forum7/HTML/
Disallow /ubb/Forum8/HTML/
Disallow /ubb/Forum9/HTML/

Other Records

Field Value
crawl-delay 5

dialect

Rule Path
Disallow /

psbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

proximic

Rule Path
Disallow /php/

obot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap http://www.lymenet.org/sitemap.xml.gz
Back to top

Comments

  • lyme.net robots.txt template
  • $Id: robots.txt,v 1.4 2013/08/14 14:08:19 root Exp $
  • (ref http://info.webcrawler.com/mak/projects/robots/norobots.html)
  • http://www.robotstxt.org/wc/exclusion.html
  • $Log: robots.txt,v $
  • Revision 1.4 2013/08/14 14:08:19 root
  • Updated from flash.lyme.net settings
  • Revision 1.3 2013/08/12 16:45:33 root
  • Block legacy HTML forum links
  • Revision 1.2 2009/02/08 20:50:54 root
  • allow /ubb/ now that spider-friendly URLs enabled in UBB
  • Revision 1.5 2008/08/24 18:04:49 root
  • add YodaoBot
  • Revision 1.5 2008/08/24 17:59:09 root
  • add YodaoBot
  • Revision 1.4 2008/04/06 15:17:27 root
  • add psbot, twiceler
  • Revision 1.3 2008/01/29 03:38:52 root
  • update disallow entries
  • Revision 1.2 2008/01/26 14:55:56 root
  • add robot extensions
  • Revision 1.1 2008/01/26 14:53:40 root
  • Initial revision
  • Allow selected access to WWW robots (wanderers/spiders)
  • twb, Tue Feb 18 20:20:34 EST 1997
  • twb, Tue Jan 30 11:29:17 EST 2001 - update directories
  • twb, Wed Mar 24 09:58:03 EST 2004 - add dialectizer disallow
  • twb, Sun Feb 8 15:50:27 EST 2009 - allow /ubb/ and /scripts/ now that spider-friendly URLs enabled in UBB
  • twb, Sun Mar 27 13:54:15 EDT 2016 - add AhrefsBot
  • Disallow: /files/
  • Disallow: /ubb/
  • Disallow: /scripts/
  • http://www.rinkworks.com/dialect/disallow.shtml
  • http://www.picsearch.com/menu.cgi?item=Psbot
  • http://www.cuil.com/info/webmaster_info/
  • http://www.yodao.com/help/webmaster/robot/004/
  • https://ahrefs.com/robot
  • SEOPROFILER.COM
  • ezooms.com - One of the absolute must to block in every way you can from spying on you !!!
  • sistrix
  • Yandex bot - A rule breaker, just as Baidu spiders
  • User-agent: Yandex
  • Disallow: /
  • proximic.com/info/spider.php
  • http://filterdb.iss.net/crawler/
  • MJ12bot - http://mj12bot.com/
  • https://megaindex.com/crawler
Back to top

Warnings

  • `host` is not a known field.
  • `request-rate` is not a known field.
  • `visit-time` is not a known field.
Back to top