Log In Sign Up

martybugs.net
robots.txt

Robots Exclusion Standard data for martybugs.net

Resource Scan

Scan Details

Site Domain martybugs.net
Base Domain martybugs.net
Scan Status Ok
Last Scan2025-12-26T08:52:16+00:00
Next Scan 2026-01-02T08:52:16+00:00

Last Scan

Scanned2025-12-26T08:52:16+00:00
URL https://martybugs.net/robots.txt
Domain IPs 116.90.61.41
Response IP 116.90.61.41
Found Yes
Hash a019edb838303241858a3dfd6e779c6926135f468a7664fc1ec39b1c8ac6e7d0
SimHash 439e8bd2e8f1

Groups

turnitinbot

Rule Path
Disallow /

npbot

Rule Path
Disallow /

sitecheck

Rule Path
Disallow /

rumours-agent

Rule Path
Disallow /

larbin

Rule Path
Disallow /

almaden

Rule Path
Disallow /

http://www.almaden.ibm.com/cs/crawler

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webstripper/2

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

openbot

Rule Path
Disallow /

webzip

Rule Path
Disallow /

quepasacreep

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

superbot

Rule Path
Disallow /

wget

Rule Path
Disallow /

web downloader

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

msrbot

Rule Path
Disallow /

duggmirror

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

pagenest

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

similarpages

Rule Path
Disallow /

yodaobot

Rule Path
Disallow /

camontspider

Rule Path
Disallow /

purebot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

oodlebot

Rule Path
Disallow /

mp3bot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

garlikcrawler

Rule Path
Disallow /

gosospider

Rule Path
Disallow /

bender

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

covario

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

mail.ru

Rule Path
Disallow /

lemurwebcrawler

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

beetlebot

Rule Path
Disallow /

linkscrawler

Rule Path
Disallow /

easouspider

Rule Path
Disallow /

solomonobot

Rule Path
Disallow /

integromedb

Rule Path
Disallow /

linguee

Rule Path
Disallow /

a6-indexer

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

openwebspider

Rule Path
Disallow /

robosourcer

Rule Path
Disallow /

niki-bot

Rule Path
Disallow /

pimonster

Rule Path
Disallow /

aboundex

Rule Path
Disallow /

spbot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

betabot

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

toweyabot

Rule Path
Disallow /

blackboard safeassign

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

bubing

Rule Path
Disallow /

experibot

Rule Path
Disallow /

gigablastopensource

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

riddler

Rule Path
Disallow /

testua

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

adsbot

Rule Path
Disallow /
Back to top

Comments

  • stop some bots
  • TurnitinBot/1.5 (http://www.turnitin.com/robot/crawlerinfo.html)
  • NPBot-1/2.0 (http://www.nameprotect.com/botinfo.html)
  • sitecheck.internetseer.com (For more info see: http://sitecheck.internetseer.com)
  • Rumours-Agent
  • larbin_2.6.2 (larbin2.6.2@unspecified.mail)
  • http://www.almaden.ibm.com/cs/crawler [wf160]
  • http://www.almaden.ibm.com/cs/crawler [c01]
  • Teleport Pro (http://www.tenmax.com/teleport/pro/home.htm)
  • WebCopier
  • WebStripper (http://www.webstripper.net/)
  • MSIECrawler - added 12/02/04
  • Openbot (http://www.openfind.com.tw/robot.html) - added 12/02/04
  • WebZIP (http://www.spidersoft.com/webzip/default.asp) - added 08/03/04
  • QuepasaCreep - added 28/06/04
  • WebReaper (http://www.webreaper.net/) - added 20/07/04
  • SuperBot (aka Website Downloader ?) - added 02/08/04
  • wget - added 16/08/04
  • Web Downloader - added 29/10/04
  • ShopWiki bot - added 18/08/06
  • MSRBOT - added 29/09/06
  • DugMirror - added 07/01/07
  • Twiceler-0.9 http://www.cuill.com/twiceler/robot.html - added 31/07/07
  • pagenest.com - added 01/06/08
  • dotbot - added 29/07/08
  • discobot - added 26/12/08
  • SimilarPages Nutch Crawler
  • added 07/06/09
  • added 17/01/10
  • added 29/06/10
  • added 26/11/10
  • added 12/12/10
  • added 13/02/11
  • added 15/04/11
  • added 02/06/11
  • added 09/06/11
  • added 10/07/11
  • added 28/07/11
  • added 09/08/11
  • added 24/09/11
  • added 05/10/11
  • added 08/10/11
  • added 18/12/11
  • added 15/01/12
  • added 26/01/12
  • added 30/01/12
  • added 01/03/12
  • added 09/06/12
  • added 10/06/12
  • added 30/07/12
  • added 28/08/12
  • added 05/10/12
  • added 21/11/12
  • added 11/02/13
  • added 22/02/13
  • added 26/06/13
  • added 17/07/13
  • added 10/10/13
  • added 24/11/13
  • added 11/12/13
  • added 25/01/14
  • added 11/02/14
  • added 26/03/14
  • added 27/03/14
  • added 29/03/14
  • added 06/04/14
  • added 11/04/14
  • added 05/10/14
  • added 08/09/15
  • added 10/12/15
  • added 22/03/16
  • added 21/05/16
  • added 19/01/17
  • added 03/02/17
  • added 08/02/17
  • added 26/02/17
  • added 02/07/17
  • added 02/07/17
  • added 07/07/17
  • added 01/08/17
  • added 13/02/2020
  • added 14/01/2020
Back to top

Warnings

  • 2 invalid lines.
Back to top