Log In Sign Up

mofa.gov.mm
robots.txt

Robots Exclusion Standard data for mofa.gov.mm

Resource Scan

Scan Details

Site Domain mofa.gov.mm
Base Domain mofa.gov.mm
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-09-29T22:03:14+00:00
Next Scan 2024-12-28T22:03:14+00:00

Last Successful Scan

Scanned2023-02-15T21:28:17+00:00
URL https://mofa.gov.mm/robots.txt
Domain IPs 103.89.48.21
Response IP 103.89.48.21
Found Yes
Hash f8906c564cfd1c0cc1b93f4722dc27ba59389a8a8a75311383a0b125a3196a9f
SimHash 704bfb4ddec2

Groups

*

Rule Path
Disallow /

*

Rule Path
Disallow /wp-admin/
Disallow /wp-content/
Disallow /wp-includes/
Disallow /comment-page-
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /feed/
Allow /wp-content/uploads/

powermapper

Rule Path
Allow /

exabot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

seebot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

genieo

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

gigablastopensource

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

obot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

vegebot

Rule Path
Disallow /

screenerbot

Rule Path
Disallow /

garlik

Rule Path
Disallow /

businessbot

Rule Path
Disallow /

fast enterprise

Rule Path
Disallow /

nutch

Rule Path
Disallow /

spbot

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

kraken

Rule Path
Disallow /

flipboard

Rule Path
Disallow /

linkdex

Rule Path
Disallow /

sitecheck-sitecrawl by siteimprove.com

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

prlog

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

kinglandsystemscorp

Rule Path
Disallow /

sitemapbot

Rule Path
Disallow /

deepcrawl

Rule Path
Disallow /

yeti

Rule Path
Disallow /

netshelter

Rule Path
Disallow /

lssrocketcrawler

Rule Path
Disallow /

idmarch

Rule Path
Disallow /

cukbot

Rule Path
Disallow /

expertsearchspider

Rule Path
Disallow /

goodzer

Rule Path
Disallow /

wesee

Rule Path
Disallow /

symfony2

Rule Path
Disallow /

rogerbot

Rule Path
Allow /
Disallow /assets
Disallow /media
Disallow /images
Disallow /bundles
Disallow /App_Images
Disallow /locations

bitlybot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

searchmassive

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

aboutusbot

Rule Path
Disallow /

dotbot

Rule Path
Allow /
Disallow /assets
Disallow /media
Disallow /images
Disallow /bundles
Disallow /App_Images
Disallow /locations

brightedge

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

go 1.1

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

y!j-asr

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

crazywebcrawler

Rule Path
Disallow /

dow jones searchbot

Rule Path
Disallow /

wecrawlforthepeace

Rule Path
Disallow /

welikelinks

Rule Path
Disallow /

toweyabot

Rule Path
Disallow /

tscoutbot

Rule Path
Disallow /

wappalyzer

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap sitemap.xml
sitemap sitemap.xml
Back to top

Comments

  • Disallowed and allowed directories and files
  • SortSite
  • exabot.com
  • Mozilla/5.0+(compatible;+Exabot/3.0;++http://www.exabot.com/go/robot)
  • http://www.exabot.com/go/robot
  • Riddler
  • Riddler (http://riddler.io/about)
  • http://riddler.io/about
  • SMTBot
  • http://www.similartech.com/smtbot
  • Mozilla/5.0+(compatible;+SMTBot/1.0;++http://www.similartech.com/smtbot)
  • Majestic Bot (MJ12bot)
  • http://www.majestic12.co.uk/bot.php
  • Mozilla/5.0+(compatible;+MJ12bot/v1.4.5;+http://www.majestic12.co.uk/bot.php?+)
  • SeeBot
  • http://www.seegnify.com/bot
  • seebot/2.0 (+http://www.seegnify.com/bot)
  • Archive.org Bot
  • http://archive.org/details/archive.org_bot
  • Mozilla/5.0+(compatible;+archive.org_bot;+Wayback+Machine+Live+Record;++http://archive.org/details/archive.org_bot)
  • Mozilla/5.0+(compatible;+special_archiver/3.1.1++http://www.archive.org/details/archive.org_bot)
  • AdBeat Bot
  • https://www.adbeat.com/operation_policy
  • adbeat_bot
  • MeanPathBot
  • http://www.meanpath.com/meanpathbot.html
  • Mozilla/5.0+(compatible;+meanpathbot/1.0;++http://www.meanpath.com/meanpathbot.html)
  • Mojeek Bot
  • https://www.mojeek.com/bot.html
  • Mozilla/5.0+(compatible;+MojeekBot/0.6;++https://www.mojeek.com/bot.html)
  • Speedy Spider / EntireWeb Search
  • http://www.entireweb.com
  • Speedy+Spider+(http://www.entireweb.com)
  • GenieO
  • http://www.genieo.com/webfilter.html
  • Mozilla/5.0+(compatible;+Genieo/1.0+http://www.genieo.com/webfilter.html)
  • scrapy.org
  • by default ROBOTSTXT_OBEY is set to false
  • http://doc.scrapy.org/en/latest/topics/settings.html
  • Gigablast Open Source Search Engine/Crawler/Spider
  • You can ignore robots.txt
  • http://gigablast.com/admin.html
  • GigablastOpenSource/1.0
  • Semrush Bot
  • http://www.semrush.com/bot.html
  • Mozilla/5.0+(compatible;+SemrushBot-SA/0.97;++http://www.semrush.com/bot.html)
  • Proximic
  • http://www.proximic.com/info/spider.php
  • Mozilla/5.0+(compatible;+proximic;++http://www.proximic.com/info/spider.php)
  • AppleBot
  • Used by Siri/Spotlight suggestions (commented out for now)
  • http://www.apple.com/go/applebot
  • Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_1)+AppleWebKit/600.2.5+(KHTML,+like+Gecko)+Version/8.0.2+Safari/600.2.5+(Applebot/0.1;++http://www.apple.com/go/applebot)
  • Mozilla/5.0+(iPhone;+CPU+iPhone+OS+8_1+like+Mac+OS+X)+AppleWebKit/600.1.4+(KHTML,+like+Gecko)+Version/8.0+Mobile/12B410+Safari/600.1.4+(Applebot/0.1;++http://www.apple.com/go/applebot)
  • Mozilla/5.0+(compatible;+Applebot/0.3;++http://www.apple.com/go/applebot)
  • User-agent: applebot
  • Disallow: /
  • CCBot
  • http://commoncrawl.org/faq/
  • CCBot/2.0+(http://commoncrawl.org/faq/)
  • Plukkie
  • http://www.botje.com/plukkie.htm
  • Mozilla/5.0+(compatible;+Plukkie/1.5;+http://www.botje.com/plukkie.htm)
  • Lipperhey Kaus Australis
  • https://www.lipperhey.com/en/about/
  • Mozilla/5.0+(compatible;+Lipperhey-Kaus-Australis/5.0;++https://www.lipperhey.com/en/about/)
  • oBot
  • http://filterdb.iss.net/crawler/
  • Mozilla/5.0+(compatible;+oBot/2.3.1;+http://filterdb.iss.net/crawler/)
  • Mozilla/5.0+(compatible;+oBot/2.3.1;++http://filterdb.iss.net/crawler/)
  • Screaming Frog SEO Spider
  • Downloaded software, robots.txt can be ignored
  • http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
  • Screaming+Frog+SEO+Spider/2.55
  • Screaming+Frog+SEO+Spider/3.3
  • VegeBot
  • http://www.exactbot.com/vegebot/index.html
  • ScreenerBot
  • ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
  • Garlik from Experian
  • GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
  • http-kit/2.0
  • BusinessBot: Nathan@lead-caddy.com
  • FAST Enterprise Crawler/5.3.4 (crawler@fast.no)
  • Nutch
  • http://nutch.apache.org/bot.html
  • Nutch+Spider/Nutch-1.5
  • mycrowl/Nutch-1.9
  • nutch+crawler/Nutch-1.9
  • sky+nutch+crawler/Nutch-1.9
  • Kraken/Nutch-2.2.1+(Nutch+crawler+launched+by+Integral+Ad+Science,+Inc.;+TBD;+TBD)
  • Open Link Profiler
  • Mozilla/5.0+(compatible;+spbot/4.4.2;++http://OpenLinkProfiler.org/bot+)
  • http://OpenLinkProfiler.org/bot
  • James Bot
  • Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.8.1.6)+Gecko/20070725+Firefox/2.0.0.6+-+James+BOT+-+WebCrawler+http://cognitiveseo.com/bot.html
  • http://cognitiveseo.com/bot.html
  • Qwantify
  • Mozilla/5.0+(compatible;+Qwantify/2.1n;++https://www.qwant.com/)/*
  • Mozilla/5.0+(compatible;+Qwantify/2.0n;++https://www.qwant.com/)/*
  • https://www.qwant.com/
  • Kraken
  • Mozilla/5.0+(compatible;+Kraken/0.1;+http://linkfluence.net/;+bot@linkfluence.net)
  • http://linkfluence.net/
  • Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10.9;+rv:28.0)+Gecko/20100101+Firefox/28.0+(FlipboardProxy/1.1;++http://flipboard.com/browserproxy)
  • http://flipboard.com/browserproxy
  • Mozilla/5.0+(compatible;+linkdexbot/2.0;++http://www.linkdex.com/bots/)
  • http://www.linkdex.com/bots/
  • Mozilla/5.0+(compatible;+MSIE+10.0;+Windows+NT+6.1;+Trident/6.0)+SiteCheck-sitecrawl+by+Siteimprove.com
  • Siteimprove.com
  • Mozilla/5.0+(compatible;+MegaIndex.ru/2.0;++http://megaindex.com/crawler)
  • http://megaindex.com/crawler
  • Prlog
  • Mozilla/5.0+(compatible;+Prlog/1.0;++http://prlog.ru/)
  • http://prlog.ru/
  • Mozilla/5.0+(compatible;+aiHitBot/2.9;++https://www.aihitdata.com/about)
  • https://www.aihitdata.com/about
  • KinglandSystemsCorp/KinglandSystemsCorp-crawler-2.0.1+(A+prototype+nutch+crawler+configuration+from+Kingland+Systems;+http://www.kingland.com;+kyle+at+kingland+dot+com)
  • http://www.kingland.com
  • Mozilla/5.0+(compatible;+SitemapBot/0.4;++http://www.SitemapBot.com;+SitemapBot@stmp.com)
  • http://www.sitemapbot.com
  • Mozilla/5.0+(compatible;+Googlebot/2.1;+https://www.deepcrawl.com/bot)
  • https://www.deepcrawl.com/bot
  • Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:38.0)+Gecko/20100101+Firefox/38.0+AlexaToolbar/alxf-2.21
  • http://www.alexa.com/toolbar
  • User-agent: AlexaToolbar
  • Disallow: /
  • Mozilla/5.0+(compatible;+Yeti/1.1;++http://help.naver.com/robots/)
  • http://help.naver.com/robots/
  • Mozilla/5.0+(Windows+NT+6.3;+WOW64;+rv:36.0)+Gecko/20100101+Firefox/36.0+(NetShelter+ContentScan,+contact+abuse@inpwrd.com+for+information)
  • VSE/1.0+(rabraham@multiview.com)
  • LSSRocketCrawler/1.0 LightspeedSystems
  • http://lightspeedsystems.com
  • Mozilla/5.0+(compatible;+idmarch+Automatic.beta/1.3;++http://www.idmarch.org/bot.html)
  • http://www.idmarch.org/bot.html
  • Mozilla/5.0+(compatible;+CukBot;+Not+a+spammer;+++https://www.companiesintheuk.co.uk/bot.html)
  • https://www.companiesintheuk.co.uk/bot.html
  • Mozilla/5.0+(compatible;+ExpertSearchSpider++http://www.expertsearch.nl/spider)
  • http://www.expertsearch.nl/spider
  • Mozilla/5.0+(compatible;+Goodzer/2.0;+crawler@goodzer.com)
  • WeSEE
  • http://www.wesee.com/
  • Symfony2+BrowserKit
  • rogerbot/1.0+(http://moz.com/help/pro/what-is-rogerbot-,+rogerbot-crawler+shiny@moz.com)
  • http://moz.com/help/pro/what-is-rogerbot-
  • bitlybot
  • http://bit.ly ??
  • Mozilla/5.0+(Windows;+U;+Windows+NT+6.0;+en-GB;+rv:1.0;+trendictionbot0.5.0;+trendiction+search;+http://www.trendiction.de/bot;+please+let+us+know+of+any+problems;+web+at+trendiction.com)+Gecko/20071127+Firefox/3.0.0.11
  • http://www.trendiction.de/bot
  • SearchMassive+internal+links+crawler/Nutch-1.10+(Crawl+and+collect+internal+links+from+websites)
  • spiderbot
  • yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+Europe/de)+http://yacy.net/bot.html
  • yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+America/en)+http://yacy.net/bot.html
  • AboutUsBot/Harpy+(Website+Analysis;+http://www.aboutus.org/Aboutus:Bot;+help@aboutus.org)
  • http://www.aboutus.org/Aboutus:Bot
  • http://www.nominet.org.uk/privacypolicy
  • Mozilla/5.0+(compatible;+DotBot/1.1;+http://www.opensiteexplorer.org/dotbot,+help@moz.com)
  • http://www.opensiteexplorer.org/dotbot
  • BrightEdge
  • Perl-Win32::Internet/0.087
  • Twitterbot/1.0
  • Xenu Link Sleuth 1.3.4
  • Xenu Link Sleuth/1.3.8
  • Go language
  • Go+1.1+package+http
  • wotbox.com
  • http://www.wotbox.com/bot/
  • Wotbox/2.01+(+http://www.wotbox.com/bot/)
  • yandex
  • http://yandex.com/bots
  • Mozilla/5.0+(compatible;+YandexBot/3.0;++http://yandex.com/bots)
  • Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)
  • http://help.yahoo.com/help/us/ysearch/slurp
  • User-agent: Yahoo! Slurp
  • Disallow: /
  • Yahoo Japan
  • http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/
  • Y!J-ASR/0.1+crawler+(http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
  • crazywebcrawler.com
  • crazywebcrawler.com
  • Dow Jones Searchbot
  • Mozilla/5.0+(compatible;+Dow+Jones+Searchbot)
  • WeCrawlForThePeace - We are not Evil
  • WeLikeLinks - WeAreNotEvil
  • icevikatam/1.0
  • User-agent: icevikatam
  • Disallow: /
  • Toweya search engine (FR)
  • Toweyabot: toweya.com
  • TScoutBot 1.0
  • Wappalyzer
  • Chrome/49.0.2623.87 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
Back to top