Log In Sign Up

midfirst.com
robots.txt

Robots Exclusion Standard data for midfirst.com

Resource Scan

Scan Details

Site Domain midfirst.com
Base Domain midfirst.com
Scan Status Ok
Last Scan2025-05-24T20:00:58+00:00
Next Scan 2025-06-23T20:00:58+00:00

Last Scan

Scanned2025-05-24T20:00:58+00:00
URL https://midfirst.com/robots.txt
Redirect https://www.midfirst.com/robots.txt
Redirect Domain www.midfirst.com
Redirect Base midfirst.com
Domain IPs 45.60.0.172, 45.60.4.172
Redirect IPs 45.60.0.172
Response IP 45.60.0.172
Found Yes
Hash 1067bf83d9b5d392384a1db87b648eb56b920bb4d77c036693f245a1efc6c7a0
SimHash 70cb794d57c2

Groups

powermapper

Rule Path
Allow /

exabot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

seebot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

genieo

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

gigablastopensource

Rule Path
Disallow /

proximic

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

obot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

vegebot

Rule Path
Disallow /

screenerbot

Rule Path
Disallow /

garlik

Rule Path
Disallow /

businessbot

Rule Path
Disallow /

fast enterprise

Rule Path
Disallow /

nutch

Rule Path
Disallow /

spbot

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

kraken

Rule Path
Disallow /

flipboard

Rule Path
Disallow /

linkdex

Rule Path
Disallow /

sitecheck-sitecrawl by siteimprove.com

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

prlog

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

kinglandsystemscorp

Rule Path
Disallow /

sitemapbot

Rule Path
Disallow /

deepcrawl

Rule Path
Disallow /

yeti

Rule Path
Disallow /

netshelter

Rule Path
Disallow /

lssrocketcrawler

Rule Path
Disallow /

idmarch

Rule Path
Disallow /

cukbot

Rule Path
Disallow /

expertsearchspider

Rule Path
Disallow /

goodzer

Rule Path
Disallow /

wesee

Rule Path
Disallow /

symfony2

Rule Path
Disallow /

rogerbot

Rule Path
Allow /
Disallow /assets
Disallow /media
Disallow /images
Disallow /bundles
Disallow /App_Images
Disallow /locations

bitlybot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

searchmassive

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

aboutusbot

Rule Path
Disallow /

dotbot

Rule Path
Allow /
Disallow /assets
Disallow /media
Disallow /images
Disallow /bundles
Disallow /App_Images
Disallow /locations

brightedge

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

go 1.1

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

y!j-asr

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

crazywebcrawler

Rule Path
Disallow /

dow jones searchbot

Rule Path
Disallow /

wecrawlforthepeace

Rule Path
Disallow /

welikelinks

Rule Path
Disallow /

toweyabot

Rule Path
Disallow /

tscoutbot

Rule Path
Disallow /

wappalyzer

Rule Path
Disallow /

*

Rule Path
Disallow /*.axd?
Disallow /*.axd
Disallow /*%3A*
Disallow /*?*%27*
Disallow /page-not-found
Disallow /general-error
Disallow /cookies-not-enabled
Disallow /maintenance-page
Disallow /unsupported-browser
Disallow /complete
Disallow /online-banking-return/
Disallow /search/
Disallow /splashpages/*
Disallow /App_Images/*
Disallow /assets/*
Disallow /media/*
Disallow /documents/*
Disallow /locations/select-location
Disallow /locations/directions?*
Disallow /locations/temporary-operating-hours
Allow /locations/map-detail?*
Back to top

Other Records

Field Value
sitemap https://www.midfirst.com/SiteMap.xml
sitemap https://www.midfirst.com/SiteMap.xml
sitemap https://www.midfirst.com/SiteMap.xml
Back to top

Comments

  • /robots.txt file for http://www.midfirst.com/
  • SortSite
  • exabot.com
  • Mozilla/5.0+(compatible;+Exabot/3.0;++http://www.exabot.com/go/robot)
  • http://www.exabot.com/go/robot
  • Riddler
  • Riddler (http://riddler.io/about)
  • http://riddler.io/about
  • SMTBot
  • http://www.similartech.com/smtbot
  • Mozilla/5.0+(compatible;+SMTBot/1.0;++http://www.similartech.com/smtbot)
  • Majestic Bot (MJ12bot)
  • http://www.majestic12.co.uk/bot.php
  • Mozilla/5.0+(compatible;+MJ12bot/v1.4.5;+http://www.majestic12.co.uk/bot.php?+)
  • SeeBot
  • http://www.seegnify.com/bot
  • seebot/2.0 (+http://www.seegnify.com/bot)
  • Archive.org Bot
  • http://archive.org/details/archive.org_bot
  • Mozilla/5.0+(compatible;+archive.org_bot;+Wayback+Machine+Live+Record;++http://archive.org/details/archive.org_bot)
  • Mozilla/5.0+(compatible;+special_archiver/3.1.1++http://www.archive.org/details/archive.org_bot)
  • AdBeat Bot
  • https://www.adbeat.com/operation_policy
  • adbeat_bot
  • MeanPathBot
  • http://www.meanpath.com/meanpathbot.html
  • Mozilla/5.0+(compatible;+meanpathbot/1.0;++http://www.meanpath.com/meanpathbot.html)
  • Mojeek Bot
  • https://www.mojeek.com/bot.html
  • Mozilla/5.0+(compatible;+MojeekBot/0.6;++https://www.mojeek.com/bot.html)
  • Speedy Spider / EntireWeb Search
  • http://www.entireweb.com
  • Speedy+Spider+(http://www.entireweb.com)
  • GenieO
  • http://www.genieo.com/webfilter.html
  • Mozilla/5.0+(compatible;+Genieo/1.0+http://www.genieo.com/webfilter.html)
  • scrapy.org
  • by default ROBOTSTXT_OBEY is set to false
  • http://doc.scrapy.org/en/latest/topics/settings.html
  • Gigablast Open Source Search Engine/Crawler/Spider
  • You can ignore robots.txt
  • http://gigablast.com/admin.html
  • GigablastOpenSource/1.0
  • Proximic
  • http://www.proximic.com/info/spider.php
  • Mozilla/5.0+(compatible;+proximic;++http://www.proximic.com/info/spider.php)
  • AppleBot
  • Used by Siri/Spotlight suggestions (commented out for now)
  • http://www.apple.com/go/applebot
  • Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_10_1)+AppleWebKit/600.2.5+(KHTML,+like+Gecko)+Version/8.0.2+Safari/600.2.5+(Applebot/0.1;++http://www.apple.com/go/applebot)
  • Mozilla/5.0+(iPhone;+CPU+iPhone+OS+8_1+like+Mac+OS+X)+AppleWebKit/600.1.4+(KHTML,+like+Gecko)+Version/8.0+Mobile/12B410+Safari/600.1.4+(Applebot/0.1;++http://www.apple.com/go/applebot)
  • Mozilla/5.0+(compatible;+Applebot/0.3;++http://www.apple.com/go/applebot)
  • User-agent: applebot
  • Disallow: /
  • CCBot
  • http://commoncrawl.org/faq/
  • CCBot/2.0+(http://commoncrawl.org/faq/)
  • Plukkie
  • http://www.botje.com/plukkie.htm
  • Mozilla/5.0+(compatible;+Plukkie/1.5;+http://www.botje.com/plukkie.htm)
  • Lipperhey Kaus Australis
  • https://www.lipperhey.com/en/about/
  • Mozilla/5.0+(compatible;+Lipperhey-Kaus-Australis/5.0;++https://www.lipperhey.com/en/about/)
  • oBot
  • http://filterdb.iss.net/crawler/
  • Mozilla/5.0+(compatible;+oBot/2.3.1;+http://filterdb.iss.net/crawler/)
  • Mozilla/5.0+(compatible;+oBot/2.3.1;++http://filterdb.iss.net/crawler/)
  • Screaming Frog SEO Spider
  • Downloaded software, robots.txt can be ignored
  • http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
  • Screaming+Frog+SEO+Spider/2.55
  • Screaming+Frog+SEO+Spider/3.3
  • VegeBot
  • http://www.exactbot.com/vegebot/index.html
  • ScreenerBot
  • ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
  • Garlik from Experian
  • GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
  • http-kit/2.0
  • BusinessBot: Nathan@lead-caddy.com
  • FAST Enterprise Crawler/5.3.4 (crawler@fast.no)
  • Nutch
  • http://nutch.apache.org/bot.html
  • Nutch+Spider/Nutch-1.5
  • mycrowl/Nutch-1.9
  • nutch+crawler/Nutch-1.9
  • sky+nutch+crawler/Nutch-1.9
  • Kraken/Nutch-2.2.1+(Nutch+crawler+launched+by+Integral+Ad+Science,+Inc.;+TBD;+TBD)
  • Open Link Profiler
  • Mozilla/5.0+(compatible;+spbot/4.4.2;++http://OpenLinkProfiler.org/bot+)
  • http://OpenLinkProfiler.org/bot
  • James Bot
  • Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.8.1.6)+Gecko/20070725+Firefox/2.0.0.6+-+James+BOT+-+WebCrawler+http://cognitiveseo.com/bot.html
  • http://cognitiveseo.com/bot.html
  • Qwantify
  • Mozilla/5.0+(compatible;+Qwantify/2.1n;++https://www.qwant.com/)/*
  • Mozilla/5.0+(compatible;+Qwantify/2.0n;++https://www.qwant.com/)/*
  • https://www.qwant.com/
  • Kraken
  • Mozilla/5.0+(compatible;+Kraken/0.1;+http://linkfluence.net/;+bot@linkfluence.net)
  • http://linkfluence.net/
  • Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10.9;+rv:28.0)+Gecko/20100101+Firefox/28.0+(FlipboardProxy/1.1;++http://flipboard.com/browserproxy)
  • http://flipboard.com/browserproxy
  • Mozilla/5.0+(compatible;+linkdexbot/2.0;++http://www.linkdex.com/bots/)
  • http://www.linkdex.com/bots/
  • Mozilla/5.0+(compatible;+MSIE+10.0;+Windows+NT+6.1;+Trident/6.0)+SiteCheck-sitecrawl+by+Siteimprove.com
  • Siteimprove.com
  • Mozilla/5.0+(compatible;+MegaIndex.ru/2.0;++http://megaindex.com/crawler)
  • http://megaindex.com/crawler
  • Prlog
  • Mozilla/5.0+(compatible;+Prlog/1.0;++http://prlog.ru/)
  • http://prlog.ru/
  • Mozilla/5.0+(compatible;+aiHitBot/2.9;++https://www.aihitdata.com/about)
  • https://www.aihitdata.com/about
  • KinglandSystemsCorp/KinglandSystemsCorp-crawler-2.0.1+(A+prototype+nutch+crawler+configuration+from+Kingland+Systems;+http://www.kingland.com;+kyle+at+kingland+dot+com)
  • http://www.kingland.com
  • Mozilla/5.0+(compatible;+SitemapBot/0.4;++http://www.SitemapBot.com;+SitemapBot@stmp.com)
  • http://www.sitemapbot.com
  • Mozilla/5.0+(compatible;+Googlebot/2.1;+https://www.deepcrawl.com/bot)
  • https://www.deepcrawl.com/bot
  • Mozilla/5.0+(Windows+NT+6.1;+WOW64;+rv:38.0)+Gecko/20100101+Firefox/38.0+AlexaToolbar/alxf-2.21
  • http://www.alexa.com/toolbar
  • User-agent: AlexaToolbar
  • Disallow: /
  • Mozilla/5.0+(compatible;+Yeti/1.1;++http://help.naver.com/robots/)
  • http://help.naver.com/robots/
  • Mozilla/5.0+(Windows+NT+6.3;+WOW64;+rv:36.0)+Gecko/20100101+Firefox/36.0+(NetShelter+ContentScan,+contact+abuse@inpwrd.com+for+information)
  • VSE/1.0+(rabraham@multiview.com)
  • LSSRocketCrawler/1.0 LightspeedSystems
  • http://lightspeedsystems.com
  • Mozilla/5.0+(compatible;+idmarch+Automatic.beta/1.3;++http://www.idmarch.org/bot.html)
  • http://www.idmarch.org/bot.html
  • Mozilla/5.0+(compatible;+CukBot;+Not+a+spammer;+++https://www.companiesintheuk.co.uk/bot.html)
  • https://www.companiesintheuk.co.uk/bot.html
  • Mozilla/5.0+(compatible;+ExpertSearchSpider++http://www.expertsearch.nl/spider)
  • http://www.expertsearch.nl/spider
  • Mozilla/5.0+(compatible;+Goodzer/2.0;+crawler@goodzer.com)
  • WeSEE
  • http://www.wesee.com/
  • Symfony2+BrowserKit
  • rogerbot/1.0+(http://moz.com/help/pro/what-is-rogerbot-,+rogerbot-crawler+shiny@moz.com)
  • http://moz.com/help/pro/what-is-rogerbot-
  • bitlybot
  • http://bit.ly ??
  • Mozilla/5.0+(Windows;+U;+Windows+NT+6.0;+en-GB;+rv:1.0;+trendictionbot0.5.0;+trendiction+search;+http://www.trendiction.de/bot;+please+let+us+know+of+any+problems;+web+at+trendiction.com)+Gecko/20071127+Firefox/3.0.0.11
  • http://www.trendiction.de/bot
  • SearchMassive+internal+links+crawler/Nutch-1.10+(Crawl+and+collect+internal+links+from+websites)
  • spiderbot
  • yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+Europe/de)+http://yacy.net/bot.html
  • yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+America/en)+http://yacy.net/bot.html
  • AboutUsBot/Harpy+(Website+Analysis;+http://www.aboutus.org/Aboutus:Bot;+help@aboutus.org)
  • http://www.aboutus.org/Aboutus:Bot
  • http://www.nominet.org.uk/privacypolicy
  • Mozilla/5.0+(compatible;+DotBot/1.1;+http://www.opensiteexplorer.org/dotbot,+help@moz.com)
  • http://www.opensiteexplorer.org/dotbot
  • BrightEdge
  • Perl-Win32::Internet/0.087
  • Twitterbot/1.0
  • Xenu Link Sleuth 1.3.4
  • Xenu Link Sleuth/1.3.8
  • Go language
  • Go+1.1+package+http
  • wotbox.com
  • http://www.wotbox.com/bot/
  • Wotbox/2.01+(+http://www.wotbox.com/bot/)
  • yandex
  • http://yandex.com/bots
  • Mozilla/5.0+(compatible;+YandexBot/3.0;++http://yandex.com/bots)
  • Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)
  • http://help.yahoo.com/help/us/ysearch/slurp
  • User-agent: Yahoo! Slurp
  • Disallow: /
  • Yahoo Japan
  • http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/
  • Y!J-ASR/0.1+crawler+(http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)
  • crazywebcrawler.com
  • crazywebcrawler.com
  • Dow Jones Searchbot
  • Mozilla/5.0+(compatible;+Dow+Jones+Searchbot)
  • WeCrawlForThePeace - We are not Evil
  • WeLikeLinks - WeAreNotEvil
  • icevikatam/1.0
  • User-agent: icevikatam
  • Disallow: /
  • Toweya search engine (FR)
  • Toweyabot: toweya.com
  • TScoutBot 1.0
  • Wappalyzer
  • Chrome/49.0.2623.87 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
  • standard users
  • Disallow: /*?
  • Disallow: /*?*
  • Disallow: /*&*
Back to top