Log In Sign Up

freebiesupply.com
robots.txt

Robots Exclusion Standard data for freebiesupply.com

Resource Scan

Scan Details

Site Domain freebiesupply.com
Base Domain freebiesupply.com
Scan Status Ok
Last Scan2024-11-13T22:16:34+00:00
Next Scan 2024-11-20T22:16:34+00:00

Last Scan

Scanned2024-11-13T22:16:34+00:00
URL https://freebiesupply.com/robots.txt
Domain IPs 104.131.48.134
Response IP 104.131.48.134
Found Yes
Hash 8144b3b761edd1068081676c7d0d647dc8e52868e40461b40d3834755c5c5348
SimHash 70d95b45d2c2

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /logos/*/page/*
Disallow /author/ylppuseibeer/page/*
Disallow /tag/*
Disallow /freebiesupply.html
Allow /wp-admin/admin-ajax.php

msnbot

Rule Path
Disallow */page/*/?s=*
Disallow /s/

Other Records

Field Value
crawl-delay 120

ahrefsbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

smtbot

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

seebot

Rule Path
Disallow /

adbeat_bot

Rule Path
Disallow /

meanpathbot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

speedy spider

Rule Path
Disallow /

genieo

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

gigablastopensource

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

lipperhey

Rule Path
Disallow /

obot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

vegebot

Rule Path
Disallow /

screenerbot

Rule Path
Disallow /

garlik

Rule Path
Disallow /

businessbot

Rule Path
Disallow /

fast enterprise

Rule Path
Disallow /

nutch

Rule Path
Disallow /

spbot

Rule Path
Disallow /

jamesbot

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

kraken

Rule Path
Disallow /

linkdex

Rule Path
Disallow /

sitecheck-sitecrawl by siteimprove.com

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

prlog

Rule Path
Disallow /

aihitbot

Rule Path
Disallow /

kinglandsystemscorp

Rule Path
Disallow /

sitemapbot

Rule Path
Disallow /

deepcrawl

Rule Path
Disallow /

heritrix

Rule Path
Disallow /

yeti

Rule Path
Disallow /

netshelter

Rule Path
Disallow /

lssrocketcrawler

Rule Path
Disallow /

idmarch

Rule Path
Disallow /

daumoa

Rule Path
Disallow /

cpython

Rule Path
Disallow /

cukbot

Rule Path
Disallow /

expertsearchspider

Rule Path
Disallow /

goodzer

Rule Path
Disallow /

wesee

Rule Path
Disallow /

ltx71

Rule Path
Disallow /

symfony2

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

searchmassive

Rule Path
Disallow /

spiderbot

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

aboutusbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

xenu link sleuth

Rule Path
Disallow /

go 1.1

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

domain re-animator bot

Rule Path
Disallow /

crazywebcrawler-spider

Rule Path
Disallow /

crazywebcrawler

Rule Path
Disallow /

wecrawforthepeace

Rule Path
Disallow /

welikelinks

Rule Path
Disallow /

toweyabot

Rule Path
Disallow /

tscoutbot

Rule Path
Disallow /

wappalyzer

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

seoscanners

Rule Path
Disallow /

crawl.sogou.com

Rule Path
Disallow /

sogouspider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

semanticscholarbot

Rule Path
Disallow /

mail.ru_bot/2.0

Rule Path
Disallow /

ias_crawler

Rule Path
Disallow /

coccocbot-image/1.0

Rule Path
Disallow /

barkrowler/0.9

Rule Path
Disallow /

istellabot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://freebiesupply.com/sitemap.xml
Back to top

Comments

  • bing.com
  • ahrefs.com
  • http://ahrefs.com/robot/
  • Mozilla/5.0+(compatible;+AhrefsBot/5.0;++http://ahrefs.com/robot/)
  • exabot.com
  • Mozilla/5.0+(compatible;+Exabot/3.0;++http://www.exabot.com/go/robot)
  • http://www.exabot.com/go/robot
  • Riddler
  • Riddler (http://riddler.io/about)
  • http://riddler.io/about
  • SMTBot
  • http://www.similartech.com/smtbot
  • Mozilla/5.0+(compatible;+SMTBot/1.0;++http://www.similartech.com/smtbot)
  • ScoutJet
  • http://www.scoutjet.com
  • Mozilla/5.0+(compatible;+ScoutJet;++http://www.scoutjet.com/)
  • Mozilla/5.0+(compatible;+Blekkobot;+ScoutJet;++http://blekko.com/about/blekkobot)
  • Majestic Bot (MJ12bot)
  • http://www.majestic12.co.uk/bot.php
  • Mozilla/5.0+(compatible;+MJ12bot/v1.4.5;+http://www.majestic12.co.uk/bot.php?+)
  • SeeBot
  • http://www.seegnify.com/bot
  • seebot/2.0 (+http://www.seegnify.com/bot)
  • AdBeat Bot
  • https://www.adbeat.com/operation_policy
  • adbeat_bot
  • MeanPathBot
  • http://www.meanpath.com/meanpathbot.html
  • Mozilla/5.0+(compatible;+meanpathbot/1.0;++http://www.meanpath.com/meanpathbot.html)
  • Mojeek Bot
  • https://www.mojeek.com/bot.html
  • Mozilla/5.0+(compatible;+MojeekBot/0.6;++https://www.mojeek.com/bot.html)
  • Speedy Spider / EntireWeb Search
  • http://www.entireweb.com
  • Speedy+Spider+(http://www.entireweb.com)
  • GenieO
  • http://www.genieo.com/webfilter.html
  • Mozilla/5.0+(compatible;+Genieo/1.0+http://www.genieo.com/webfilter.html)
  • scrapy.org
  • by default ROBOTSTXT_OBEY is set to false
  • http://doc.scrapy.org/en/latest/topics/settings.html
  • Gigablast Open Source Search Engine/Crawler/Spider
  • You can ignore robots.txt
  • http://gigablast.com/admin.html
  • GigablastOpenSource/1.0
  • Semrush Bot
  • http://www.semrush.com/bot.html
  • Mozilla/5.0+(compatible;+SemrushBot-SA/0.97;++http://www.semrush.com/bot.html)
  • Proximic
  • http://www.proximic.com/info/spider.php
  • Mozilla/5.0+(compatible;+proximic;++http://www.proximic.com/info/spider.php)
  • CCBot
  • http://commoncrawl.org/faq/
  • CCBot/2.0+(http://commoncrawl.org/faq/)
  • Plukkie
  • http://www.botje.com/plukkie.htm
  • Mozilla/5.0+(compatible;+Plukkie/1.5;+http://www.botje.com/plukkie.htm)
  • Lipperhey Kaus Australis
  • https://www.lipperhey.com/en/about/
  • Mozilla/5.0+(compatible;+Lipperhey-Kaus-Australis/5.0;++https://www.lipperhey.com/en/about/)
  • oBot
  • http://filterdb.iss.net/crawler/
  • Mozilla/5.0+(compatible;+oBot/2.3.1;+http://filterdb.iss.net/crawler/)
  • Mozilla/5.0+(compatible;+oBot/2.3.1;++http://filterdb.iss.net/crawler/)
  • Screaming Frog SEO Spider
  • Downloaded software, robots.txt can be ignored
  • http://www.screamingfrog.co.uk/seo-spider/user-guide/general/
  • Screaming+Frog+SEO+Spider/2.55
  • Screaming+Frog+SEO+Spider/3.3
  • VegeBot
  • http://www.exactbot.com/vegebot/index.html
  • ScreenerBot
  • ScreenerBot Crawler Beta 2.0 (+http://www.ScreenerBot.com)
  • Garlik from Experian
  • GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)
  • http-kit/2.0
  • BusinessBot: Nathan@lead-caddy.com
  • FAST Enterprise Crawler/5.3.4 (crawler@fast.no)
  • Nutch
  • http://nutch.apache.org/bot.html
  • Nutch+Spider/Nutch-1.5
  • mycrowl/Nutch-1.9
  • nutch+crawler/Nutch-1.9
  • sky+nutch+crawler/Nutch-1.9
  • Kraken/Nutch-2.2.1+(Nutch+crawler+launched+by+Integral+Ad+Science,+Inc.;+TBD;+TBD)
  • Open Link Profiler
  • Mozilla/5.0+(compatible;+spbot/4.4.2;++http://OpenLinkProfiler.org/bot+)
  • http://OpenLinkProfiler.org/bot
  • James Bot
  • Mozilla/5.0+(Windows;+U;+Windows+NT+5.1;+en-US;+rv:1.8.1.6)+Gecko/20070725+Firefox/2.0.0.6+-+James+BOT+-+WebCrawler+http://cognitiveseo.com/bot.html
  • http://cognitiveseo.com/bot.html
  • Qwantify
  • Mozilla/5.0+(compatible;+Qwantify/2.1n;++https://www.qwant.com/)/*
  • Mozilla/5.0+(compatible;+Qwantify/2.0n;++https://www.qwant.com/)/*
  • https://www.qwant.com/
  • Kraken
  • Mozilla/5.0+(compatible;+Kraken/0.1;+http://linkfluence.net/;+bot@linkfluence.net)
  • http://linkfluence.net/
  • Mozilla/5.0+(compatible;+linkdexbot/2.0;++http://www.linkdex.com/bots/)
  • http://www.linkdex.com/bots/
  • Mozilla/5.0+(compatible;+MSIE+10.0;+Windows+NT+6.1;+Trident/6.0)+SiteCheck-sitecrawl+by+Siteimprove.com
  • Siteimprove.com
  • Mozilla/5.0+(compatible;+MegaIndex.ru/2.0;++http://megaindex.com/crawler)
  • http://megaindex.com/crawler
  • Prlog
  • Mozilla/5.0+(compatible;+Prlog/1.0;++http://prlog.ru/)
  • http://prlog.ru/
  • Mozilla/5.0+(compatible;+aiHitBot/2.9;++https://www.aihitdata.com/about)
  • https://www.aihitdata.com/about
  • KinglandSystemsCorp/KinglandSystemsCorp-crawler-2.0.1+(A+prototype+nutch+crawler+configuration+from+Kingland+Systems;+http://www.kingland.com;+kyle+at+kingland+dot+com)
  • http://www.kingland.com
  • Mozilla/5.0+(compatible;+SitemapBot/0.4;++http://www.SitemapBot.com;+SitemapBot@stmp.com)
  • http://www.sitemapbot.com
  • Mozilla/5.0+(compatible;+Googlebot/2.1;+https://www.deepcrawl.com/bot)
  • https://www.deepcrawl.com/bot
  • Mozilla/5.0+(compatible;+heritrix/3.3.0-SNAPSHOT-20150302-2206++http://127.0.0.1)
  • https://webarchive.jira.com/wiki/display/Heritrix/Heritrix
  • Mozilla/5.0+(compatible;+Yeti/1.1;++http://help.naver.com/robots/)
  • http://help.naver.com/robots/
  • Mozilla/5.0+(Windows+NT+6.3;+WOW64;+rv:36.0)+Gecko/20100101+Firefox/36.0+(NetShelter+ContentScan,+contact+abuse@inpwrd.com+for+information)
  • VSE/1.0+(rabraham@multiview.com)
  • LSSRocketCrawler/1.0 LightspeedSystems
  • http://lightspeedsystems.com
  • Mozilla/5.0+(compatible;+idmarch+Automatic.beta/1.3;++http://www.idmarch.org/bot.html)
  • http://www.idmarch.org/bot.html
  • Mozilla/5.0+(compatible;+MSIE+or+Firefox+mutant;+not+on+Windows+server;)+Daumoa/4.0
  • Mozilla/5.0+(compatible;+MSIE+or+Firefox+mutant;+not+on+Windows+server;)+Daumoa/4.0+(Following+Mediapartners-Google)
  • python-requests/2.4.3+CPython/3.4.2+Linux/3.14.42-31.38.amzn1.x86
  • python-requests/2.0.1+CPython/2.7.6+Linux/3.13.0-36-generic
  • python-requests/2.7.0 CPython/2.7.3 Linux/3.12-0.bpo.1-amd64
  • Mozilla/5.0+(compatible;+CukBot;+Not+a+spammer;+++https://www.companiesintheuk.co.uk/bot.html)
  • https://www.companiesintheuk.co.uk/bot.html
  • Mozilla/5.0+(compatible;+ExpertSearchSpider++http://www.expertsearch.nl/spider)
  • http://www.expertsearch.nl/spider
  • Mozilla/5.0+(compatible;+Goodzer/2.0;+crawler@goodzer.com)
  • WeSEE
  • http://www.wesee.com/
  • http://ltx71.com/
  • ltx71 - (http://ltx71.com/)
  • Symfony2+BrowserKit
  • rogerbot/1.0+(http://moz.com/help/pro/what-is-rogerbot-,+rogerbot-crawler+shiny@moz.com)
  • http://moz.com/help/pro/what-is-rogerbot-
  • Mozilla/5.0+(Windows;+U;+Windows+NT+6.0;+en-GB;+rv:1.0;+trendictionbot0.5.0;+trendiction+search;+http://www.trendiction.de/bot;+please+let+us+know+of+any+problems;+web+at+trendiction.com)+Gecko/20071127+Firefox/3.0.0.11
  • http://www.trendiction.de/bot
  • SearchMassive+internal+links+crawler/Nutch-1.10+(Crawl+and+collect+internal+links+from+websites)
  • spiderbot
  • yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+Europe/de)+http://yacy.net/bot.html
  • yacybot+(/global;+amd64+Linux+3.16.0-4-amd64;+java+1.7.0_79;+America/en)+http://yacy.net/bot.html
  • AboutUsBot/Harpy+(Website+Analysis;+http://www.aboutus.org/Aboutus:Bot;+help@aboutus.org)
  • http://www.aboutus.org/Aboutus:Bot
  • http://www.nominet.org.uk/privacypolicy
  • Mozilla/5.0+(compatible;+DotBot/1.1;+http://www.opensiteexplorer.org/dotbot,+help@moz.com)
  • http://www.opensiteexplorer.org/dotbot
  • Mozilla/5.0+(compatible;+BLEXBot/1.0;++http://webmeup-crawler.com/)
  • http://webmeup-crawler.com/
  • Perl-Win32::Internet/0.087
  • Twitterbot/1.0
  • Xenu Link Sleuth 1.3.4
  • Xenu Link Sleuth/1.3.8
  • Go language
  • Go+1.1+package+http
  • wotbox.com
  • http://www.wotbox.com/bot/
  • Wotbox/2.01+(+http://www.wotbox.com/bot/)
  • yandex
  • http://yandex.com/bots
  • Mozilla/5.0+(compatible;+YandexBot/3.0;++http://yandex.com/bots)
  • Mozilla/5.0+(compatible;+Yahoo!+Slurp;+http://help.yahoo.com/help/us/ysearch/slurp)
  • http://help.yahoo.com/help/us/ysearch/slurp
  • User-agent: Yahoo! Slurp
  • Disallow: /
  • domainreanimator.com
  • crazywebcrawler.com
  • crazywebcrawler.com
  • OpenVAS.org
  • User-agent: OpenVAS
  • Disallow: /
  • WeCrawlForThePeace - We are not Evil
  • WeLikeLinks - WeAreNotEvil
  • icevikatam/1.0
  • User-agent: icevikatam
  • Disallow: /
  • Toweya search engine (FR)
  • Toweyabot: toweya.com
  • TScoutBot 1.0
  • Wappalyzer
  • Chrome/49.0.2623.87 (compatible; Wappalyzer; +https://github.com/AliasIO/Wappalyzer)
  • http://cliqz.com/company/cliqzbot
  • http://napoveda.seznam.cz/en/seznambot-intro/
  • Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)
Back to top