Log In Sign Up

devmarkets.net
robots.txt

Robots Exclusion Standard data for devmarkets.net

Resource Scan

Scan Details

Site Domain devmarkets.net
Base Domain devmarkets.net
Scan Status Ok
Last Scan2024-06-07T20:48:10+00:00
Next Scan 2024-06-14T20:48:10+00:00

Last Scan

Scanned2024-06-07T20:48:10+00:00
URL https://devmarkets.net/robots.txt
Domain IPs 13.227.254.25, 13.227.254.35, 13.227.254.47, 13.227.254.52
Response IP 13.227.254.52
Found Yes
Hash 50e7b1a7c2d9bb265757fed0263b2ff41854052e74d5e58f0989bd2fe241de9b
SimHash 3a0021c2c18a

Groups

twitterbot/1.0

Rule Path
Disallow /

mozilla/5.0 (compatible; googlebot/2.1; +http://www.google.com/bot.html)

Rule Path
Disallow /

googlebot/2.1 (+http://www.google.com/bot.html)

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

mozilla/5.0 (compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Rule Path
Disallow /

mozilla/5.0 (compatible; cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)

Rule Path
Disallow /

mozilla/5.0 (compatible; bomborabot/1.0; +http://www.bombora.com/bot)

Rule Path
Disallow /

a6-indexer/1.0 (http://www.a6corp.com/a6-web-scraping-policy/)

Rule Path
Disallow /

mozilla/5.0 (compatible; ahrefsbot/5.1; +http://ahrefs.com/robot/)

Rule Path
Disallow /

mozilla/5.0 (compatible; genieo/1.0 http://www.genieo.com/webfilter.html)

Rule Path
Disallow /

mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)

Rule Path
Disallow /

mozilla/5.0 (compatible; yandexbot/3.0; +http://yandex.com/bots)

Rule Path
Disallow /

mozilla/5.0 (iphone; cpu iphone os 7_0 like mac os x) applewebkit/537.51.1 (khtml, like gecko) version/7.0 mobile/11a465 safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)

Rule Path
Disallow /

mozilla/5.0 (linux; android 7.0;) applewebkit/537.36 (khtml, like gecko) mobile safari/537.36 (compatible; petalbot;+https://webmaster.petalsearch.com/site/petalbot)

Rule Path
Disallow /

mozilla/5.0 applewebkit/537.36 (khtml, like gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) chrome/103.0.5060.134 safari/537.36

Rule Path
Disallow /

mozilla/5.0 (compatible; grapeshotcrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)

Rule Path
Disallow /

criteobot/0.1 (+https://www.criteo.com/criteo-crawler/)

Rule Path
Disallow

sirdatabot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

acicrawler/1.0

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

mozilla/5.0 (compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

unwindfetchor/1.0

Rule Path
Disallow /

faraday v0.9.2

Rule Path
Disallow /

faraday v0.11.0

Rule Path
Disallow /

mozilla/5.0+(compatible; yahoo! slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Rule Path
Disallow /

yahoo! slurp

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

metainspector/4.6.0

Rule Path
Disallow /

pcore-http/v0.25.0

Rule Path
Disallow /

zmeu

Rule Path
Disallow /

mozilla/5.0 (compatible; dotbot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

mozilla/5.0 (compatible; semrushbot-si/0.97; +http://www.semrush.com/bot.html)

Rule Path
Disallow /

mozilla/5.0 (compatible; semrushbot/1.2~bl; +http://www.semrush.com/bot.html)

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

mozilla/5.0 (compatible; marketwirebot/1.0; +http://www.marketwire.com)

Rule Path
Disallow /

istellabot/t.1.13

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

mozilla/5.0 (x11; compatible; semantic-visions.com crawler; httpclient 3.1)

Rule Path
Disallow /

mozilla/5.0 (compatible; paperlibot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

mozilla/5.0 ( compatible ; veooz/1.0 ; +http://www.veooz.com/veoozbot.html)

Rule Path
Disallow /

dlvr.it/1.0 ( http://dlvr.it/)

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

socialrankiobot; http://socialrank.io/about

Rule Path
Disallow /

mozilla/5.0 (compatible; inoreader.com-like feedfetcher-google)

Rule Path
Disallow /

mozilla/5.0 (compatible; embedly/0.2; http://support.embed.ly/)

Rule Path
Disallow /

zoominfobot (zoominfobot at zoominfo dot com)

Rule Path
Disallow /

livelapbot

Rule Path
Disallow /

mozilla/5.0 (compatible; mj12bot/v1.4.8; http://mj12bot.com/)

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

pinterest

Rule Path
Disallow /

mozilla/5.0 (compatible; mojeekbot/0.6; +https://www.mojeek.com/bot.html)

Rule Path
Disallow /

bubing

Rule Path
Disallow /

*

Rule Path
Disallow /admin/

*

Rule Path
Disallow /portal-widgets/

*

Rule Path
Disallow /c/

mozilla/5.0 (compatible; taboolabot/3.7; +http://www.taboola.com)

Rule Path
Disallow /admin/

bingbot

Rule Path
Disallow

mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

Rule Path
Disallow

turnitinbot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

riddler

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://talkmarkets.com/sitemap.xml
sitemap https://talkmarkets.com/sitemap.xml
sitemap https://devmarkets.net/sitemap.xml
Back to top

Comments

  • Allow crawlers to discover sitemap
Back to top