Log In Sign Up

talkmarkets.com
robots.txt

Robots Exclusion Standard data for talkmarkets.com

Resource Scan

Scan Details

Site Domain talkmarkets.com
Base Domain talkmarkets.com
Scan Status Ok
Last Scan2024-06-14T03:34:45+00:00
Next Scan 2024-06-21T03:34:45+00:00

Last Scan

Scanned2024-06-14T03:34:45+00:00
URL https://talkmarkets.com/robots.txt
Domain IPs 13.33.30.120, 13.33.30.28, 13.33.30.60, 13.33.30.96, 2600:9000:229f:2a00:0:2e13:6780:93a1, 2600:9000:229f:3e00:0:2e13:6780:93a1, 2600:9000:229f:4e00:0:2e13:6780:93a1, 2600:9000:229f:5e00:0:2e13:6780:93a1, 2600:9000:229f:9600:0:2e13:6780:93a1, 2600:9000:229f:e800:0:2e13:6780:93a1, 2600:9000:229f:f000:0:2e13:6780:93a1, 2600:9000:229f:f800:0:2e13:6780:93a1
Response IP 13.33.30.28
Found Yes
Hash fdd1c8e691c6c03f4540d8a2bd9baacdf34432834ba2707d5ade05a6b8d02264
SimHash 7a102682c1a8

Groups

twitterbot/1.0

Rule Path
Allow /

mozilla/5.0 (compatible; yandexbot/3.0; +http://yandex.com/bots)

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot/3.0

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

mozilla/5.0 (compatible; baiduspider/2.0; +http://www.baidu.com/search/spider.html)

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

unwindfetchor/1.0

Rule Path
Disallow /

faraday v0.9.2

Rule Path
Disallow /

faraday v0.11.0

Rule Path
Disallow /

mozilla/5.0+(compatible; yahoo! slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Rule Path
Disallow /

yahoo! slurp

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

metainspector/4.6.0

Rule Path
Disallow /

pcore-http/v0.25.0

Rule Path
Disallow /

zmeu

Rule Path
Disallow /

mozilla/5.0 (compatible; dotbot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

mozilla/5.0 (compatible; semrushbot-si/0.97; +http://www.semrush.com/bot.html)

Rule Path
Disallow /

mozilla/5.0 (compatible; semrushbot/1.2~bl; +http://www.semrush.com/bot.html)

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

mozilla/5.0 (compatible; marketwirebot/1.0; +http://www.marketwire.com)

Rule Path
Disallow /

istellabot/t.1.13

Rule Path
Disallow /

scoutjet

Rule Path
Disallow /

mozilla/5.0 (x11; compatible; semantic-visions.com crawler; httpclient 3.1)

Rule Path
Disallow /

mozilla/5.0 (compatible; paperlibot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

siteexplorer

Rule Path
Disallow /

mozilla/5.0 ( compatible ; veooz/1.0 ; +http://www.veooz.com/veoozbot.html)

Rule Path
Disallow /

dlvr.it/1.0 ( http://dlvr.it/)

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

socialrankiobot; http://socialrank.io/about

Rule Path
Disallow /

mozilla/5.0 (compatible; embedly/0.2; http://support.embed.ly/)

Rule Path
Disallow /

zoominfobot (zoominfobot at zoominfo dot com)

Rule Path
Disallow /

livelapbot

Rule Path
Disallow /

mozilla/5.0 (compatible; mj12bot/v1.4.8; http://mj12bot.com/)

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

pinterest

Rule Path
Disallow /

mozilla/5.0 (compatible; mojeekbot/0.6; +https://www.mojeek.com/bot.html)

Rule Path
Disallow /

bubing

Rule Path
Disallow /

*

Rule Path
Disallow /admin/

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow /portal-widgets/

*

Rule Path
Disallow /c/

turnitinbot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

riddler

Rule Path
Disallow /

mozilla/5.0 (linux; android 5.0) applewebkit/537.36 (khtml, like gecko) mobile safari/537.36 (compatible; bytespider; spider-feedback@bytedance.com)

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://talkmarkets.com/sitemap.xml
sitemap https://talkmarkets.com/sitemap.xml
Back to top

Comments

  • Allow crawlers to discover sitemap
Back to top