investsmart.com.au
robots.txt

Robots Exclusion Standard data for investsmart.com.au

Resource Scan

Scan Details

Site Domain investsmart.com.au
Base Domain investsmart.com.au
Scan Status Ok
Last Scan2024-10-11T10:25:41+00:00
Next Scan 2024-10-18T10:25:41+00:00

Last Scan

Scanned2024-10-11T10:25:41+00:00
URL https://investsmart.com.au/robots.txt
Redirect https://www.investsmart.com.au/robots.txt
Redirect Domain www.investsmart.com.au
Redirect Base investsmart.com.au
Domain IPs 20.53.115.182
Redirect IPs 18.155.68.127, 18.155.68.24, 18.155.68.60, 18.155.68.90, 2600:9000:23d2:0:1a:303a:78c0:93a1, 2600:9000:23d2:2800:1a:303a:78c0:93a1, 2600:9000:23d2:7000:1a:303a:78c0:93a1, 2600:9000:23d2:7800:1a:303a:78c0:93a1, 2600:9000:23d2:8a00:1a:303a:78c0:93a1, 2600:9000:23d2:e600:1a:303a:78c0:93a1, 2600:9000:23d2:e800:1a:303a:78c0:93a1, 2600:9000:23d2:f000:1a:303a:78c0:93a1
Response IP 18.155.68.24
Found Yes
Hash c48b6c3ba77a50f066fc29abb1cab614b962e8c4d311834dc2fe6a7ed091d0b5
SimHash 1e301851cdb1

Groups

*

Rule Path
Disallow /identity/logon?returnUrl*
Disallow /home/_investment-news-articles
Disallow /home/_investment-ideas
Disallow /home/_asx-announcements
Disallow /home/_smh-guide
Disallow /home/_my-investsmart-widget
Disallow /search/_shares
Disallow /search/_Shares
Disallow /search/_announcements
Disallow /search/_Announcements
Disallow /search/_dividends
Disallow /search/_Dividends
Disallow /search/_funds
Disallow /search/_Funds
Disallow /search/_news
Disallow /search/_News
Disallow /investment-news/newsletter/
Disallow /*.asp$
Disallow /*_price-chart
Disallow /portfolio-manager/diversified-portfolios/_getdetails
Disallow /portfolio-manager/diversified-portfolios/_getholdings
Disallow /portfolio-manager/diversified-portfolios/_getdepositswithdrawls
Disallow /portfolio-manager/diversified-portfolios/_gettransactions
Disallow /portfolio-manager/diversified-portfolios/_getdividendsinterest
Disallow /portfolio-manager/diversified-portfolios/_getfees
Disallow /test/throwerror
Disallow /test/version
Disallow /*%26size%3D*
Disallow /resources/pdf/*
Disallow /account/*
Disallow /events/registermodel
Disallow /widgets/*
Disallow /*_related-articles*
Disallow /callback/*
Disallow /oauth/*

Other Records

Field Value
crawl-delay 20

yandex
baiduspider
baidu
mail.ru_bot/2.0
mail.ru_bot
mail.ru
megaindex.ru/2.0
megaindex.ru
megaindex
vagabondo
exabot
bspider
screaming frog seo spider

Rule Path
Disallow /

twitterbot

Rule Path
Disallow

Other Records

Field Value
sitemap https://www.investsmart.com.au/sitemap-is.xml
sitemap https://www.investsmart.com.au/sitemap-events.xml
sitemap https://www.investsmart.com.au/sitemap-myportfolio.xml
sitemap https://www.investsmart.com.au/sitemap-ourstory.xml
sitemap https://www.investsmart.com.au/sitemap-news.xml
sitemap https://www.investsmart.com.au/sitemap-model-portfolios.xml
sitemap https://www.investsmart.com.au/sitemap-videos.xml
sitemap https://www.investsmart.com.au/sitemap-etfs.xml

Comments

  • Certain social media sites are whitelisted to allow crawlers to access page markup