intelligentinvestor.com.au
robots.txt

Robots Exclusion Standard data for intelligentinvestor.com.au

Resource Scan

Scan Details

Site Domain intelligentinvestor.com.au
Base Domain intelligentinvestor.com.au
Scan Status Ok
Last Scan2024-11-11T20:52:07+00:00
Next Scan 2024-11-18T20:52:07+00:00

Last Scan

Scanned2024-11-11T20:52:07+00:00
URL https://intelligentinvestor.com.au/robots.txt
Redirect https://www.intelligentinvestor.com.au/robots.txt
Redirect Domain www.intelligentinvestor.com.au
Redirect Base intelligentinvestor.com.au
Domain IPs 20.53.115.182
Redirect IPs 18.155.68.127, 18.155.68.24, 18.155.68.60, 18.155.68.90, 2600:9000:23d2:7600:1a:303a:78c0:93a1, 2600:9000:23d2:7e00:1a:303a:78c0:93a1, 2600:9000:23d2:9400:1a:303a:78c0:93a1, 2600:9000:23d2:ba00:1a:303a:78c0:93a1, 2600:9000:23d2:bc00:1a:303a:78c0:93a1, 2600:9000:23d2:c000:1a:303a:78c0:93a1, 2600:9000:23d2:de00:1a:303a:78c0:93a1, 2600:9000:23d2:e00:1a:303a:78c0:93a1
Response IP 18.155.68.60
Found Yes
Hash 3d70660eb726941fe96c4775afff2667531fc1a4356068d0ea5acdfdf7ba4a1e
SimHash 1a5018578fb5

Groups

*

Rule Path
Disallow /identity/logon?returnUrl*
Disallow /home/_investment-news-articles
Disallow /home/_investment-ideas
Disallow /home/_asx-announcements
Disallow /home/_smh-guide
Disallow /home/_my-investsmart-widget
Disallow /search/_shares
Disallow /search/_Shares
Disallow /search/_announcements
Disallow /search/_Announcements
Disallow /search/_dividends
Disallow /search/_Dividends
Disallow /search/_funds
Disallow /search/_Funds
Disallow /search/_news
Disallow /search/_News
Disallow /investment-news/newsletter/
Disallow /*.asp$
Disallow /*_price-chart
Disallow /portfolio-manager/diversified-portfolios/_getdetails
Disallow /portfolio-manager/diversified-portfolios/_getholdings
Disallow /portfolio-manager/diversified-portfolios/_getdepositswithdrawls
Disallow /portfolio-manager/diversified-portfolios/_gettransactions
Disallow /portfolio-manager/diversified-portfolios/_getdividendsinterest
Disallow /portfolio-manager/diversified-portfolios/_getfees
Disallow /test/throwerror
Disallow /test/version
Disallow /*%26size%3D*
Disallow /resources/pdf/*
Disallow /account/*
Disallow /events/registermodel
Disallow /widgets/*
Disallow /callback/*
Disallow /oauth/*
Disallow /shares/_announcementsummaries*

Other Records

Field Value
crawl-delay 20

yandex
baiduspider
baidu
mail.ru_bot/2.0
mail.ru_bot
mail.ru
megaindex.ru/2.0
megaindex.ru
megaindex
vagabondo
exabot
bspider
screaming frog seo spider

Rule Path
Disallow /

twitterbot

Rule Path
Disallow

Other Records

Field Value
sitemap https://www.intelligentinvestor.com.au/sitemap-ii.xml
sitemap https://www.intelligentinvestor.com.au/sitemap-news.xml
sitemap https://www.intelligentinvestor.com.au/sitemap-shares.xml
sitemap https://www.intelligentinvestor.com.au/sitemap-indices.xml
sitemap https://www.intelligentinvestor.com.au/sitemap-videos.xml
sitemap https://www.intelligentinvestor.com.au/sitemap-floats.xml

Comments

  • Certain social media sites are whitelisted to allow crawlers to access page markup