Log In Sign Up

realestate.com.au
robots.txt

Robots Exclusion Standard data for realestate.com.au

Resource Scan

Scan Details

Site Domain realestate.com.au
Base Domain realestate.com.au
Scan Status Ok
Last Scan2024-06-22T07:21:59+00:00
Next Scan 2024-06-29T07:21:59+00:00

Last Scan

Scanned2024-06-22T07:21:59+00:00
URL https://realestate.com.au/robots.txt
Redirect https://www.realestate.com.au/robots.txt
Redirect Domain www.realestate.com.au
Redirect Base realestate.com.au
Domain IPs 104.69.168.168, 2600:1413:1:993::3413
Redirect IPs 104.69.168.168, 2600:1413:5000:381::3413, 2600:1413:5000:397::3413
Response IP 104.76.130.71
Found Yes
Hash e003660ac5ac7d4326d34eb7925a55325eefb89d30c20227f2dc0400b3de5273
SimHash 9c91451345c5

Groups

*

Rule Path
Disallow

googlebot-news

Rule Path
Disallow /
Allow /news/

yakazbot

Rule Path
Disallow /

trovitbot

Rule Path
Disallow /

eyeotabot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

criteobot/0.1

Rule Path
Disallow /

trendkite-akashic-crawler

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /
Back to top

Comments

  • In accessing or using any REA Group Website you agree that you will not use any automated device,
  • software, process or means to access, retrieve, scrape, or index any REA Group Website or any
  • content on any REA Group Website. Notwithstanding the foregoing, REA Group may permit automated
  • access to certain REA Group Website pages strictly for the purpose of including content in publicly
  • available general search engines. This does not include any such access by websites that
  • specifically aggregate property listings and/or information as part of their business. REA Group
  • strictly prohibits any automated access by these types of websites.
Back to top