Log In Sign Up

autotrader.com.au
robots.txt

Robots Exclusion Standard data for autotrader.com.au

Resource Scan

Scan Details

Site Domain autotrader.com.au
Base Domain autotrader.com.au
Scan Status Ok
Last Scan2024-09-18T14:40:23+00:00
Next Scan 2024-09-25T14:40:23+00:00

Last Scan

Scanned2024-09-18T14:40:23+00:00
URL https://autotrader.com.au/robots.txt
Redirect https://www.autotrader.com.au/robots.txt
Redirect Domain www.autotrader.com.au
Redirect Base autotrader.com.au
Domain IPs 13.210.167.86, 13.239.155.225, 13.54.155.10
Redirect IPs 23.32.29.106, 23.32.29.98
Response IP 23.32.29.106
Found Yes
Hash 479e71fd9236b497720ac20df90071a8a648316f21b303dd6cf156fba62d4e9a
SimHash b8109d0b4564

Groups

*

Rule Path
Allow /
Disallow *?store=*
Disallow *?sortBy=*
Disallow /*/*/*/*/*/*/*/*/*/
Disallow /for-sale*/unleaded*
Disallow /for-sale*/leaded*
Disallow /for-sale*/premium*
Disallow /for-sale*/other-fuel-type*
Back to top

Other Records

Field Value
sitemap https://www.autotrader.com.au/atxmlsitemap/sitemap/index.xml
Back to top

Comments

  • robots.txt
  • This file is to prevent the crawling and indexing of certain parts
  • of your site by web crawlers and spiders run by sites like Yahoo!
  • and Google. By telling these "robots" where not to go on your site,
  • you save bandwidth and server resources.
  • This file will be ignored unless it is at the root of your host:
  • Used: http://example.com/robots.txt
  • Ignored: http://example.com/site/robots.txt
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/robotstxt.html
Back to top