Log In Sign Up

waholidayguide.com.au
robots.txt

Robots Exclusion Standard data for waholidayguide.com.au

Resource Scan

Scan Details

Site Domain waholidayguide.com.au
Base Domain waholidayguide.com.au
Scan Status Ok
Last Scan2024-10-20T19:14:53+00:00
Next Scan 2024-11-19T19:14:53+00:00

Last Scan

Scanned2024-10-20T19:14:53+00:00
URL https://waholidayguide.com.au/robots.txt
Redirect https://www.waholidayguide.com.au/robots.txt
Redirect Domain www.waholidayguide.com.au
Redirect Base waholidayguide.com.au
Domain IPs 13.238.77.79
Redirect IPs 13.33.30.110, 13.33.30.27, 13.33.30.32, 13.33.30.94, 2600:9000:229f:1800:b:ae74:2240:93a1, 2600:9000:229f:400:b:ae74:2240:93a1, 2600:9000:229f:4400:b:ae74:2240:93a1, 2600:9000:229f:7800:b:ae74:2240:93a1, 2600:9000:229f:a000:b:ae74:2240:93a1, 2600:9000:229f:be00:b:ae74:2240:93a1, 2600:9000:229f:e000:b:ae74:2240:93a1, 2600:9000:229f:e800:b:ae74:2240:93a1
Response IP 13.33.30.27
Found Yes
Hash 4f5b150ba2ab62117e69b151eda79701f7b56f1cb8dea0ae97127125353f9cda
SimHash b25e1d49c1e3

Groups

*

Rule Path
Disallow /administrator/
Disallow /cli/
Disallow /component/
Disallow /components/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /libraries/
Disallow /logs/
Disallow /modules/
Disallow /plugins/
Disallow /tmp/

yandex
moget
ichiro
naverbot
yeti
baiduspider
baiduspider-video
baiduspider-image
sogou spider
youdaobot
mj12bot
grub-client
npbot
seokicks-robot
speedy
sogou web spider
ezooms
magpie-crawler
yandeximages
yodaobot
nerdbynature.bot
discobot
knowaboutbot
unwindfetchor
sitecheck.internetseer.com
zealbot
msiecrawler
webreaper
sitesnagger
webstripper
webcopier
fetch
offline explorer
teleport
teleportpro
webzip
linko
httrack
microsoft.url.control
xenu
larbin
libwww
zyborg
download ninja
mail.ru_bot
ahrefsbot
spbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap http://www.waholidayguide.com.au/sitemap.xml
Back to top

Comments

  • If the Joomla site is installed within a folder such as at
  • e.g. www.example.com/joomla/ the robots.txt file MUST be
  • moved to the site root at e.g. www.example.com/robots.txt
  • AND the joomla folder name MUST be prefixed to the disallowed
  • path, e.g. the Disallow rule for the /administrator/ folder
  • MUST be changed to read Disallow: /joomla/administrator/
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/orig.html
  • For syntax checking, see:
  • http://www.sxw.org.uk/computing/robots/check.html
  • 9597
  • This is disallowing some links from Google
  • User-agent: *
  • Disallow: /*?*
Back to top

Warnings

  • 1 invalid line.
Back to top