Log In Sign Up

hrpulse.co.za
robots.txt

Robots Exclusion Standard data for hrpulse.co.za

Resource Scan

Scan Details

Site Domain hrpulse.co.za
Base Domain hrpulse.co.za
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-09-23T02:15:00+00:00
Next Scan 2024-10-23T02:15:00+00:00

Last Successful Scan

Scanned2024-08-01T21:47:39+00:00
URL http://hrpulse.co.za/robots.txt
Domain IPs 41.86.108.254
Response IP 41.86.108.254
Found Yes
Hash 7088acb2237d1ec30adb0a13a545018335ec5e72ca271fe90ce95944c669919f
SimHash 231d955942f5

Groups

*

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 600

googlebot
googlebot-image
mediapartners-google
msnbot
msnbot-media
slurp
yahoo-blogs
yahoo-mmcrawler

Rule Path
Disallow /administrator/
Disallow /cache/
Disallow /components/
Disallow /images/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /libraries/
Disallow /logs/
Disallow /media/
Disallow /modules/
Disallow /plugins/
Disallow /templates/
Disallow /tmp/
Disallow /CHANGELOG.txt
Disallow /cron.php
Disallow /INSTALL.mysql.txt
Disallow /INSTALL.pgsql.txt
Disallow /install.php
Disallow /INSTALL.txt
Disallow /LICENSE.txt
Disallow /MAINTAINERS.txt
Disallow /update.php
Disallow /UPGRADE.txt
Disallow /xmlrpc.php

Other Records

Field Value
crawl-delay 600
Back to top

Comments

  • If the Joomla site is installed within a folder such as at
  • e.g. www.example.com/joomla/ the robots.txt file MUST be
  • moved to the site root at e.g. www.example.com/robots.txt
  • AND the joomla folder name MUST be prefixed to the disallowed
  • path, e.g. the Disallow rule for the /administrator/ folder
  • MUST be changed to read Disallow: /joomla/administrator/
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/orig.html
  • For syntax checking, see:
  • http://www.sxw.org.uk/computing/robots/check.html
  • disallow all
  • but allow only important bots
  • Directories
  • Files
Back to top