Log In Sign Up

ukrgb.co.uk
robots.txt

Robots Exclusion Standard data for ukrgb.co.uk

Resource Scan

Scan Details

Site Domain ukrgb.co.uk
Base Domain ukrgb.co.uk
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-10-27T09:16:17+00:00
Next Scan 2024-12-26T09:16:17+00:00

Last Successful Scan

Scanned2024-08-28T20:43:21+00:00
URL https://ukrgb.co.uk/robots.txt
Redirect https://www.ukriversguidebook.co.uk/robots.txt
Redirect Domain www.ukriversguidebook.co.uk
Redirect Base ukriversguidebook.co.uk
Domain IPs 52.18.200.81
Redirect IPs 13.224.163.39, 13.224.163.54, 13.224.163.83, 13.224.163.84
Response IP 52.84.229.108
Found Yes
Hash b0b5507bb957164b438dd70e584a3e6af17da0d12464091731f2c3588738597f
SimHash a21e95588bed

Groups

*

Rule Path
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /components/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /logs/
Disallow /modules/
Disallow /plugins/
Disallow /tmp/

semrushbot

Rule Path
Disallow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

yandex

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

amazonbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 30

gptbot/1.0

Rule Path
Disallow /
Back to top

Comments

  • If the Joomla site is installed within a folder such as at
  • e.g. www.example.com/joomla/ the robots.txt file MUST be
  • moved to the site root at e.g. www.example.com/robots.txt
  • AND the joomla folder name MUST be prefixed to the disallowed
  • path, e.g. the Disallow rule for the /administrator/ folder
  • MUST be changed to read Disallow: /joomla/administrator/
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/orig.html
  • For syntax checking, see:
  • http://tool.motoricerca.info/robots-checker.phtml
Back to top