Log In Sign Up

british-caving.org.uk
robots.txt

Robots Exclusion Standard data for british-caving.org.uk

Resource Scan

Scan Details

Site Domain british-caving.org.uk
Base Domain british-caving.org.uk
Scan Status Ok
Last Scan2024-09-21T22:49:59+00:00
Next Scan 2024-10-21T22:49:59+00:00

Last Scan

Scanned2024-09-21T22:49:59+00:00
URL https://british-caving.org.uk/robots.txt
Domain IPs 35.178.58.240
Response IP 35.178.58.240
Found Yes
Hash 8fef572f098cbf6fe8601e5c9ef1be767ef72f56593453b50af9c66de8ed8e8c
SimHash f25d1d1acbfc

Groups

semrushbot

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

adsbot/3.1

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

linespider/1.1

Rule Path
Disallow /

yeti/1.1

Rule Path
Disallow /

adsbot/3.1

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Back to top

Comments

  • If the Joomla site is installed within a folder
  • eg www.example.com/joomla/ then the robots.txt file
  • MUST be moved to the site root
  • eg www.example.com/robots.txt
  • AND the joomla folder name MUST be prefixed to all of the
  • paths.
  • eg the Disallow rule for the /administrator/ folder MUST
  • be changed to read
  • Disallow: /joomla/administrator/
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/orig.html
  • For syntax checking, see:
  • http://tool.motoricerca.info/robots-checker.phtml
Back to top