Log In Sign Up

smbnation.com
robots.txt

Robots Exclusion Standard data for smbnation.com

Resource Scan

Scan Details

Site Domain smbnation.com
Base Domain smbnation.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-05-30T20:00:43+00:00
Next Scan 2024-08-28T20:00:43+00:00

Last Successful Scan

Scanned2021-09-28T11:19:50+00:00
URL https://smbnation.com/robots.txt
Redirect http://www.smbnation.com/robots.txt
Redirect Domain www.smbnation.com
Redirect Base smbnation.com
Found Yes
Hash 30ef5dc94632111ed7b23da7cdbb726cfe9b61c243d4d4bb89d2734bebf407df
SimHash c21e9450cbf5

Groups

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

grammarly

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

blexbot/1.0

Rule Path
Disallow /

dotbot/1.1

Rule Path
Disallow /

*

Rule Path
Allow /*.js*
Allow /*.css*
Allow /*.png*
Allow /*.jpg*
Allow /*.gif*
Allow /index.php?option=com_jmap&view=sitemap&format=xml
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /logs/
Disallow /tags/
Disallow /event/
Disallow /tmp/
Disallow /*/events-calendar/*
Disallow /events-calendar/*
Disallow /events-calendar/

Other Records

Field Value
crawl-delay 30
Back to top

Other Records

Field Value
sitemap http://www.smbnation.com/index.php?option=com_jmap&view=sitemap&format=xml
Back to top

Comments

  • If the Joomla site is installed within a folder such as at
  • e.g. www.example.com/joomla/ the robots.txt file MUST be
  • moved to the site root at e.g. www.example.com/robots.txt
  • AND the joomla folder name MUST be prefixed to the disallowed
  • path, e.g. the Disallow rule for the /administrator/ folder
  • MUST be changed to read Disallow: /joomla/administrator/
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/orig.html
  • For syntax checking, see:
  • http://tool.motoricerca.info/robots-checker.phtml
  • Block MegaIndex.ru
  • JSitemap entries
Back to top