Log In Sign Up

roboteq.com
robots.txt

Robots Exclusion Standard data for roboteq.com

Resource Scan

Scan Details

Site Domain roboteq.com
Base Domain roboteq.com
Scan Status Ok
Last Scan2024-06-28T23:21:21+00:00
Next Scan 2024-07-28T23:21:21+00:00

Last Scan

Scanned2024-06-28T23:21:21+00:00
URL https://roboteq.com/robots.txt
Domain IPs 192.124.249.2
Response IP 192.124.249.2
Found Yes
Hash 4e063b05aa2ba27e885f641a47d9da2ce09cbbbb50d9dfe340cf55b32a8e0c53
SimHash eb1f255343f5

Groups

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

*

Rule Path
Allow /*.js*
Allow /*.css*
Allow /*.png*
Allow /*.jpg*
Allow /*.gif*
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /logs/
Disallow /docman-list/
Disallow /forum/user/
Disallow /component/users/
Disallow /tmp/
Disallow /login?
Disallow /custom-filters/*
Allow /*.js*
Allow /*.css*
Allow /*.png*
Allow /*.jpg*
Allow /*.gif*

gigabot

Rule Path
Disallow /

liebaofast

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

mauibot (crawler.feedback+wc@gmail.com)

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

mqqbrowser

Rule Path
Disallow /

nimbostratus-bot/v1.3.2

Rule Path
Disallow /

qwant-news

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

seekport crawler

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

sputnikbot/2.3

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

tinytestbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ucbrowser

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

yisouspider

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.roboteq.com/index.php?option=com_jmap&view=sitemap&format=xml
sitemap https://www.roboteq.com/index.php?option=com_jmap&view=sitemap&format=images
sitemap https://www.roboteq.com/index.php?option=com_jmap&view=sitemap&format=mobile
Back to top

Comments

  • If the Joomla site is installed within a folder
  • eg www.example.com/joomla/ then the robots.txt file
  • MUST be moved to the site root
  • eg www.example.com/robots.txt
  • AND the joomla folder name MUST be prefixed to all of the
  • paths.
  • eg the Disallow rule for the /administrator/ folder MUST
  • be changed to read
  • Disallow: /joomla/administrator/
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/orig.html
  • For syntax checking, see:
  • http://tool.motoricerca.info/robots-checker.phtml
  • Disallow: /*?
  • Disallow: /*=
  • JSitemap entries
Back to top