Log In Sign Up

collectiveray.com
robots.txt

Robots Exclusion Standard data for collectiveray.com

Resource Scan

Scan Details

Site Domain collectiveray.com
Base Domain collectiveray.com
Scan Status Ok
Last Scan2024-11-12T11:42:01+00:00
Next Scan 2024-11-19T11:42:01+00:00

Last Scan

Scanned2024-11-12T11:42:01+00:00
URL https://collectiveray.com/robots.txt
Redirect https://www.collectiveray.com/robots.txt
Redirect Domain www.collectiveray.com
Redirect Base collectiveray.com
Domain IPs 104.21.30.54, 172.67.150.155, 2606:4700:3034::6815:1e36, 2606:4700:3035::ac43:969b
Redirect IPs 104.21.30.54, 172.67.150.155, 2606:4700:3034::6815:1e36, 2606:4700:3035::ac43:969b
Response IP 172.67.150.155
Found Yes
Hash acc0ebad66572dd7f416c8cbcb67ad3228618d762be84c6b402404cc52afc343
SimHash 331ebf1dddf4

Groups

googlebot

Rule Path
Allow /*.js*
Allow /*.css*
Allow /*.png*
Allow /*.jpg*
Allow /*.gif*
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /logs/
Disallow /out/
Disallow /banners/
Disallow /BI/
Disallow /article-tree/
Disallow /business-tree/
Disallow /*/component/
Disallow /fi/komponentti/*
Disallow /pt/componente/*
Disallow /no/komponent/*
Disallow /nl/bestanddeel/*
Disallow /de/Komponente/*
Disallow /sv/komponent/*
Disallow /it/componente/*
Disallow /es/componente/*
Disallow /da/komponent/*
Disallow /is/hluti/*
Disallow /fr/composant/*
Disallow */com_docmanpaypal/*
Disallow *com_joomlatools*
Disallow /joomlaspeedtest/
Disallow /example/
Disallow /exampleAMP/
Disallow /tmp/
Disallow */tag/*
Disallow /tags.html*
Disallow /tags*
Disallow /log-in.html
Disallow */log-in/*
Disallow */contact-us.html*
Disallow /forums/
Disallow /forum/
Disallow /osdownloads.html
Disallow */download.php*
Disallow /index.php?option=com_users*
Disallow *jos_change_template*
Disallow /*?task=view
Disallow /*?format=html
Disallow /index.php?option=com_banners*
Disallow /index.php?option=com_acymailing*
Disallow /index.php?subid=&option=com_acymailing*
Disallow /acymailing/
Disallow */file.html
Disallow /index.php?option=com_ninjarsssyndicator
Disallow */page-*.html
Disallow */page-*
Disallow */search*
Disallow *com_search*
Disallow /*/joomla-25-and-joomla-3-plugins*
Disallow /*/joomla-25-and-joomla-3-modules*
Disallow /*/html-templates*
Disallow /*/psd-templates*
Disallow /*/extension-demos/*
Disallow /*/downloads*
Disallow /*/joomla/joomla-extension-demos/*
Disallow /*/j/extension-demos*
Disallow */porpoiseant/*
Disallow */tardisrocinante/*
Disallow */detroitchicago/*
Disallow */humix/*
Disallow *contact-us.html*
Disallow *_escaped_fragment_*
Disallow *ccomment-comment*
Disallow *cpnb_method*
Disallow *expand_article*

*

Rule Path
Allow /*.js*
Allow /*.css*
Allow /*.png*
Allow /*.jpg*
Allow /*.gif*
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /logs/
Disallow /out/
Disallow /banners/
Disallow /BI/
Disallow /article-tree/
Disallow /business-tree/
Disallow /*/component/
Disallow /fi/komponentti/*
Disallow /pt/componente/*
Disallow /no/komponent/*
Disallow /nl/bestanddeel/*
Disallow /de/Komponente/*
Disallow /sv/komponent/*
Disallow /it/componente/*
Disallow /es/componente/*
Disallow /da/komponent/*
Disallow /fr/composant/*
Disallow /is/hluti/*
Disallow */com_docmanpaypal/*
Disallow *com_joomlatools*
Disallow /joomlaspeedtest/
Disallow /example/
Disallow /exampleAMP/
Disallow /tmp/
Disallow */tag/*
Disallow /tags.html*
Disallow /tags*
Disallow /log-in.html
Disallow */log-in/*
Disallow */contact-us.html*
Disallow /forums/
Disallow /forum/
Disallow /osdownloads.html
Disallow */download.php*
Disallow /index.php?option=com_users*
Disallow *jos_change_template*
Disallow /*?task=view
Disallow /*?format=html
Disallow /index.php?option=com_banners*
Disallow /index.php?option=com_acymailing*
Disallow /index.php?subid=&option=com_acymailing*
Disallow /acymailing/
Disallow */file.html
Disallow /index.php?option=com_ninjarsssyndicator
Disallow */page-*.html
Disallow */page-*
Disallow */search*
Disallow *com_search*
Disallow /*/joomla-25-and-joomla-3-plugins*
Disallow /*/joomla-25-and-joomla-3-modules*
Disallow /*/html-templates*
Disallow /*/psd-templates*
Disallow /*/extension-demos/*
Disallow /*/downloads*
Disallow /*/joomla/joomla-extension-demos/*
Disallow /*/j/extension-demos*
Disallow *contact-us.html*
Disallow *_escaped_fragment_*
Disallow *ccomment-comment*
Disallow *cpnb_method*
Disallow *expand_article*
Disallow */porpoiseant/*
Disallow */tardisrocinante/*
Disallow */detroitchicago/*
Disallow */humix/*
Back to top

Other Records

Field Value
sitemap https://www.collectiveray.com/sitemap.xml
Back to top

Comments

  • If the Joomla site is installed within a folder such as at
  • e.g. www.example.com/joomla/ the robots.txt file MUST be
  • moved to the site root at e.g. www.example.com/robots.txt
  • AND the joomla folder name MUST be prefixed to the disallowed
  • path, e.g. the Disallow rule for the /administrator/ folder
  • MUST be changed to read Disallow: /joomla/administrator/
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/orig.html
  • For syntax checking, see:
  • http://tool.motoricerca.info/robots-checker.phtml
  • Disabled all tags - throw 404 so that they get deindexed
  • new downloads link
  • Disallow: *view=*
  • extra stuff being blocked due to the below
  • Disallow: */file
  • Disallow: /file/
  • Disabled feed links - we therefore don't need to block them
  • Disallow: */feed/*
  • The below was breaking some images from being crawled
  • Remove thin download / demo pages
  • Disallow: /*/wordpress-plugins/*
  • Ezoic JS pages getting index
  • Bunch of contact us pages and other pages got crawled - remove
  • Googlebot crawling Cookies Policy Notification Pages
  • Added as parameter
  • Ezoic expand article link
  • Disabled all tags - throw 404 so that they get deindexed
  • new downloads link
  • Disallow: *view=*
  • extra stuff being blocked due to the below
  • Disallow: */file
  • Disallow: /file/
  • Disabled feed links - we therefore don't need to block them
  • Disallow: */feed/*
  • The below was breaking some images from being crawled
  • Remove thin download / demo pages
  • Disallow: /*/wordpress-plugins/*
  • Bunch of contact us pages got crawled - remove
  • Googlebot crawling Cookies Policy Notification Pages
  • Added as parameter
  • Ezoic expand article link
  • Ezoic JS pages getting index
Back to top

Warnings

  • 2 invalid lines.
Back to top