Log In Sign Up

snakes.co.il
robots.txt

Robots Exclusion Standard data for snakes.co.il

Resource Scan

Scan Details

Site Domain snakes.co.il
Base Domain snakes.co.il
Scan Status Ok
Last Scan2024-11-04T10:58:54+00:00
Next Scan 2024-11-11T10:58:54+00:00

Last Scan

Scanned2024-11-04T10:58:54+00:00
URL https://snakes.co.il/robots.txt
Redirect https://www.snakes.co.il/robots.txt
Redirect Domain www.snakes.co.il
Redirect Base snakes.co.il
Domain IPs 185.18.205.241
Redirect IPs 185.18.205.241
Response IP 185.18.205.241
Found Yes
Hash f2c290b1d4b2b1e048fe8148e576b579ac72e507c30b6418934ce014e060b9bd
SimHash f00cbf11c6f5

Groups

*

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 900

googlebot
googlebot-image
mediapartners-google
msnbot
msnbot-media
slurp
yahoo-blogs
yahoo-mmcrawler
twitterbot
facebookexternalhit

Rule Path
Disallow /administrator/
Disallow /bin/
Disallow /cache/
Disallow /cli/
Disallow /includes/
Disallow /installation/
Disallow /language/
Disallow /layouts/
Disallow /libraries/
Disallow /logs/
Disallow /tmp/
Disallow /font/
Disallow /README.txt
Disallow /LICENSE.txt
Disallow /privacy-policy/
Disallow /search/
Disallow /register/
Disallow /reset/
Disallow /username/
Disallow /login/

Other Records

Field Value
crawl-delay 900

twitterbot

Rule Path
Allow /

facebookexternalhit

Rule Path
Allow /

googlebot

Rule Path
Allow /*.js*
Allow /*.css*
Allow /*.png*
Allow /*.jpg*
Allow /*.jpeg*
Allow /*.gif*
Allow /*.eot*
Allow /*.woff*
Allow /*.ttf*
Allow /*.svg*
Allow /*.otf*
Allow /*.pdf*
Allow /*.PNG*
Allow /*.JPG*
Allow /*.JPEG*
Allow /*.mp3*
Allow /*.pdf*
Back to top

Comments

  • If the Joomla site is installed within a folder such as at
  • e.g. www.example.com/joomla/ the robots.txt file MUST be
  • moved to the site root at e.g. www.example.com/robots.txt
  • AND the joomla folder name MUST be prefixed to the disallowed
  • path, e.g. the Disallow rule for the /administrator/ folder
  • MUST be changed to read Disallow: /joomla/administrator/
  • For more information about the robots.txt standard, see:
  • http://www.robotstxt.org/orig.html
  • For syntax checking, see:
  • http://tool.motoricerca.info/robots-checker.phtml
  • disallow all
  • but allow only important bots
  • Directories
  • Files
  • Paths (clean URLs)
Back to top