ruby-doc.org
robots.txt

Robots Exclusion Standard data for ruby-doc.org

Resource Scan

Scan Details

Site Domain ruby-doc.org
Base Domain ruby-doc.org
Scan Status Ok
Last Scan2024-11-13T14:17:13+00:00
Next Scan 2024-11-20T14:17:13+00:00

Last Scan

Scanned2024-11-13T14:17:13+00:00
URL https://ruby-doc.org/robots.txt
Domain IPs 52.118.46.4
Response IP 52.118.46.4
Found Yes
Hash 7431a0e7b6f059d4c73d40b39d67fa12cdde28bdb1d1b12cac60aa6b1e8dc701
SimHash 846fe0ca4974

Groups

*

Rule Path
Disallow /emailaddresses/
Disallow /core-1.8.7/
Disallow /stdlib-1.8.7/
Disallow /core-1.9.2/
Disallow /stdlib-1.9.2/
Disallow /core-1.9.3/
Disallow /stdlib-1.9.3/
Disallow /core-2.0.0/
Disallow /stdlib-2.0.0/
Disallow /core-2.1.0/
Disallow /stdlib-2.1.0/
Disallow /core-2.1.1/
Disallow /stdlib-2.1.1/
Disallow /core-2.1.2/
Disallow /stdlib-2.1.2/
Disallow /core-2.1.3/
Disallow /stdlib-2.1.3/
Disallow /core-2.1.4/
Disallow /stdlib-2.1.4/
Disallow /core-2.1.5/
Disallow /stdlib-2.1.5/
Disallow /core-2.1.6/
Disallow /stdlib-2.1.6/
Disallow /core-2.1.7/
Disallow /stdlib-2.1.7/
Disallow /core-2.1.8/
Disallow /stdlib-2.1.8/
Disallow /core-2.1.9/
Disallow /stdlib-2.1.9/
Disallow /core-2.2.0/
Disallow /stdlib-2.2.0/
Disallow /core-2.2.1/
Disallow /stdlib-2.2.1/
Disallow /core-2.2.2/
Disallow /stdlib-2.2.2/
Disallow /core-2.2.3/
Disallow /stdlib-2.2.3/
Disallow /core-2.2.4/
Disallow /stdlib-2.2.4/
Disallow /core-2.2.5/
Disallow /stdlib-2.2.5/
Disallow /core-2.2.6/
Disallow /stdlib-2.2.6/
Disallow /core-2.2.7/
Disallow /stdlib-2.2.7/
Disallow /core-2.2.8/
Disallow /stdlib-2.2.8/
Disallow /core-2.2.9/
Disallow /stdlib-2.2.9/
Disallow /core-2.3.0/
Disallow /stdlib-2.3.0/
Disallow /core-2.3.1/
Disallow /stdlib-2.3.1/
Disallow /core-2.3.2/
Disallow /stdlib-2.3.2/
Disallow /core-2.3.3/
Disallow /stdlib-2.3.3/
Disallow /core-2.3.4/
Disallow /stdlib-2.3.4/
Disallow /core-2.3.5/
Disallow /stdlib-2.3.5/
Disallow /core-2.3.6/
Disallow /stdlib-2.3.6/
Disallow /core-2.3.7/
Disallow /stdlib-2.3.7/
Disallow /core-2.3.8/
Disallow /stdlib-2.3.8/

nutch

Rule Path
Disallow /

googlebot

Rule Path
Disallow *.pdf
Disallow *.tgz
Disallow *.zip
Disallow *.tar
Disallow *.bz2

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

webzip

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

web downloader

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

offline explorer pro

Rule Path
Disallow /

httrack website copier

Rule Path
Disallow /

offline commander

Rule Path
Disallow /

leech

Rule Path
Disallow /

websnake

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

http weazel

Rule Path
Disallow /