[Numpy-discussion] Ruby benchmark -- numpy is slower.... was: Re: Ruby's NMatrix and NVector

Fri May 16 13:56:41 EDT 2008

la, 2008-05-17 kello 00:39 +0900, David Cournapeau kirjoitti:
> On Sat, May 17, 2008 at 12:00 AM, Anne Archibald
> <peridot.faceted at gmail.com> wrote:
> 
> >
> > There are four benchmarks: add, multiply, dot, and solve. dot and
> > solve use BLAS, and for them numpy ruby and octave are comparable. Add
> > and multiply are much slower in numpy, but they are implemented in
> > numpy itself.
> 
> The benchmark was done in 2005, and we do not know how it was done (no
> source). I don't know anything about ruby (that's my first ruby
> "program") but:
[clip]

The benchmark sources are in Narray's source directory.

I took a look and my conclusion is that the benchmark is simply flawed:
for Ruby, only user time is counted, while for Python, both user and
system times are counted. The code uses Python's time.clock() which
according to the documentation returns the CPU time (apparently user +
system). On the Ruby side it uses Process.times.utime which is the
elapsed user time.

Running the original tests as they are in NArray 0.5.9 yields (I took
representative ones from several runs. Eyeballing, the std between runs
appeared of the order of 0.1...0.2s):

        ### Numeric 24.2 (24.2-8ubuntu2)
        ### Narray 0.5.9 (0.5.9-2)
        ### numpy 1.0.4 (1:1.0.4-6ubuntu3)
        ###
        ### All of these from Ubuntu 8.04 packages.

        $ time ruby mul.rb 
        a = NArray.float(1000000): 
        [ 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0,
        12.0, ... ]
        b = NArray.float(1000000): 
        [ 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0,
        12.0, ... ]
        calculating c = a*b ...
         Time: 3.05 sec

        real	0m5.039s
        user	0m3.116s
        sys	0m1.564s

Obviously, the reported time here is the user time only!

        $ time python mul.py   # the old Numeric
        a.typecode: d ,  a.shape: (1000000,)
        b.typecode: d ,  b.shape: (1000000,)
        calculating c = a*b ...
          Time:   6.020 sec

        real	0m6.999s
        user	0m4.308s
        sys	0m2.164s

Whereas here it must be the sum of the user and system times!

Running tests for numpy and fixed time counting for Ruby:

        $ time python mul_numpy.py   # the new numpy
        a.typecode: float64 ,  a.shape: (1000000,)
        b.typecode: float64 ,  b.shape: (1000000,)
        calculating c = a*b ...
          Time:   4.580 sec

        real	0m5.774s
        user	0m3.352s
        sys	0m1.996s

        $ time ruby mul_correct.rb  # using T.times.utime +
        T.times.stime
        a = NArray.float(1000000): 
        [ 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0,
        12.0, ... ]
        b = NArray.float(1000000): 
        [ 0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, 11.0,
        12.0, ... ]
        calculating c = a*b ...
         Time: 4.57 sec

        real	0m5.045s
        user	0m3.060s
        sys	0m1.620s

I think this shows that there is no discernible difference between the
performance of numpy and Ruby's NArray. Even though the performance of
numpy and NArray is indeed better than that of Numeric, the difference
is not as large as the original benchmark led to believe.

Benchmark files attached, in case someone wants to contest my analysis.

-- 
Pauli Virtanen

-------------- next part --------------
A non-text attachment was scrubbed...
Name: mul.py
Type: text/x-python
Size: 270 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080516/d32aeec5/attachment.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mul.rb
Type: application/x-ruby
Size: 134 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080516/d32aeec5/attachment.rb>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mul_correct.rb
Type: application/x-ruby
Size: 142 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080516/d32aeec5/attachment-0001.rb>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mul_numpy.py
Type: text/x-python
Size: 266 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080516/d32aeec5/attachment-0001.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mybench.py
Type: text/x-python
Size: 313 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080516/d32aeec5/attachment-0002.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mybench.rb
Type: application/x-ruby
Size: 476 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080516/d32aeec5/attachment-0002.rb>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mybench_correct.rb
Type: application/x-ruby
Size: 508 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080516/d32aeec5/attachment-0003.rb>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mybench_numpy.py
Type: text/x-python
Size: 307 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080516/d32aeec5/attachment-0003.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: Digitaalisesti allekirjoitettu viestin osa
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20080516/d32aeec5/attachment.sig>