[Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe

Olivier Grisel olivier.grisel at ensta.org
Thu Feb 20 09:40:12 EST 2014


2014-02-20 14:28 GMT+01:00 Sturla Molden <sturla.molden at gmail.com>:
> Will this mean NumPy, SciPy et al. can start using OpenBLAS in the
> "official" binary packages, e.g. on Windows and Mac OS X? ATLAS is slow and
> Accelerate conflicts with fork as well.

This what I would like to do personnally. Ideally as a distribution of
wheel packages

To do so I built the current develop branch of OpenBLAS with:

  make USE_OPENMP=0 NUM_THREAD=32 NO_AFFINITY=1
  make PREFIX=/opt/OpenBLAS-noomp install

Then I added a site.cfg file in the numpy source folder with the lines:

  [openblas]
  libraries = openblas
  library_dirs = /opt/OpenBLAS-noomp/lib
  include_dirs = /opt/OpenBLAS-noomp/include


> Will dotblas be built against OpenBLAS?

Yes:

  $ ldd numpy/core/_dotblas.so
      linux-vdso.so.1 =>  (0x00007fff24d04000)
      libopenblas.so.0 => /opt/OpenBLAS-noomp/lib/libopenblas.so.0
(0x00007f432882f000)
      libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f4328449000)
      libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f432814c000)
      libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f4327f2f000)
      libgfortran.so.3 => /usr/lib/x86_64-linux-gnu/libgfortran.so.3
(0x00007f4327c18000)
      /lib64/ld-linux-x86-64.so.2 (0x00007f43298d3000)
      libquadmath.so.0 => /usr/lib/x86_64-linux-gnu/libquadmath.so.0
(0x00007f43279e1000)


However when testing this I noticed the following strange slow import
and memory usage in an IPython session:

>>> import os, psutil
>>> psutil.Process(os.getpid()).get_memory_info().rss / 1e6
20.324352
>>> %time import numpy
CPU times: user 1.95 s, sys: 1.3 s, total: 3.25 s
Wall time: 530 ms
>>> psutil.Process(os.getpid()).get_memory_info().rss / 1e6
349.507584

The libopenblas.so file is just 14MB so I don't understand how I could
get those 330MB from.

It's even worst when using static linking (libopenblas.a instead of
libopenblas.so under linux).

With Atlas or MKL I get import times under 50 ms and the memory
overhead of the numpy import is just ~15MB.

I would be very interested in any help on this:

- can you reproduce this behavior?
- do you have an idea of a possible cause?
- how to investigate?

-- 
Olivier
http://twitter.com/ogrisel - http://github.com/ogrisel



More information about the NumPy-Discussion mailing list