[Numpy-discussion] Default builds of OpenBLAS development branch are now fork safe
Carl Kleffner
cmkleffner at gmail.com
Thu Feb 20 09:43:06 EST 2014
Hi,
some days ago I put some preliminary mingw-w64 binaries and code based on
python2.7 on my google drive to discuss it with Matthew Brett. Maybe its
time for a broader discussion. IMHO it is ready for testing but not for
consumption.
url:
https://drive.google.com/folderview?id=0B4DmELLTwYmldUVpSjdpZlpNM1k&usp=sharing
contains:
(1) patches used
numpy.patch
scipy.patch
(2) 64 bit GCC toolchain
amd64/
mingw-w64-toolchain-static_amd64-gcc-4.8.2_vc90_rev-20140131.7z
libpython27.a
(3) numpy-1.8.0 linked against OpenBLAS
amd64/numpy-1.8.0/
numpy-1.8.0.win-amd64-py2.7.exe
numpy-1.8.0-cp27-none-win_amd64.whl
numpy_amd64_fcompiler.log
numpy_amd64_build.log
numpy_amd64_test.log
_numpyconfig.h
config.h
(4) scipy-0.13.3 linked against OpenBLAS
amd64/scipy-0.13.3/
scipy-0.13.3.win-amd64-py2.7.exe
scipy-0.13.3-cp27-none-win_amd64.whl
scipy_amd64_fcompiler.log
scipy_amd64_build.log
scipy_amd64_build_cont.log
scipy_amd64_test._segfault.log
scipy_amd64_test.log
(5) 32 bit GCC toolchain
win32/
mingw-w64-toolchain-static_win32-gcc-4.8.2_vc90_rev-20140131.7z
libpython27.a
(6) numpy-1.8.0 linked against OpenBLAS
win32/numpy-1.8.0/
numpy-1.8.0.win32-py2.7.exe
numpy-1.8.0-cp27-none-win32.whl
numpy_win32_fcompiler.log
numpy_win32_build.log
numpy_win32_test.log
_numpyconfig.h
config.h
(7) scipy-0.13.3 linked against OpenBLAS
win32/scipy-0.13.3/
scipy-0.13.3.win32-py2.7.exe
scipy-0.13.3-cp27-none-win32.whl
scipy_win32_fcompiler.log
scipy_win32_build.log
scipy_win32_build_cont.log
scipy_win32_test.log
Summary to compile numpy:
(1) <mingw>\bin and python should be in the PATH. Choose 32 bit or 64 bit
architecture.
(2) copy libpython27.a to <python>\libs
check, that <python>\libs does not contain libmsvcr90.a
(3) apply numpy.patch
(4) copy libopenblas.dll from <mingw>\bin to numpy\core
of course don't ever mix 32bit and 64 bit code
(5) create a site.cfg in the numpy folder with the absolute path to the
mingw import
files/header files. I copied the openblas header files, importlibs into
the GCC toolchain.
(6) create a mingw distutils.cfg file
(7) test the configuration
python setup.py config_fc --verbose
and
python setup.py build --help-fcompiler
(8) build
python setup.py build --fcompiler=gnu95
(9) make a distro
python setup.py bdist --format=wininst
(10) make a wheel
wininst2wheel numpy-1.8.0.win32-py2.7.exe (for 32 bit)
(11) install
wheel install numpy-1.8.0-cp27-none-win32.whl
(12) import numpy; numpy.test()
Summary to compile scipy:
(1) apply scipy.patch
(2) python setup.py build --fcompiler=gnu95
and a second time
python setup.py build --fcompiler=gnu95
(3) python setup.py bdist --format=wininst
(4) install
(5) import scipy; scipy.test()
Hints:
(1) libpython import file:
The libpython27.a import files has been generated with gendef and dlltool
according to the recommendations on the mingw-w64 faq site. It is
essential to not use import libraries from anywhere, but create it with the
tools in the GCC toolchain. The GCC toolchains contains correct generated
mscvrXX import files per default.
(2) OpenBLAS:
the openblas DLL must be copied to numpy/core before building numpy. All
Blas and Lapack code will be linked dynamically to this DLL. Because of
this the overall distro size gets much smaller compared to numpy-MKL or
scipy-MKL. It is not necessary to add numpy/core to the path! (at least on
my machine). To load libopenblas.dll to the process space it is only
necessary to import numpy - nothing else. libopenblas.dll is linked against
the msvcr90.dll, just like python. The DLL itself is a fat binary
containing all optimized kernels for all supported platforms. DLL, headers
and import files have been included into the toolchain.
(3) mingw-w64 toolchain:
In short it is an extended version of the 'recommended' mingw-builds
toolchain with some minor patches and customizations. I used
https://github.com/niXman/mingw-builds for my build. It is a 'statically'
build, thus all gcc related runtimes are linked statically into the
resulting binaries.
(4) Results:
Some FAILS - see corresp. log-files. I got a segfault with scipy.test() (64
bit) with multithreaded OpenBLAS (test_ssygv_1) but not in single threaded
mode. Due to time constraints I didn't made further tests right now.
Regards
Carl
2014-02-20 14:28 GMT+01:00 Sturla Molden <sturla.molden at gmail.com>:
> Will this mean NumPy, SciPy et al. can start using OpenBLAS in the
> "official" binary packages, e.g. on Windows and Mac OS X? ATLAS is slow and
> Accelerate conflicts with fork as well.
>
> Will dotblas be built against OpenBLAS? AFAIK, it is only buit against
> ATLAS or MKL, not any other BLAS, but it should just be a matter of
> changing the build/link process.
>
> Sturla
>
>
>
> Nathaniel Smith <njs at pobox.com> wrote:
> > Hey all,
> >
> > Just a heads up: thanks to the tireless work of Olivier Grisel, the
> > OpenBLAS development branch is now fork-safe when built with its default
> > threading support. (It is still not thread-safe when built using OMP for
> > threading and gcc, but this is not the default.)
> >
> > Gory details: <a
> > href="https://github.com/xianyi/OpenBLAS/issues/294">
> https://github.com/xianyi/OpenBLAS/issues/294</a>
> >
> > Check it out - if it works you might want to consider lobbying your
> > favorite distro to backport it.
> >
> > -n
> >
> > _______________________________________________ NumPy-Discussion mailing
> list
> > NumPy-Discussion at scipy.org <a
> > href="http://mail.scipy.org/mailman/listinfo/numpy-discussion">
> http://mail.scipy.org/mailman/listinfo/numpy-discussion</a>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20140220/0050997c/attachment.html>
More information about the NumPy-Discussion
mailing list