[SciPy-Dev] win32 binaries and scipy 0.17.0 release

[Carl Kleffner]

2016-01-04 20:31 GMT+01:00 Tony Kelman <tony at kelman.net>:

> > On Mon, Jan 4, 2016 at 4:03 PM, Carl Kleffner <cmkleffner <at> gmail.com
> >
> wrote:
> > > Python binary extensions build with mingw-w64 and MSVC based CPython
> > > combatibility in mind should be compiled with a toolchain that has to
> be
> > > build with specific configurations and patches.
> > >
> > > The most important demands are exclusive linking the appropriate MS CRT
> > > runtime and statically linking the gcc runtime objects. Some other
> > > configurations are handling long double as double (as MS ABI does not
> > > support extended precision), standrad exception and win32 thread
> handling.
> > > Universal runtime support (VS 2015) is planned for the future.
>
> I'd encourage you to place the defaulting to static linking somewhere
> other than the toolchain build configuration itself if you want your
> work to be adopted upstream. Even just wrapper scripts might do the
> trick. Or work on standardizing a location to put the shared runtime
> libraries. I highly doubt Linux distributions or msys2 are ever going
> to accept static-by-default mingw-w64 configurations upstream. Throwing
> C++ exceptions across library boundaries is going to be an issue.
>

I propose different configurations of the toolchains based upon its users:
user programmers: mingwpy wheel packages on PYPI and package maintainers:
cross compiler within a Linux docker VM.

mingwpy wheels with a statically configured toolchain gives a much better
programming experience to python users. Typical use cases are cython or
theano based extensions. As a programmer you don't ever except additional
dependency problems for these use cases. And throwing C++ exceptions across
library boundaries is not an issues at all for scipy as there are no
library boundaries (this is true for most of the python packages I'm aware
of)

If you want python binaries depend on gcc shared runtime libraries you get
another major problem on CPython for Windows. It is not garantueed, that
correct versions of the libraries are found automatically at runtime. And
python binaries (pyd's) are spread all around within the site-packages
folder and subfolders. There is only one workaround I can think of:
explicit preloading of all necessary runtime DLLs into the process space.
This could be handled within a dedicated mingw-runtime package. That
unfortunately means: an additional windows-only package dependency to all
mingw-w64 based python binaries.

BTW: There is nothing wrong to use an alternative mingw-w64 package with
posix threads and gcc shared libraries. wxpython and PyQT are the only two
packages that comes into my mind for these use case. I guess in this case
the msys2 mingw-w64 toolchains could do with some specs trickery, i.e.
additional specs files that could be activated per command line line. (or
use a linux cross compiler)

> > I doubt that this demands can be easily handled with mingw-w64 cross
> > > compilers found a certain Linux distros. Is the idea to build a
> specialized
> > > cross compiler for that purpose?
>
> If needed. Eventually your work would ideally be upstreamed and picked
> up by the distributions' mingw-w64 packages. In the short term, let's
> start by educating and familiarizing the Linux-based developer community
> with how the basics of cross-compiling work, then work on Python-specific
> issues. This can happen entirely in parallel with what you're doing on
> the toolchain side. If you'd rather make all of your builds from Windows
> for ease of testing, go ahead and do so, though I'll say that if you're
> building GCC itself frequently, you will save *a lot* of time by
> leveraging cross compilation. The openSUSE build service makes it
> incredibly easy to build cross-compilers, Windows exe compiler builds
> (cross-compiling GCC itself with --build=linux, --host=mingw-w64,
> --target=mingw-w64), and .a and .dll libraries all from a web interface
> (or a CLI, but you may have to be running opensuse to use it? not
> positive). It's a bit like github for binaries - I was able to make a
> custom mingw-w64 build including a patch for a gcc bug via the build
> service web interface over the weekend. I have a hacky script in the
> Julia repository that can be used to download these builds including
> their dependencies on non-RPM-based systems, it would be worth porting
> that script to Python at some point.
>
> > > Using the mingwpy package together with msys2 can be used instead if
> > > autotools have to be used. As mingwpy is not a cross compiler a
> windows box
> > > or a windows VM is needed.
>
> This is a common limitation with Python build tools that I think is
> worth a bit of extra work to avoid inheriting.
>
>
> Matthew Brett <matthew.brett <at> gmail.com> writes:
>
> > Yes, my idea was, that it would be easier for many of us to work on
> > cross-compilers in Linux VMs or docker containers than to work on
> > Windows VMs.  We would have to customize the cross compiler.  I'm not
> > suggesting this should be the first attack on the problem, only that
> > it could be useful for the medium term.
>
> > That docker container would be extremely useful - what can I do to help?
>
> Name a preferred Linux distribution and a particular library to use as
> an initial starting example. As one example, I put together a cross
> compiled build of reference netlib Lapack here
> https://build.opensuse.org/package/show/windows:mingw:win64/mingw64-lapack
> that uses Lapack's cmake build system. We could translate that from an
> opensuse-specific spec file to a more widely usable docker container.
> If we want to get ambitious, we could do openblas instead, if you have
> a particular build configuration of openblas (DYNAMIC_ARCH obviously,
> and maybe including openblas' optimized port of lapack too?) you like.
>
>

>

>
> _______________________________________________
> SciPy-Dev mailing list
> SciPy-Dev at scipy.org
> https://mail.scipy.org/mailman/listinfo/scipy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scipy-dev/attachments/20160104/d7d6f779/attachment.html>