[Wheel-builders] [RFC] Proposal for packaging native libraries into Python wheels

[Nathaniel Smith]

On Apr 11, 2016 05:04, "Donald Stufft" <donald at stufft.io> wrote:
>
>
> > On Apr 11, 2016, at 4:50 AM, Nathaniel Smith <njs at pobox.com> wrote:
> >
> > **What about executables, like the ``openssl`` command-line tool?** We
> > can handle this by stashing the executable itself in another hidden
> > directory::
> >
> >    pynativelib_openssl/
> >        <...all the stuff above...>
> >        _bins/
> >            openssl
> >
> > and then install a Python dispatch script (using the same mechanisms
> > as we'd use to ship any Python script with wheel), where the script
> > looks like::
> >
> >    #!python
> >
> >    import os
> >    import sys
> >
> >    import pynativelib_openssl
> >
> >    pynativelib_openssl.enable()
> >    os.execv(pynativelib_openssl._openssl_bin_path, sys.argv)
>
>
> You don’t need to do this indirection, distutils has a scripts= option
> which allows you to pass arbitrary files that will get installed into
> the bin dir with executable bit set.

The indirection isn't needed to make it executable, it's needed so that we
have a chance to munge LD_LIBRARY_PATH before loading the real binary.

> >
> >
> > - **How can we make this play better with other distribution
> >  mechanisms?** E.g. Debian or conda package builders who when
> >  building ``cryptography`` or ``numpy`` actually *want* them to link
> >  to an external non-pynativelib version of libssl and libblas.
> >
> >  This is the one place where I'm most uncertain about the design
> >  described above.
> >
> >  The minimal answer is "well, Debian/conda/... can carry a little
> >  patch to their copy of ``cryptography``/``numpy``, and that's not
> >  wrong, but it would be really nice if we could do better than
> >  that. I'm not sure how. The biggest issue is that in the design
> >  above, downstream packages like ``numpy`` have a hardcoded ``import
> >  pynativelib_...`` in their ``__init__.py``, which should be removed
> >  when linked against a non-pynativelib version of the library. What
> >  would be very nice is if we could make the public pynativelib
> >  interface be entirely a build-time thing -- so that instead of a
> >  hard-coded runtime import, the build-time interface would provide
> >  the code that needed to be called at run-time. (Perhaps it could
> >  drop it into a special ``.py`` file as part of the build process,
> >  and then the package ``__init__.py`` would have to import that.)
> >  This would also potentially be useful in other cases too like
> >  separation of development and runtime libraries and header-only
> >  libraries, and maybe if we had some sort of general machinery for
> >  saying "run this code when you're imported" then it would also help
> >  with vendoring libraries on Windows. Unfortunately, build-time
> >  configuration is something that distutils is extremely terrible
> >  at...
> >
>
>
> I think this is important, not because of Debian/conda who can pretty
> easily carry a patch, but because of people who want to install on
> those platforms using something like pip, but depend on the platform
> OpenSSL (or whatever). Right now those people can get that by doing:
>
>     pip install —no-binary cryptography
>
> and I think it would be a regression to lose that. Perhaps we could
> leverage some other mechanism for the runtime bits. One idea that
> springs to mind (which may be terrible) is using a .pth file to
> enable the environment modifications instead of needing people to do
> it at runtime in their library code. This would mean that it gets an
> implicit enable() call for every Python process regardless of whether
> it needs it or not, which may be a bad thing I’m not sure.

Huh, that's an interesting and slightly terrifying idea. Can a wheel drop a
.pth file into site-packages? I mean, does that even work technically?

> If it is
> we could possibly use sys.meta_path to register an import hook which
> only enabled on import of a using lib and otherwise just did nothing.

I think this hits the bootstrap problem: who's going to modify
sys.meta_path before our package starts loading?

The alternative also is probably not *that* bad. It's just bad enough that
I decided I needed to stop fiddling and get this posted and more eyes on it
before trying to solve it properly :-). But basically during your build
you'd ask the pynativelib object to drop some generated file into your
source, and then modify your __init__.py to do something like

  import ._pynativelib_openssl_enable

There are a lot of fiddly variations on this idea and I'm not sure which is
the cleanest. And making this work with distutils requires some nasty hack,
because everything in distutils requires some nasty hack, but it's
basically the same nasty hack as all the other cases where you have to auto
generate a source file during the build -- e.g. every project using cython
already does something morally equivalent to sneak the Cython-generated .c
files into place before distutils tries compiling them.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20160411/0dbf1d2d/attachment.html>