[Wheel-builders] [RFC] Proposal for packaging native libraries into Python wheels

[Donald Stufft]

> On Apr 11, 2016, at 4:50 AM, Nathaniel Smith <njs at pobox.com> wrote:
> 
> **What about executables, like the ``openssl`` command-line tool?** We
> can handle this by stashing the executable itself in another hidden
> directory::
> 
>    pynativelib_openssl/
>        <...all the stuff above...>
>        _bins/
>            openssl
> 
> and then install a Python dispatch script (using the same mechanisms
> as we'd use to ship any Python script with wheel), where the script
> looks like::
> 
>    #!python
> 
>    import os
>    import sys
> 
>    import pynativelib_openssl
> 
>    pynativelib_openssl.enable()
>    os.execv(pynativelib_openssl._openssl_bin_path, sys.argv)


You don’t need to do this indirection, distutils has a scripts= option
which allows you to pass arbitrary files that will get installed into
the bin dir with executable bit set.

> 
> 
> - **How can we make this play better with other distribution
>  mechanisms?** E.g. Debian or conda package builders who when
>  building ``cryptography`` or ``numpy`` actually *want* them to link
>  to an external non-pynativelib version of libssl and libblas.
> 
>  This is the one place where I'm most uncertain about the design
>  described above.
> 
>  The minimal answer is "well, Debian/conda/... can carry a little
>  patch to their copy of ``cryptography``/``numpy``, and that's not
>  wrong, but it would be really nice if we could do better than
>  that. I'm not sure how. The biggest issue is that in the design
>  above, downstream packages like ``numpy`` have a hardcoded ``import
>  pynativelib_...`` in their ``__init__.py``, which should be removed
>  when linked against a non-pynativelib version of the library. What
>  would be very nice is if we could make the public pynativelib
>  interface be entirely a build-time thing -- so that instead of a
>  hard-coded runtime import, the build-time interface would provide
>  the code that needed to be called at run-time. (Perhaps it could
>  drop it into a special ``.py`` file as part of the build process,
>  and then the package ``__init__.py`` would have to import that.)
>  This would also potentially be useful in other cases too like
>  separation of development and runtime libraries and header-only
>  libraries, and maybe if we had some sort of general machinery for
>  saying "run this code when you're imported" then it would also help
>  with vendoring libraries on Windows. Unfortunately, build-time
>  configuration is something that distutils is extremely terrible
>  at...
> 


I think this is important, not because of Debian/conda who can pretty
easily carry a patch, but because of people who want to install on
those platforms using something like pip, but depend on the platform
OpenSSL (or whatever). Right now those people can get that by doing:

    pip install —no-binary cryptography

and I think it would be a regression to lose that. Perhaps we could
leverage some other mechanism for the runtime bits. One idea that
springs to mind (which may be terrible) is using a .pth file to
enable the environment modifications instead of needing people to do
it at runtime in their library code. This would mean that it gets an
implicit enable() call for every Python process regardless of whether
it needs it or not, which may be a bad thing I’m not sure. If it is
we could possibly use sys.meta_path to register an import hook which
only enabled on import of a using lib and otherwise just did nothing.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 842 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/wheel-builders/attachments/20160411/a93110ad/attachment.sig>