[Wheel-builders] [RFC] Proposal for packaging native libraries into Python wheels

[Nick Coghlan]

On 11 April 2016 at 22:03, Donald Stufft <donald at stufft.io> wrote:
> I think this is important, not because of Debian/conda who can pretty
> easily carry a patch, but because of people who want to install on
> those platforms using something like pip, but depend on the platform
> OpenSSL (or whatever). Right now those people can get that by doing:
>
>     pip install —no-binary cryptography
>
> and I think it would be a regression to lose that.

I missed the start of this thread, but yes - once you're no longer
supporting linking with system packages, you've become conda (a
complete binary platform in its own right) rather than pip (a plugin
manager for Python runtimes).

> Perhaps we could
> leverage some other mechanism for the runtime bits. One idea that
> springs to mind (which may be terrible) is using a .pth file to
> enable the environment modifications instead of needing people to do
> it at runtime in their library code.

Aside from slowing down Python startup time, the other main challenge
with .pth files is that they don't get run if Python is started
without running the site module (which is standard practice for Linux
system scripts, for example).

> This would mean that it gets an
> implicit enable() call for every Python process regardless of whether
> it needs it or not, which may be a bad thing I’m not sure. If it is
> we could possibly use sys.meta_path to register an import hook which
> only enabled on import of a using lib and otherwise just did nothing.

A meta_path hook should work fine, and it can be pure Python to
address the bootstrapping problem (as long as whatever native libs you
need are loaded before the affected extension module starts loading
there shouldn't be a problem). Pure extension modules with no Python
component would potentially face a problem, but the
Python-module-with-C-accelerator pattern could potentially address
those.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia