[Distutils] Towards a simple and standard sdist format that isn't intertwined with distutils

[Chris Barker]

I'm not sure if I understand what Nathaniel is getting at, but...

> As the very simplest example, every package that uses the numpy C API
> > gets a runtime dependency on "numpy >= [whatever version happened to
> > be installed on the *build* machine]".



> From my point of view, it's not a source distribution or a binary
> distribution that depends on something (numpy or whatever) - it's the
> *project*. If project foo needs numpy to work, it depends on numpy. If
> it depends on features in numpy 1.9, it depends on numpy>=1.9.
>

Here is the gap (I'm a bit confused about what a "project" is -- so I"ll
use the term "package", meaning a python package).

A given package might depend on numpy, as you say, and it may work with all
numpy versions 1.6 to 1.9. Fine, so we specify that in install_requires.
And this shodl be the dependency in the sdist, too. If the package is pur
python, this is fine and done.

But if the package has some extensions code that used the numpy C API ( a
very common occurrence), then when it is built, it will only work
(reliably) with the version of numpy it was built with.

So the project itself, and the sdist depend on numpy >=1.6, but a build
binary wheel depends on numpy == 1.7 (for instance).

Which requires a binary (wheel) dependency that is somewhat different than
the source dependency.

In a way, this is a lot like a python package that may work fine on py2.7
to py3.5, but a binary wheel is for py3.4 specifically (yes?)

Note that conda, being developed originally for scipy, has packages like:

gsw-3.0.3-np19py27_0.tar.bz2

so this binary depends specifically on py2.7 and numpy1.9

but once you get beyond python_numpy, you cold get a REALLY BIG matrix of
possible version combnations!

nevertheless, I think it would be really helpful to have the concept of a
binary dependency that is distinct from the "package" dependency. Maybe
down to "this wheel depends on this other particular wheel" - not only
version, but also wheel specific.

That would open the door to making wheels for binary non-python
dependencies like libpng: then other wheels would depend on the libpng
wheel, but different wheels built from the same package might have those
dependencies handled differently, particularly on different platforms.

I can understand that a binary wheel may need a certain set of
> libraries installed - but that's about the platform tags that are part
> of the wheel definition, not about dependencies. Platform tags are an
> ongoing discussion, and a good example of a partial solution that
> needs to be extended, certainly, but they aren't really relevant in
> any way that I can see to how the build chain works.
>

I haven't followed the discussion well, but I really think platform tags
are never going to be specific enough to handle these use-cases And these
really ARE dependencies, even if they aren't pure-python ones...

And if my
> understanding is correct, I'm against trying to fit that information
> into a dependency simply to work around the current limitations of the
> platform tag mechanism.
>

I can't imagine how you could extend platform tags to cover all of this --
but maybe I'm being unimaginative...

BTW, numpy is nearly unique here -- I can't think of any other package that
is a python package with a C API that is widely used -- most other cases
are either a python package OR a regular old compiled lib. Which is
probably why conda can get away with essentially special casing it.

-Chris


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20151002/4d26ee17/attachment-0001.html>