[Distutils] PEP 426 moved back to Draft status

[Nick Coghlan]

On 14 March 2017 at 09:41, Nathaniel Smith <njs at pobox.com> wrote:

> On Fri, Mar 10, 2017 at 7:55 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > On 11 March 2017 at 00:52, Nathaniel Smith <njs at pobox.com> wrote:
> >> We have lots of metadata files in the wild that already claim to be
> >> version 2.0. If you're reviving this I think you might need to change
> >> the version number?
> >
> > They're mostly in metadata.json files, though. That said, version numbers
> > are cheap, so I'm happy to skip straight to 3.0 if folks think it makes
> more
> > sense.
>
> AFAICT bdist_wheel produces METADATA files with Metadata-Version: 2.0
> by default, and has for some time. Certainly this one I just
> spot-checked does that.
>

We could always retroactively declare "2.0" to just mean 1.3 +
Provides-Extra + (optionally) Description-Content-Type (once that has been
defined in a way that makes sense for PyPI).

Either way, I'm convinced that the JSON based format should start out at
3.0.


> > There's a lot to be said for treating the file as immutable, and instead
> > adding *other* metadata files as a component moves through the
> distribution
> > process. If so, then it may actually be more appropriate to call the
> > rendered file "pysdist.json", since it contains the sdist metadata
> > specifically, rather than arbitrary distribution metadata.
>
> I guess there are three possible kinds of build dependencies:
> - those that are known statically
> - those that are determined by running some code at sdist creation time
> - those that are determined by running some code at build time
>
> But all the examples I can think of fall into either bucket A (which
> pyproject.toml handles), or bucket C (which pydist.json can't handle).
> So it seems like the metadata here is either going to be redundant or
> wrong?
>

pyproject.toml only handles the bootstrapping dependencies for the build
system itself, it *doesn't* necessarily include all the build dependencies,
which may be in tool specific files (like setup_requires in setup.py) or
otherwise added by the build system without and record of it in
pyproject.toml. The build system knows the latter when it generates the
sdist, and it means PyPI can extract and republish them without having to
actually invoke the build system.

For dynamic dependencies where the environment marker system isn't flexible
enough to express the installation conditions (so they can't be generated
at sdist creation time), that will be something for the publishers of a
particular project to resolve with the folks that want the ability to do
builds in environments that are isolated from the internet, and hence can't
download arbitrary additional dependencies at build time.


> I'm not sure I understand the motivation for wanting wheels to have a
> file which says "here's the metadata describing the sdist that you
> would have, if you had an sdist (which you don't)"? I guess it doesn't
> hurt anything, but it seems odd.
>

Wheels still have a corresponding source artifact, even if it hasn't been
published anywhere using the Python-specific sdist format. Accordingly, I
don't think it makes sense to be able to tell just from looking at a wheel
file whether the generation process was:

* tree -> sdist -> wheel; or
* tree -> wheel

> I'd also be fairly strongly opposed to converting extras from an optional
> > dependency management system to a "let multiple PyPI packages target the
> > same site-packages subdirectory" because we already know that's a
> nightmare
> > from the Linux distro experience (having a clear "main" package that owns
> > the parent directory with optional subpackages solves *some* of the
> > problems, but my main reaction is still "Run awaaay").
>
> The "let multiple PyPI packages target the same site-packages
> directory" problem is orthogonal to the reified extras proposal. I
> actually think we can't avoid handling the same site-packages
> directory problem, but the solution is namespace packages and/or
> better Conflicts: metadata.
>
> Example illustrating why the site-packages conflict problem arises
> independently of reified extras: people want to distribute numpy built
> against different BLAS backends, especially MKL (which is good but
> zero-cost proprietary) versus OpenBLAS (which is not as good but is
> free). Right now that's possible by distributing 'numpy' and
> 'numpy-mkl' packages, but of course ugly stuff happens if you try to
> install both; some sort of Conflicts: metadata would help. If we
> instead have the packages be named 'numpy' and 'numpy[mkl]', then
> they're in exactly the same position with respect to conflicts. The
> very significant advantage is that we know that 'numpy[mkl]' "belongs
> to" the numpy project, so 'numpy[mkl]' can say 'Provides-Dist: numpy'
> without all the security issues that Provides-Dist otherwise runs
> into.
>

Do other components need to be rebuilt or relinked if the NumPy BLAS
backend changes?

If the answer is yes, then this is something I'd strongly prefer to leave
to conda and other package management systems like Nix that better support
parallel installation of multiple versions of C/C++ dependencies.

If the answer is no, then it seems like a better solution might be to allow
for rich dependencies, where numpy could depend on
"_numpy_backends.openblas or _numpy_backends.mkl" and figure out the
details of exactly what's available and which one it's going to use at
import time.

Either way, contorting the Extras system to try to cover such a
significantly different set of needs doesn't seem like a good idea.

>
> Example illustrating why reifed extras are useful totally
> independently of site-packages conflicts: it would be REALLY NICE if
> numpy could say 'Provides-Dist: numpy[abi=7]' and then packages could
> depend on 'numpy[abi=7]' and have that do something sensible. This
> would be a pure virtual package.
>

PEP 459 has a whole separate "python.constraints" extension rather than
trying to cover environmental constraints within the existing Extras
system:
https://www.python.org/dev/peps/pep-0459/#the-python-constraints-extension


> > It especially isn't needed just to solve the "pip forgets what extras it
> > installed" problem - that technically doesn't even need a PEP to
> resolve, it
> > just needs pip to drop a pip specific file into the PEP 376 dist-info
> > directory that says what extras to request when doing future upgrades.
>
> But that breaks if people use a package manager other than pip, which
> is something we want to support, right? And in any case it requires a
> bunch more redundant special-case logic inside pip, to basically make
> extras act like virtual packages.
>

OK, it would still need a PEP to make the file name and format standardised
across tools. Either way, it's an "installed packages database" problem,
not a software publication problem.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170314/17c8be88/attachment-0001.html>