[Distutils] PEP 426 moved back to Draft status

[Brett Cannon]

On Fri, 10 Mar 2017 at 07:56 Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 11 March 2017 at 00:52, Nathaniel Smith <njs at pobox.com> wrote:
>
> On Fri, Mar 10, 2017 at 1:26 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> > Hi folks,
> >
> > After a few years of dormancy, I've finally moved the metadata 2.0
> > specification back to Draft status:
> >
> https://github.com/python/peps/commit/8ae8b612d4ea8b3bf5d8a7b795ae8aec48bbb7a3
>
> We have lots of metadata files in the wild that already claim to be
> version 2.0. If you're reviving this I think you might need to change
> the version number?
>
>
> They're mostly in metadata.json files, though. That said, version numbers
> are cheap, so I'm happy to skip straight to 3.0 if folks think it makes
> more sense.
>

+1 on jumping.


>
>
> > Based on our last round of discussion, I've culled a lot of the
> complexity
> > around dependency declarations, cutting it back to just 4 pre-declared
> > extras (dev, doc, build, test),
>
> I think we can drop 'build' in favor of pyproject.toml?
>
>
> No, as that's a human edited input file, not an output file from the sdist
> generation process.
>
>
> Actually all of the pre-declared extras are really relevant for sdists
> rather than wheels. Maybe they should all move into pyproject.toml?
>
>
> Think "static release metadata in an API response from PyPI" for this
> particular specification, rather than something you'd necessarily check
> into source control.
>

Or "stuff PyPI has to parse, not you". ;)


> That's actually one of the big benefits of doing this post pyproject.toml
> -  with that taking care of the build system bootstrapping problem, it
> frees up pydist.json to be entirely an artifact of the sdist generation
> process (and then copying it along to the wheel archives and the installed
> package as well).
>
> That said, that's actually an important open question: is pydist.json
> always preserved unmodified through the sdist->wheel->install and
> sdist->install process?
>

Is there a reason not to?


>
> There's a lot to be said for treating the file as immutable, and instead
> adding *other* metadata files as a component moves through the distribution
> process. If so, then it may actually be more appropriate to call the
> rendered file "pysdist.json", since it contains the sdist metadata
> specifically, rather than arbitrary distribution metadata.
>

Since this is meant for tool consumption and not human consumption,
breaking the steps into individual files so that they are considered
immutable by tools farther down the toolchain makes sense to me.


>
>
> > and some reserved extras that can be used to
> > say "don't install this, even though you normally would" (self, runtime).
>
> Hmm. While it's not the most urgent problem we face, I really think in
> the long run we need to move the extras system to something like:
>
>
> https://mail.python.org/pipermail/distutils-sig/2015-October/027364.html
>
> The current extras system is inherently broken with respect to
> upgrades, and reified extras would solve this, along with several
> other intractable problems (e.g. numpy ABI tracking).
>
> So from that perspective, I'm wary of adding new special case "magic"
> to the extras system. Adding conventional names for things like
> test-dependencies is fine, that doesn't pose any new obstacles to a
> future migration. But adding complexity to the "extras language" like
> "*", "self", "runtime", etc. does make it harder to change how extras
> work in the future.
>
>
> Technically the only part of that which the PEP really locks in is barring
> the use of "self" and "runtime" as extras names (which needs to be
> validated by a check against currently published metadata to see if anyone
> is already using them).
>

Do you have something planned for these names?


>
> '*' is already illegal due to the naming rules, and the '-extra' syntax is
> also an illegal name, so neither of those actually impacts the metadata
> format, only what installation tools allow. The main purpose of having them
> in the PEP is to disallow using those spellings for anything else and
> instead reserve them for the purposes described in the PEP.
>
> I'd also be fairly strongly opposed to converting extras from an optional
> dependency management system to a "let multiple PyPI packages target the
> same site-packages subdirectory" because we already know that's a nightmare
> from the Linux distro experience (having a clear "main" package that owns
> the parent directory with optional subpackages solves *some* of the
> problems, but my main reaction is still "Run awaaay").
>
> It especially isn't needed just to solve the "pip forgets what extras it
> installed" problem - that technically doesn't even need a PEP to resolve,
> it just needs pip to drop a pip specific file into the PEP 376 dist-info
> directory that says what extras to request when doing future upgrades.
> Similarly, the import system offers so much flexibility in checking for
> optional packages at startup and lying about where imports are coming from
> that it would be hard to convince me that installation customisation to use
> particular optional dependencies *had* to be done at install time.
>
>
> I feel like most of the value we get out of these could be had by just
> standardizing the existing convention that packages should have an
> explicit "all" extra that includes all the feature-based extras,
>
>
> That's the first I've heard of that convention, so it may not be as
> widespread as you thought it was :)
>
>
> but
> not the special development extras? This also provides flexibility for
> cases like, a package where there are two extras that conflict with
> each other -- the package authors can pick which one they recommend to
> put into "all".
>
>
> That's actually the main problem I had with '*' - it didn't work anywhere
> near as nicely once the semantic dependencies were migrated over to being
> part of the extras system.
>
> Repeating the same dependencies under multiple extra names in order to
> model pseudo-sets seems error prone and messy to me, though.
>
> So perhaps we should add the notion of "extra_sets" as a first class
> entity, where they're named sets of declared extras? And if you don't
> declare an "all" set explicitly, you get an implied one that consists of
> all your declared extras.
>

I think that's a tool decision that doesn't tie into the PEP (unless you're
going to ban the use of the name "all").


>
> For migration of existing metadata that uses "all" as a normal extra, the
> translation would be:
>
> - declared extras are added to "all" in order until all of the
> dependencies in all are covered or all declared extras are included
> - any dependency in "all" that isn't in another extra gets added to a new
> "_all" extra
> - "extras" and "extra_sets" are populated accordingly
>
> Tools consuming the metadata would then just need to read "extra_sets" and
> expand any named sets before passing the list of extras over to their
> existing dependency processing machinery.
>

If this is meant to be generated by pyproject.toml consumers then I think
it should be up to the build tools to support that concept. Then the build
tools can statically declare the union of some extras to get extra sets
since the information isn't changing once the pydist.json file is generated
(dynamic calculation is only necessary if the value could change between
data generation and consumption).


>
> > I've also deleted a lot of the text related to thing that we now don't
> need
> > to worry about until the first few standard metadata extensions are being
> > defined.
> >
> > I think the biggest thing it needs right now is a major editing pass from
> > someone that isn't me to help figure out which explanatory sections can
> be
> > culled completely, while still having the specification itself make
> sense.
> >
> > From a technical point of view, the main "different from today" piece
> that
> > we have left is the Provide & Obsoleted-By fields, and I'm seriously
> > wondering if it might make sense to just delete those entirely for now,
> and
> > reconsider them later as a potential metadata extension.
>
> Overall the vibe I get from the Provides and Obsoleted-By sections is
> that these are surprisingly complicated and could really do with their
> own PEP, yeah, where the spec will have room to breathe and properly
> cover all the details.
>
> In particular, the language in the "provides" spec about how the
> interpretation of the metadata depends on whether you get it from a
> public index server versus somewhere else makes me really nervous.
>
>
> Yeah, virtual provides are a security nightmare on a public index server -
> distros are only able to get away with it because they maintain relatively
> strict control over the package review process.
>
>
> Experience suggests that splitting up packaging PEPs is basically
> never a bad idea, right? :-)
>
>
> Indeed :)
>
> OK, I'll put them on the chopping block too, under the assumption they may
> come back as an extension some day if it ever makes it to the top of
> someone's list of "thing that bothers them enough about Python packaging to
> do something about it".
>
>
> As a general note I guess I should say that I'm still not convinced
> that migrating to json is worth the effort, but you've heard those
> arguments before and I don't have anything new to add now, so :-).
>
>
> The main benefit I see will be to empower utility APIs like distlib (and
> potentially Warehouse itself) to better hide both the historical and
> migratory cruft by translating everything to the PEP 426 format, even if
> the source artifact only includes the legacy metadata. Unless the plumbing
> actually breaks, nobody other than the plumber cares when it's a mess, as
> long as the porcelain is shiny and clean :)
>
> Cheers,
> Nick.
>
> P.S. Something I'm getting out of this experience: if you can afford to
> sit on your hands for 3-4 years, that's a *really good way* to avoid
> falling prey to "second system syndrome" [1] :)
>
> P.P.S Having no budget to pay anyone else and only limited time and
> attention of your own also turns out to make it easier to avoid ;)
>

Yes, getting to stew on an idea for any length of time lets those random
ideas one gets to properly die when they are bad. ;)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170310/5045d120/attachment-0001.html>