<div dir="ltr"><div dir="ltr" class="gmail_msg">You lost me a bit at 'extra sets'. FYI it is already possible to depend on your own extras in another extra.</div><div dir="ltr" class="gmail_msg"><br></div><div class="gmail_msg">Extra pseudo code:</div><div dir="ltr" class="gmail_msg">spampackage</div><div class="gmail_msg">extra['spam'] = 'spampackage[eggs]'</div><div class="gmail_msg">extra['eggs'] = ...</div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">+1 on extras. The extras feature has the wonderful property that people understand it. Lots of projects have a 'test' extra instead of tests_require for example, and you don't have to look up how to install them.</div></div><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Fri, Mar 10, 2017 at 1:14 PM Brett Cannon <<a href="mailto:brett@python.org" class="gmail_msg" target="_blank">brett@python.org</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Fri, 10 Mar 2017 at 07:56 Nick Coghlan <<a href="mailto:ncoghlan@gmail.com" class="gmail_msg" target="_blank">ncoghlan@gmail.com</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg">On 11 March 2017 at 00:52, Nathaniel Smith <span dir="ltr" class="gmail_msg"><<a href="mailto:njs@pobox.com" class="gmail_msg" target="_blank">njs@pobox.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="m_7141403437789079588m_-1080643061533729559m_-2400512228545068137gmail- gmail_msg">On Fri, Mar 10, 2017 at 1:26 AM, Nick Coghlan <<a href="mailto:ncoghlan@gmail.com" class="gmail_msg" target="_blank">ncoghlan@gmail.com</a>> wrote:<br class="gmail_msg">
> Hi folks,<br class="gmail_msg">
><br class="gmail_msg">
> After a few years of dormancy, I've finally moved the metadata 2.0<br class="gmail_msg">
> specification back to Draft status:<br class="gmail_msg">
> <a href="https://github.com/python/peps/commit/8ae8b612d4ea8b3bf5d8a7b795ae8aec48bbb7a3" rel="noreferrer" class="gmail_msg" target="_blank">https://github.com/python/peps/commit/8ae8b612d4ea8b3bf5d8a7b795ae8aec48bbb7a3</a><br class="gmail_msg">
<br class="gmail_msg">
</span>We have lots of metadata files in the wild that already claim to be<br class="gmail_msg">
version 2.0. If you're reviving this I think you might need to change<br class="gmail_msg">
the version number?<br class="gmail_msg"></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">They're mostly in metadata.json files, though. That said, version numbers are cheap, so I'm happy to skip straight to 3.0 if folks think it makes more sense.<br class="gmail_msg"></div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">+1 on jumping.<br class="gmail_msg"></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="m_7141403437789079588m_-1080643061533729559m_-2400512228545068137gmail- gmail_msg">
> Based on our last round of discussion, I've culled a lot of the complexity<br class="gmail_msg">
> around dependency declarations, cutting it back to just 4 pre-declared<br class="gmail_msg">
> extras (dev, doc, build, test),<br class="gmail_msg">
<br class="gmail_msg">
</span>I think we can drop 'build' in favor of pyproject.toml?<br class="gmail_msg"></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">No, as that's a human edited input file, not an output file from the sdist generation process.<br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Actually all of the pre-declared extras are really relevant for sdists<br class="gmail_msg">
rather than wheels. Maybe they should all move into pyproject.toml?<br class="gmail_msg"></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Think "static release metadata in an API response from PyPI" for this particular specification, rather than something you'd necessarily check into source control. </div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Or "stuff PyPI has to parse, not you". ;)</div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">That's actually one of the big benefits of doing this post pyproject.toml - with that taking care of the build system bootstrapping problem, it frees up pydist.json to be entirely an artifact of the sdist generation process (and then copying it along to the wheel archives and the installed package as well).<br class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">That said, that's actually an important open question: is pydist.json always preserved unmodified through the sdist->wheel->install and sdist->install process?<br class="gmail_msg"></div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Is there a reason not to?</div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">There's a lot to be said for treating the file as immutable, and instead adding *other* metadata files as a component moves through the distribution process. If so, then it may actually be more appropriate to call the rendered file "pysdist.json", since it contains the sdist metadata specifically, rather than arbitrary distribution metadata.<br class="gmail_msg"></div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Since this is meant for tool consumption and not human consumption, breaking the steps into individual files so that they are considered immutable by tools farther down the toolchain makes sense to me.</div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"> <blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="m_7141403437789079588m_-1080643061533729559m_-2400512228545068137gmail- gmail_msg">
> and some reserved extras that can be used to<br class="gmail_msg">
> say "don't install this, even though you normally would" (self, runtime).<br class="gmail_msg">
<br class="gmail_msg">
</span>Hmm. While it's not the most urgent problem we face, I really think in<br class="gmail_msg">
the long run we need to move the extras system to something like:<br class="gmail_msg">
<br class="gmail_msg">
<a href="https://mail.python.org/pipermail/distutils-sig/2015-October/027364.html" rel="noreferrer" class="gmail_msg" target="_blank">https://mail.python.org/pipermail/distutils-sig/2015-October/027364.html</a><br class="gmail_msg">
<br class="gmail_msg">
The current extras system is inherently broken with respect to<br class="gmail_msg">
upgrades, and reified extras would solve this, along with several<br class="gmail_msg">
other intractable problems (e.g. numpy ABI tracking).<br class="gmail_msg">
<br class="gmail_msg">
So from that perspective, I'm wary of adding new special case "magic"<br class="gmail_msg">
to the extras system. Adding conventional names for things like<br class="gmail_msg">
test-dependencies is fine, that doesn't pose any new obstacles to a<br class="gmail_msg">
future migration. But adding complexity to the "extras language" like<br class="gmail_msg">
"*", "self", "runtime", etc. does make it harder to change how extras<br class="gmail_msg">
work in the future.<br class="gmail_msg"></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Technically the only part of that which the PEP really locks in is barring the use of "self" and "runtime" as extras names (which needs to be validated by a check against currently published metadata to see if anyone is already using them).<br class="gmail_msg"></div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Do you have something planned for these names?</div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">'*' is already illegal due to the naming rules, and the '-extra' syntax is also an illegal name, so neither of those actually impacts the metadata format, only what installation tools allow. The main purpose of having them in the PEP is to disallow using those spellings for anything else and instead reserve them for the purposes described in the PEP.<br class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I'd also be fairly strongly opposed to converting extras from an optional dependency management system to a "let multiple PyPI packages target the same site-packages subdirectory" because we already know that's a nightmare from the Linux distro experience (having a clear "main" package that owns the parent directory with optional subpackages solves *some* of the problems, but my main reaction is still "Run awaaay").<br class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">It especially isn't needed just to solve the "pip forgets what extras it installed" problem - that technically doesn't even need a PEP to resolve, it just needs pip to drop a pip specific file into the PEP 376 dist-info directory that says what extras to request when doing future upgrades. Similarly, the import system offers so much flexibility in checking for optional packages at startup and lying about where imports are coming from that it would be hard to convince me that installation customisation to use particular optional dependencies *had* to be done at install time.<br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> <br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
I feel like most of the value we get out of these could be had by just<br class="gmail_msg">
standardizing the existing convention that packages should have an<br class="gmail_msg">
explicit "all" extra that includes all the feature-based extras,</blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">That's the first I've heard of that convention, so it may not be as widespread as you thought it was :)<br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">but<br class="gmail_msg">
not the special development extras? This also provides flexibility for<br class="gmail_msg">
cases like, a package where there are two extras that conflict with<br class="gmail_msg">
each other -- the package authors can pick which one they recommend to<br class="gmail_msg">
put into "all".<br class="gmail_msg"></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">That's actually the main problem I had with '*' - it didn't work anywhere near as nicely once the semantic dependencies were migrated over to being part of the extras system.<br class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Repeating the same dependencies under multiple extra names in order to model pseudo-sets seems error prone and messy to me, though.<br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">So perhaps we should add the notion of "extra_sets" as a first class entity, where they're named sets of declared extras? And if you don't declare an "all" set explicitly, you get an implied one that consists of all your declared extras.<br class="gmail_msg"></div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">I think that's a tool decision that doesn't tie into the PEP (unless you're going to ban the use of the name "all").</div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">For migration of existing metadata that uses "all" as a normal extra, the translation would be:<br class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">- declared extras are added to "all" in order until all of the dependencies in all are covered or all declared extras are included<br class="gmail_msg"></div><div class="gmail_msg">- any dependency in "all" that isn't in another extra gets added to a new "_all" extra<br class="gmail_msg"></div><div class="gmail_msg">- "extras" and "extra_sets" are populated accordingly<br class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Tools consuming the metadata would then just need to read "extra_sets" and expand any named sets before passing the list of extras over to their existing dependency processing machinery.<br class="gmail_msg"></div></div></div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">If this is meant to be generated by pyproject.toml consumers then I think it should be up to the build tools to support that concept. Then the build tools can statically declare the union of some extras to get extra sets since the information isn't changing once the pydist.json file is generated (dynamic calculation is only necessary if the value could change between data generation and consumption).</div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="m_7141403437789079588m_-1080643061533729559m_-2400512228545068137gmail- gmail_msg">
> I've also deleted a lot of the text related to thing that we now don't need<br class="gmail_msg">
> to worry about until the first few standard metadata extensions are being<br class="gmail_msg">
> defined.<br class="gmail_msg">
><br class="gmail_msg">
> I think the biggest thing it needs right now is a major editing pass from<br class="gmail_msg">
> someone that isn't me to help figure out which explanatory sections can be<br class="gmail_msg">
> culled completely, while still having the specification itself make sense.<br class="gmail_msg">
><br class="gmail_msg">
> From a technical point of view, the main "different from today" piece that<br class="gmail_msg">
> we have left is the Provide & Obsoleted-By fields, and I'm seriously<br class="gmail_msg">
> wondering if it might make sense to just delete those entirely for now, and<br class="gmail_msg">
> reconsider them later as a potential metadata extension.<br class="gmail_msg">
<br class="gmail_msg">
</span>Overall the vibe I get from the Provides and Obsoleted-By sections is<br class="gmail_msg">
that these are surprisingly complicated and could really do with their<br class="gmail_msg">
own PEP, yeah, where the spec will have room to breathe and properly<br class="gmail_msg">
cover all the details.<br class="gmail_msg">
<br class="gmail_msg">
In particular, the language in the "provides" spec about how the<br class="gmail_msg">
interpretation of the metadata depends on whether you get it from a<br class="gmail_msg">
public index server versus somewhere else makes me really nervous.<br class="gmail_msg"></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Yeah, virtual provides are a security nightmare on a public index server - distros are only able to get away with it because they maintain relatively strict control over the package review process.<br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Experience suggests that splitting up packaging PEPs is basically<br class="gmail_msg">
never a bad idea, right? :-)<br class="gmail_msg"></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Indeed :)<br class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">OK, I'll put them on the chopping block too, under the assumption they may come back as an extension some day if it ever makes it to the top of someone's list of "thing that bothers them enough about Python packaging to do something about it".<br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg"> </div><blockquote class="gmail_quote gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
As a general note I guess I should say that I'm still not convinced<br class="gmail_msg">
that migrating to json is worth the effort, but you've heard those<br class="gmail_msg">
arguments before and I don't have anything new to add now, so :-).<br class="gmail_msg"></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">The main benefit I see will be to empower utility APIs like distlib (and potentially Warehouse itself) to better hide both the historical and migratory cruft by translating everything to the PEP 426 format, even if the source artifact only includes the legacy metadata. Unless the plumbing actually breaks, nobody other than the plumber cares when it's a mess, as long as the porcelain is shiny and clean :)<br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div></div>Cheers,<br class="gmail_msg"></div><div class="gmail_extra gmail_msg">Nick.<br class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_extra gmail_msg">P.S. Something I'm getting out of this experience: if you can afford to sit on your hands for 3-4 years, that's a *really good way* to avoid falling prey to "second system syndrome" [1] :)<br class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_extra gmail_msg">P.P.S Having no budget to pay anyone else and only limited time and attention of your own also turns out to make it easier to avoid ;)</div></div></blockquote><div class="gmail_msg"><br class="gmail_msg"></div></div></div><div dir="ltr" class="gmail_msg"><div class="gmail_quote gmail_msg"><div class="gmail_msg">Yes, getting to stew on an idea for any length of time lets those random ideas one gets to properly die when they are bad. ;) </div></div></div>
_______________________________________________<br class="gmail_msg">
Distutils-SIG maillist - <a href="mailto:Distutils-SIG@python.org" class="gmail_msg" target="_blank">Distutils-SIG@python.org</a><br class="gmail_msg">
<a href="https://mail.python.org/mailman/listinfo/distutils-sig" rel="noreferrer" class="gmail_msg" target="_blank">https://mail.python.org/mailman/listinfo/distutils-sig</a><br class="gmail_msg">
</blockquote></div></div>