[Distutils] distlib and wheel metadata

[Nick Coghlan]

On 23 February 2017 at 18:03, Nick Coghlan <ncoghlan at gmail.com> wrote:

>
>
> On 15 Feb 2017 23:40, "Nathaniel Smith" <njs at pobox.com> wrote:
>
> On Feb 15, 2017 07:41, "Nick Coghlan" <ncoghlan at gmail.com> wrote:
>
>
>
> Ah-hah, this does make sense as a problem, thanks!
>
> However, your solution seems very odd to me :-).
>
> If the goal is to put an "are you sure/yes I'm sure" UX barrier between
> users and certain version settings, then why make a distinction that every
> piece of downstream software has to be aware of and ignore? Pypi seems like
> a funny place in the stack to be implementing this. It would be much
> simpler to implement this feature at the build system level, like e.g.
> setuptools could require that dependencies that you think are over strict
> be specified in an install_requires_yes_i_really_mean_it= field, without
> requiring any metadata changes.
>
>
> If you're publishing to a *private* index server then version pinning
> should be allowed by default and you shouldn't get a warning.
>
> It's only when publishing to PyPI as a *public* index server that overly
> restrictive dependencies become a UX problem.
>
> The simplest way of modelling this that I've come up with is a boolean
> "allow pinned dependencies" flag - without the flag, "==" and "===" would
> emit warnings or errors when releasing to a public index server, with it
> they wouldn't trigger any complaints.
>
> Basically it sounds like you're saying you want to extend the metadata so
> that it can represent both broken and non-broken packages, so that both can
> be created, passed around, and checked for. And I'm saying, how about
> instead we do that checking when creating the package in the first place.
>
>
> Build time isn't right, due to this being a perfectly acceptable thing to
> do when building solely for private use. It's only you make the "I'm going
> to publish this for the entire community to use" that the intent needs to
> be clarified (as at that point you're switching from "I'm solving to my own
> problems" to "My problems may be shared by other people, and I'd like to
> help them out if I can").
>

And TIL that Ctrl-Enter is Gmail's keyboard shortcut for sending an email :)


> (Of course I can't see any way to do any of this that won't break existing
> sdists, but I guess you've already decided you're OK with that. I guess I
> should say that I'm a bit dubious that this is so important in the first
> place; I feel like there are lots of legitimate use cases for ==
> dependencies and lots of kinds of linting we might want to apply to try and
> improve the level of packaging quality.)
>
>
Existing sdists won't have pydist.json, so none of this will apply.


>
> Either way, PyPI will believe your answer, it's just refusing the
> temptation to guess that using "==" or "===" in the requires section
> is sufficient to indicate that you're deliberately publishing a
> pre-integrated project.
>
> > There's certainly a distinction to be made between the abstract
> > dependencies and the exact locked dependencies, but to me the natural
> > way to model that distinction is by re-using the distinction we
> > already have been source packages and binary packages. The build
> > process for this placeholder wheel is to "compile down" the abstract
> > dependencies into concrete dependencies, and the resulting wheel
> > encodes the result of this compilation. Again, no new concepts needed.
>
> Source vs binary isn't where the distinction applies, though. For
> example, it's legitimate for PyObjC to have pinned dependencies even
> when distributed in source form, as it's a metapackage used solely to
> integrate the various PyObjC subprojects into a single "release".
>
>
> ?? So that means that some packages have a loosely specified source that
> compiles down to a more strictly specified binary, and some have a more
> strictly specified source that compiles down to an equally strictly
> specified binary. That's... an argument in favor of my way of thinking
> about it, isn't it? That it can naturally express both situations?
>
>
Why are you bringing source vs binary into this? That has *nothing* to do
with the problem space, which is about the large grey area between
"definitely doesn't work" (aka "we tested this combination and it failed")
and "will almost certainly work" (aka "we tested this specific combination
of dependencies and it passed").

When publishing a software *component* (such as a library or application),
the most important information to communicate to users is the former (i.e.
the combinations you know *don't* work), while for applications & services
you typically want to communicate *both* (i.e. the combinations you know
definitively *don't* work, *and* the specific combinations you tested).

While you do need to do at least one build to actually run the tests, once
you have those results, the metadata is just as applicable to the original
source artifact as it is to the specific built binary.


> My point is that *for the cases where there's an important distinction
> between Pipfile and Pipfile.lock*, we already have a way to think about
> that distinction without introducing new concepts.
>
>
Most software components won't have a Pipfile or Pipfile.lock, as that's an
application & service oriented way of framing the distinction.

However, as Daniel said in his reply, we *do* want people to be able to
publish applications and services like sentry or supervisord to PyPI, and
we also want to allow people to publish metapackages like PyObjC.

The problem I'm trying to address is that we *don't* currently give
publishers a machine readable way to say definitively "This is a
pre-integrated application, service or metapackage" rather than "This is a
component intended for integration into a larger application, service or
metapackage".

I'm not a huge fan of having simple boolean toggles in metadata definitions
(hence the more elaborate idea of two different kinds of dependency
declaration), but this may be a case where that's a good way to go, since
it would mean that services and tools that care can check it (with a
recommendation in the spec saying that public index servers SHOULD check
it), while those that don't care would continue to have a single unified
set of dependency declarations to work with.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170223/db876b53/attachment-0001.html>