[Distutils] distlib and wheel metadata

[Nathaniel Smith]

On Feb 15, 2017 07:41, "Nick Coghlan" <ncoghlan at gmail.com> wrote:

>> pipenv borrows the Ruby solution to modeling this by having Pipfile
>> for abstract dependency declarations and Pipfile.lock for concrete
>> integration testing ones, so the idea here is to propagate that model
>> to pydist.json by separating the "requires" field with abstract
>> development dependencies from the "integrates" field with concrete
>> deployment dependencies.
>
> What's the benefit of putting this in pydist.json? I feel like for the
> usual deployment cases (a) going straight from Pipfile.lock -> venv is
> pretty much sufficient, with no need to put this into a package, but
> (b) if you really do want to put it into a package, then the natural
> approach would be to make an empty wheel like
> "my-django-app-deploy.whl" whose dependencies were the contents of
> Pipfile.lock.

My goal with the split is to get to a state where:

- exactly zero projects on PyPI use "==" or "===" in their requires
metadata (because PyPI explicitly prohibits it)
- the vast majority of projects on PyPI *don't* have an "integrates" section
- those projects that do have an `integrates` section have a valid
reason for it (like PyObjC)

For anyone making the transition from application and web service
development to library and framework development, the transition from
"always pin exact versions of your dependencies for deployment" to
"when publishing a library or framework, only rule out the
combinations that you're pretty sure *won't* work" is one of the
trickiest to deal with as current tools *don't alert you to the fact
that there's a difference to be learned*.

Restricting what can go into requires creates an opportunity to ask
users whether they're publishing a pre-integrated project or not: if
yes, then they add the "integrates" field and put their pinned
dependencies there; if not, then they relax the "==" constraints to
"~=" or ">=".


Ah-hah, this does make sense as a problem, thanks!

However, your solution seems very odd to me :-).

If the goal is to put an "are you sure/yes I'm sure" UX barrier between
users and certain version settings, then why make a distinction that every
piece of downstream software has to be aware of and ignore? Pypi seems like
a funny place in the stack to be implementing this. It would be much
simpler to implement this feature at the build system level, like e.g.
setuptools could require that dependencies that you think are over strict
be specified in an install_requires_yes_i_really_mean_it= field, without
requiring any metadata changes.

Basically it sounds like you're saying you want to extend the metadata so
that it can represent both broken and non-broken packages, so that both can
be created, passed around, and checked for. And I'm saying, how about
instead we do that checking when creating the package in the first place.

(Of course I can't see any way to do any of this that won't break existing
sdists, but I guess you've already decided you're OK with that. I guess I
should say that I'm a bit dubious that this is so important in the first
place; I feel like there are lots of legitimate use cases for ==
dependencies and lots of kinds of linting we might want to apply to try and
improve the level of packaging quality.)


Either way, PyPI will believe your answer, it's just refusing the
temptation to guess that using "==" or "===" in the requires section
is sufficient to indicate that you're deliberately publishing a
pre-integrated project.

> There's certainly a distinction to be made between the abstract
> dependencies and the exact locked dependencies, but to me the natural
> way to model that distinction is by re-using the distinction we
> already have been source packages and binary packages. The build
> process for this placeholder wheel is to "compile down" the abstract
> dependencies into concrete dependencies, and the resulting wheel
> encodes the result of this compilation. Again, no new concepts needed.

Source vs binary isn't where the distinction applies, though. For
example, it's legitimate for PyObjC to have pinned dependencies even
when distributed in source form, as it's a metapackage used solely to
integrate the various PyObjC subprojects into a single "release".


?? So that means that some packages have a loosely specified source that
compiles down to a more strictly specified binary, and some have a more
strictly specified source that compiles down to an equally strictly
specified binary. That's... an argument in favor of my way of thinking
about it, isn't it? That it can naturally express both situations?

My point is that *for the cases where there's an important distinction
between Pipfile and Pipfile.lock*, we already have a way to think about
that distinction without introducing new concepts.

-n
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20170215/b1768910/attachment-0001.html>