[Distutils] New PEP : dependency specification

[Robert Collins]

On unknown names, current pkg_resources raises SyntaxError. So I think
we need to update the spec from that perspective to be clear. All PEP
426 defined variables are handled without error by current
pkg_resources/markerlib implementation. Perhaps the new variables
should raise rather than evaluating to '' / 0 ? Some discussion /
thought would be good. Certainly when used and evaluated by an
existing pkg_resources they will error - so perhaps we should not add
any new variables at this point?

So, if we don't unify this with the wheel encoding of extras, it will
require multiple parsers indefinitely. I need to think more about
whether it makes sense or not.

Wheel certainly needs a way to say 'this distribution exports extras
X, Y, Z (and their respective dependencies)'. flit and other tools
producing wheels need the same facility.

https://www.python.org/dev/peps/pep-0427 doesn't define this use of
markers; but pip and wheel have collaborated on it. PEP-345 doesn't
describe Provides-Extra, which pkg_resources uses when parsing
.dist-info directories as well (it determines which extra variables
get substituted into the set of requires to determine the values of
the extras...). So there's basically still a bunch of underspecified
behaviours out there in the wild, and since my strategy is to minimal
variation vs whats there, we need to figure out the right places to
split things to layer this well.

Specifying a new variable of 'extra' is fairly easy: we need to decide
on the values it will take, and thats well defined but layer crossing:
when processing a dependency with one or more extras, you need to loop
over all the dependency specifications once with each extra defined
(including I suspect for completeness '' for the non-extras) and then
union together the results.

So at this layer I think we could say that:
 - extra is only valid if the context that is interpreting the
specification defines it
 - when invalid it will raise SyntaxError

This allows a single implementation to handle .dist-info directories
as it does today, while specifying it more fully. It leaves it open in
future for us to continue modelling exported extras as
marker-filtered-specifications + a Provides-Extra, or to move to
exported extras as something in a hash in a richer serialisation
format, or some third thing. This is good I think.

I do like the idea of comments and line continuations being removed.
We can then explicitly say that this DSL is going to be embedded in a
larger context such as requirements.txt files, requires headers etc,
and that those contexts may provide multi line handling as desired.

I'll apply Nathaniel's excellent review details + this on Monday and
issue an update.

-Rob

On 8 November 2015 at 05:08, Marcus Smith <qwcode at gmail.com> wrote:
>>
>> I'm not sure this is the syntax that I would have come up with, but I
>> guess it's too late to put the genie back in the bottle, so this PEP
>> should have some way to cope with these things?
>
>
> why would this PEP deal with this?
> the higher level PEP that builds on top of this would bump the wheel
> metadata version (probably to 3.0, due to the jump to 2.0 already)
> *then* tools will cope with this based on the metadata version
>



-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud