[Distutils] New PEP : dependency specification

Nathaniel Smith njs at pobox.com
Sat Nov 7 23:10:46 EST 2015

On Sat, Nov 7, 2015 at 12:32 PM, Robert Collins
<robertc at robertcollins.net> wrote:
> On unknown names, current pkg_resources raises SyntaxError. So I think
> we need to update the spec from that perspective to be clear. All PEP
> 426 defined variables are handled without error by current
> pkg_resources/markerlib implementation. Perhaps the new variables
> should raise rather than evaluating to '' / 0 ? Some discussion /
> thought would be good. Certainly when used and evaluated by an
> existing pkg_resources they will error - so perhaps we should not add
> any new variables at this point?

I just thought of one possible strategy for allowing future extensions
without opening the door wide to every typo: add a variable which
contains the version number of the environment marker evaluation
system, define the and/or operators as being short-circuiting, and
keep the rule that trying to access an unknown variable name raises an
error. Then you could use write things like:

# 'foo is only really needed if some_new_attr > "2", but doesn't hurt otherwise.
# So when using an old installation tool that doesn't know about some_new_attr,
# install it unconditionally.
Requires-Dist: foo; marker_handling >= "2" and some_new_attr > 2
Requires-Dist: foo; marker_handling < "2"

> So, if we don't unify this with the wheel encoding of extras, it will
> require multiple parsers indefinitely. I need to think more about
> whether it makes sense or not.
> Wheel certainly needs a way to say 'this distribution exports extras
> X, Y, Z (and their respective dependencies)'. flit and other tools
> producing wheels need the same facility.
> https://www.python.org/dev/peps/pep-0427 doesn't define this use of
> markers; but pip and wheel have collaborated on it. PEP-345 doesn't
> describe Provides-Extra, which pkg_resources uses when parsing
> .dist-info directories as well (it determines which extra variables
> get substituted into the set of requires to determine the values of
> the extras...). So there's basically still a bunch of underspecified
> behaviours out there in the wild, and since my strategy is to minimal
> variation vs whats there, we need to figure out the right places to
> split things to layer this well.
> Specifying a new variable of 'extra' is fairly easy: we need to decide
> on the values it will take, and thats well defined but layer crossing:
> when processing a dependency with one or more extras, you need to loop
> over all the dependency specifications once with each extra defined
> (including I suspect for completeness '' for the non-extras) and then
> union together the results.
> So at this layer I think we could say that:
>  - extra is only valid if the context that is interpreting the
> specification defines it
>  - when invalid it will raise SyntaxError
> This allows a single implementation to handle .dist-info directories
> as it does today, while specifying it more fully. It leaves it open in
> future for us to continue modelling exported extras as
> marker-filtered-specifications + a Provides-Extra, or to move to
> exported extras as something in a hash in a richer serialisation
> format, or some third thing. This is good I think.

This seems like a good strategy to me.


Nathaniel J. Smith -- http://vorpus.org

More information about the Distutils-SIG mailing list