[Distutils] New PEP : dependency specification

[Nathaniel Smith]

On Thu, Nov 5, 2015 at 7:32 PM, Robert Collins
<robertc at robertcollins.net> wrote:
> Since we ended up with a hard dependency on this for the bootstrap
> thing (regardless of 'smaller step' or not) - I've broken this out of
> PEP 426, made it an encoding of the current status quo rather than an
> aspirational change. Since it has a dependency on markers, I had to
> choose whether to block on James' marker PEP, contribute to that, or
> include it. I think on balance it makes sense to have it in one
> document since the markers bit is actually quite shallow, so I've done
> that (after discussing with James). This could perhaps replace PEP 496
> then, or be given a new number.
>
> Donald has graciously agreed to be a BDFL-delegate for it.
>
> The PR for it is https://github.com/pypa/interoperability-peps/pull/56

Thanks, this is really great!

I made a bunch of fiddly comments inline on the PR, but some more
general comments:

1) Also mentioned this in the PR, but it's probably worth putting up
for more general discussion here: do we want to define some graceful
degradation for how to handle unrecognized variable names in the
environment marker syntax, so as to allow more easily for future
extensions? In the current PEP draft, an unrecognized variable name is
simply a syntax error (all the variable names are effectively
keywords).

One option would be to declare that any expression that contains an
unrecognized variable simply evaluates to False. The downside of this
is that it's somewhat accident prone: "os_name == posix" would
silently be always false (because "posix" should be quoted as a
string, not left unquoted and treated as a variable name), and
"os_nmae == 'posix'" would also be silently accepted.

2) The PEP seems a little uncertain about whether it wants to talk
about the "framing protocol" or not -- like whether there's a
higher-level structure defining the edges of each individual
requirement or not. In setup.py and in existing METADATA files, you
have some higher level list-of-strings syntax and then each string is
parsed as a single individual requirement, and comments don't make
much sense; in requirements.txt then newlines are meaningful and
comments are important. The PEP worries about newlines and newlines,
but doesn't quite want to come out and say that it's defining
requirements.txt -- it wants to be more general.

Maybe it would be clearer to drop the comment and newline handling
stuff from the core requirement specifier syntax (declaring that
newlines are simply a syntax error), and assume that there's some
higher-level framing protocol taking care of that stuff? So METADATA
files would use whatever rules they use to pick out a single
requirement specifier string (probably not allowing e.g. comments) and
then parse it using this PEP's rules, and separately we'd have a
definition for requirements.txt which is basically "a text file where
you strip out comments, delete newlines that are preceded by
backslash, split into lines, discard empty lines, and each resulting
string is parsed as a requirement specifier".

3) The way extras are specified in METADATA files currently (ab)uses
the environment marker syntax. E.g. here's the METADATA files from two
popular packages:

  https://gist.github.com/njsmith/ed74851c0311e858f0f7

(Nice to see Metadata-Version: 2.0 getting some real-world use! I
guess??? I thought I was starting to understand what is going on with
python packaging standards but now I am baffled again. Anyway!)

So apparently this is how you say that there is an extra called "doc"
and that this extra adds a dependency on Sphinx:

Provides-Extra: doc
Requires-Dist: Sphinx (>=1.1); extra == 'doc'

(https://gist.github.com/njsmith/ed74851c0311e858f0f7#file-ipython-4-0-0-wheel-metadata-L39)

And here's how you say that the "terminal" extra adds a dependency on
pyreadline (but only on windows):

Requires-Dist: pyreadline (>=2); sys_platform == "win32" and extra == 'terminal'

(https://gist.github.com/njsmith/ed74851c0311e858f0f7#file-ipython-4-0-0-wheel-metadata-L64)

I'm not sure this is the syntax that I would have come up with, but I
guess it's too late to put the genie back in the bottle, so this PEP
should have some way to cope with these things? Currently they are
simply syntax errors according to the PEP's grammar.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org