[Distutils] The future of env markers (was Re: platform_python_implementation not implemented.)

[James Polley]

On Sun, Sep 13, 2015 at 4:27 AM, PJ Eby <pje at telecommunity.com> wrote:

> On Thu, Sep 10, 2015 at 11:34 AM, Daniel Holth <dholth at gmail.com> wrote:
> > That is too bad. markerlib was added to pkg_resources as _markerlib in
> 2012.
> > It is used for .dist-info metadata as present in wheel. Then, only to
> > implement markers to setup.py or .egg-info style metadata, pkg_resources
> > gains its inline markers implementation in 2013, including its own
> > definitions of the key/values used in environment marker comparisons.
> Dots
> > switched to underscores in revision aa74cf234684 in the inline
> > implementation and in revision 1fc8a2d94f9f for setuptools' vendored
> > _markerlib.
>
> To clarify: the Distribute fork of setuptools added _markerlib, but
> Distribute didn't support older version of Python as the official
> setuptools 0.6 did, which is why I added the inline version there.
>

Several years later, setuptools no longer supports <2.6 (as I understand
it, please correct me if I've missed something), and Jython now supports
everything we need to be able to drop the inline version.


> There was also active discussion at the time about changing the
> markers spec, as Nick was working on PEP 426.  Previously, there were
> two other PEPs, 345 and 390, with their own not-quite compatible
> specs, and more recently there is now a PEP 496.
>

I knew about 345 but 390 is new to me.

*reads* Oh I see -
https://www.python.org/dev/peps/pep-0390/#context-dependant-sections does
describe something very similar.

Thanks for that!


>
> So, there has never really been a stable standard for environment
> markers; all of the previous specs have had various ambiguities,
> underspecification, or excessive lenience.  My hope at the time of the
> PEP 426 discussion was that we could define a *strict* grammar in
> place of a loose pseudo-grammar so that the spec would be robust to
> multiple implementations rather than forcing everyone to copy the
> quirks of one particular implementation (e.g. markerlib).
>
> Unfortunately, even PEP 496 is still a little underspecified: it
> doesn't even say what kinds of string literals are allowed, address
> encodings or character sets, etc.  Is r"foo" a legal string
> expression?


I'm not sure what you mean by this. But I think my confusion is based on a
point that I'll get to in a second..

   Can you use double-quotes?  Backslash escaping?
> Adjacent string literal concatentation?  But it's better than the
> previous versions, since the mini-language is no longer an
> ambiguously-defined subset of Python and thus can no longer be parsed
> using Python's built-in grammar, and probably not its lexer either.
>

Actually, the draft implementation I put at [1] uses the ast module to
parse the markers.

At the time I wrote that I was under a misapprehension about the status of
markerlib, so I hadn't read it (not even the version vendored into
setuptools). After I wrote that draft I ended up looking at
setuptools/_markerlib/markers.py[2] and realised that I'd written something
very similar[3]

Hence my confusion when reading "is r'foo' a legal string?" and so on - I
hadn't even considered details like that because, although it's not longer
described that way in the PEP, I'd still used the python parser for the
implementation.

I'm not sure what to do here; going back to defining it as a "subset of
Python" seems like a backward step, but I'm struggling to see what benefit
we'd get from the effort of more rigorously defining the grammar. Perhaps
that's my inexperience showing though - I'm happy to take guidance from
people who have been thinking about this for more than the 3 months or so
since I started looking at it.

[1]
https://bitbucket.org/pypa/setuptools/pull-requests/141/implementation-of-pep-0496-environment
[2]
https://bitbucket.org/pypa/setuptools/src/121794c652a32aed899515b16227f13c2cd3d60a/_markerlib/markers.py?at=default&fileviewer=file-view-default
[3] My current plan is to drop my implementation in setuptools and instead
work on https://bitbucket.org/dholth/markerlib; once PEP-496 is accepted
and markerlib implements it, we can rework setuptools (and pip, and
anything else that needs to read markers) with the new version.


>
> > I would prefer to see _markerlib._VARS used everywhere, the inline
> > pkg_resources implementation deleted, markerlib improved if necessary,
> and
> > no backwards compatibility with unspecified environment marker
> variables. My
> > hunch is that no one needs the backwards compatibility.
>
> It may have changed since when I added markers to the official
> setuptools, but I intentionally did not document the marker feature;
> it was experimental and added mainly to support setuptools' internal
> use case of including SSL backports on older Python versions to
> support easy_install checking SSL certs.  So, anybody using it was
> (and perhaps is; I haven't looked lately) using an undocumented
> experimental feature rather than an officially supported one.
>

I had a small novel written here, but it was pointless. In summary, that
you're correct to say that everyone using them markres an experimental
feature (from PEP-426) rather than the officially documented one (in
PEP-345)

>
> In any case, if compatibility is broken, it should probably be done by
> switching to the much-stricter PEP 496, rather than one of its
> even-more-ambiguous predecessors.
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20150916/db83081b/attachment.html>