PJ Eby <pje <at> telecommunity.com> writes:

I notice that distlib appears to support chaining of comparisons, but the PEP does not. ISTM that there is no sane use case for chaining, given the nature of comparisons that are allowed. (That is, if you perform one equality or contains comparison between two values, then you know how the next comparison would turn out.)

If chained comparisons are disallowed in the PEP, then distlib probably shouldn't be allowing them either, to prevent interop issues w/tools that only support what's in the PEP.

Sorry if I'm being dense, but can you confirm that what you mean by "chaining" is something like "a < b <= c"? Things can be tightened up once PEP 426 is finalised. At the moment, distlib allows expressions like "python_version >= '2.6'" or "python_version < '3.5'" which are also not mentioned in the PEP. I don't recall seeing any discussion around why allowing inequalities might be a bad idea, but perhaps someone can point me to it? Perhaps it should be mentioned in the PEP. Regards, Vinay Sajip

On Sun, Apr 28, 2013 at 12:47 AM, PJ Eby <pje@telecommunity.com> wrote:

On Sat, Apr 27, 2013 at 5:46 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

...

At the moment, distlib allows expressions like "python_version >= '2.6'" or "python_version < '3.5'" which are also not mentioned in the PEP. I don't recall seeing any discussion around why allowing inequalities might be a bad idea, but perhaps someone can point me to it? Perhaps it should be mentioned in the PEP.

I don't know of any reasons myself. Perhaps it was thought that restricting the syntax would make it easier to implement? For Python, that's not especially true.

I don't know for sure either - I pretty much inherited it as is from PEP 345, and otherwise haven't messed with it. I already have a note in the next draft saying that we might want access to the version specifier comparisons to make it easier to place constraints on the Python version.

The marker syntax has some other annoying quirks that make it more difficult to implement and weird to spell, anyway, like mixing actual things ('sys.platform') with non-real things (e.g. 'platform.machine' that's really 'platform.machine()'). It would make a lot more sense (and simplify implementations) to just say 'platform_machine' or even 'machine' in the first place.

OTOH, I hope that we can finalize at least the environment marker syntax pronto, so I don't end up with a dead-end version of the syntax.

I have my hands full with other aspects of the PEP 426 update, but if you and Vinay want to thrash out a modified version of the environment marker syntax (such as always using underscores for names which don't correspond directly to usable Python expressions), I'd certainly be willing to consider it. It's pretty decoupled from the rest of the PEP, which just says in a couple of places "environment marker strings can go here", and then defines the syntax for them once. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

Daniel Holth <dholth <at> gmail.com> writes:

Surely getting farther away from python by trying to prohibit useless makers just makes the implementation needlessly complex.

I'm not quite sure what you mean by "useless" markers. For example, distlib checks for e.g. "'2' == '2'" and e.g. "'os.name' == 'os.name'" as these are pointless to include in an environment marker, but that's only on the basis that (probable) errors shouldn't pass silently. Regards, Vinay Sajip

On Sat, Apr 27, 2013 at 11:57 AM, Vinay Sajip <vinay_sajip@yahoo.co.uk> wrote:

Daniel Holth <dholth <at> gmail.com> writes:

...

Surely getting farther away from python by trying to prohibit useless makers just makes the implementation needlessly complex.

I'm not quite sure what you mean by "useless" markers. For example, distlib checks for e.g. "'2' == '2'" and e.g. "'os.name' == 'os.name'" as these are pointless to include in an environment marker, but that's only on the basis that (probable) errors shouldn't pass silently.

My test suite actually contains checks like that in order to verify and/or logic, i.e., a bunch of "'x'=='x' and 'x'=='y'" type stuff, to ensure that operator precedence is handled correctly, so I wouldn't want to prohibit those in the spec. I'm definitely in favor of switching to '_'; it'd let me delete some lines from my implementation as well as making things clearer, and on top of that it restores compatibility with the PEP 390 marker syntax. (For whatever good that is, given that it's apparently being rejected soon. ;-) ) Anyway, I propose keeping the current spec but explicitly *prohibiting* chained comparisons and concatenated string constants (e.g. 'os.name=="win" "32"', which is valid Python 2 code.) I would also happily prohibit the use of '\' in strings, as that would let me get rid of an eval in my implementation. ;-) Hm. Actually i can get rid of that eval already if I use an appropriate codec to decode the string... but that brings up another important question: are environment markers Unicode on Python 2? Do we need to support any non-ASCII characters in any of the provided variables? What happens if someone uses an r'', u'', or b'' string constant? I think we need to be a bit pickier about what subset of Python we're supporting, so that we don't end up with implementation-defined behavior. Ideally, this would include dumbing strings down to ASCII without backslashes or prefix characters. If in the future a non-ASCII usecase occurs for these strings, we could loosen the spec then. (Hopefully after setuptools is out of the picture, since it's still stuck using 8-bit files with unspecified encodings. ;-) )

My test suite actually contains checks like that in order to verify

and/or logic, i.e., a bunch of "'x'=='x' and 'x'=='y'" type stuff, to ensure that operator precedence is handled correctly, so I wouldn't want to prohibit those in the spec.

That's OK by me, I'm not too hung up on having those checks.

I'm definitely in favor of switching to '_'; it'd let me delete some lines from my implementation as well as making things clearer, and on top of that it restores compatibility with the PEP 390 marker syntax. (For whatever good that is, given that it's apparently being rejected soon.  ;-) )

So IIUC, that just means that the variables we support in expressions are: * python_version, python_full_version, extra as in the PEP already. * os_name instead of os.name. * sys_platform instead of sys.platform * platform_version instead of platform.version * platform_machine instead of platform.machine * platform_python_implementation instead of platform.python_implementation How about just python_implementation for the last one?

Anyway, I propose keeping the current spec but explicitly *prohibiting* chained comparisons and concatenated string constants (e.g. 'os.name=="win" "32"', which is valid Python 2 code.)  I would also happily prohibit the use of '\' in strings, as that would let me get rid of an eval in my implementation.  ;-)

Why should chained comparisons be disallowed? If we assume that inequalities are allowed, then surely "'v1' <= some_var < 'v2'" is a reasonable short-hand for "'v1' <= some_var and some_var < 'v2'"?

Hm.  Actually i can get rid of that eval already if I use an appropriate codec to decode the string...  but that brings up another important question: are environment markers Unicode on Python 2?  Do we need to support any non-ASCII characters in any of the provided variables?  What happens if someone uses an r'', u'', or b'' string constant?

I would say that we shouldn't allow r'', u'' or b'' prefixes in the spec. Logically, the values are text, i.e. Unicode. Both stdlib json and simplejson support Unicode escapes (\uxxxx) in string literals and handle them as expected. I don't see why we need the prefixes, since we're reading this data from text files rather than source code.

I think we need to be a bit pickier about what subset of Python we're supporting, so that we don't end up with implementation-defined behavior.  Ideally, this would include dumbing strings down to ASCII without backslashes or prefix characters.  If in the future a non-ASCII usecase occurs for these strings, we could loosen the spec then. (Hopefully after setuptools is out of the picture, since it's still stuck using 8-bit files with unspecified encodings.  ;-) )

I'd like to know what practical problems there would be with supporting Unicode escapes. As I understand it, you can convert to Unicode on 2.x by just using s.decode('unicode_escape'). Regards, Vinay

I was not proposing that we support inequalities.  They aren't so

problematic in themselves, as they are inviting various questions and issues such as why they don't do any sort of version parsing.

I'm not sure that platform_version inequalities make sense to support, because the semantics across different platform versions aren't clear cut, even on a given platform. But as far as python_version is concerned, there is an understanding about compatibility and Python X.Y versions. It doesn't seem like we need to do version parsing for python_version in the same way as we do for packages on PyPI.

My preferred way to deal with the whole thing is to allow single or triple-quoted strings, disallow backslashes, and assume ASCII-only.

Why do we need triple-quoted strings in environment markers?

That lets me avoid eval *and* string decoding on 2.x.  (My implementation is actually single-source for 2.3 up through 3.2 at least, but does so by assuming all the strings it handles are the same type.)

I thought you wouldn't need eval if you could use e.g. s.decode('unicode_escape') - are there problems with doing this in older Python versions? Also, given that setuptools and distribute are merging,  I presume that means that there will need to be some more flexibility in dealing with Unicode in 2.x code. I'm also hoping that some use can be made of my distribute3 fork, which runs on 2.x and 3.x without alteration or need for 2to3, and so has to deal with both bytes and Unicode. Regards, Vinay Sajip

On Sat, Apr 27, 2013 at 1:45 PM, Daniel Holth <dholth@gmail.com> wrote:

How do you feel about parenthesis?

Without them, you have to express everything in DNF (disjunctive normal form), which usually means a lot of subexpression duplication when you want to express something like "(a or b) and (c or d)"; it would otherwise expand to: "a and c or a and d or b and c or b and d"... assuming I didn't make an error in the expansion. ;-) (Also, it's much harder to read and interpret correctly, even with such a relatively simple expression, and is absolutely not DRY.)

It's probably not that hard to prohibit chained comparisons. Do we need an abnf?

Maybe. If so, I would use a subset of the actual Python grammar, since it would help in translation for implementations that abuse Python's parser as part of the evaluation process. (Which is all of them, at the moment: distlib uses Python 2.6+'s "ast" module, while my work on setuptools is using the "parser" module, which is available in at least 2.3 through 3.2.)

What's wrong with eval?

It's slow, and prompts some people to be paranoid about whether you're thereby introducing some sort of to-be-discovered-in-future security hole. In truth, I've realized that I can do away with it entirely, anyway; the last usage I had was for string constants, but you can actually just strip off the quotes. Using eval was just a way to do that without needing to examine what kinds of quotes, decode escapes, etc.

On 28 Apr 2013 19:16, "Vinay Sajip" <vinay_sajip@yahoo.co.uk> wrote:

...

From: PJ Eby <pje@telecommunity.com>

...

So, can we keep inequalities (and therefore chained comparisons) out of the spec then? ;-)

But it seems a reasonable expectation to have "python_version > '2.5'" in

the spec, which needs no version parsing to work, and likewise it seems not unreasonable to also allow "'2.4' <= python_version < '2.6'". What's the actual difficulty in allowing this? It seems a reasonable use case.

...

It makes a single-source version of the code tougher, because strings don't have a decode() method in Python 3.

I must be missing something - why not something like

def ensure_unicode(s): if not isinstance(s, text_type): s = s.decode('unicode_escape') return s

assuming you only ever pass bytestrings or Unicode in s, and text_type is

unicode on 2.x and str on 3.x?

...

I definitely want to move to a single source strategy ASAP, but that's precisely why I'd prefer not to decode something that's already a string of version-appropriate type. ;-)

Sadly, some type checking is unavoidable if you can't control what people

pass in to your code, but it seems easy enough to deal with from a pragmatic point of view.

...

Mainly, I just want to keep the code size small, without opening too many interop problems or backward compatibility issues. If we outlaw absolutely everything in the first version of the spec (and enforce it), we don't end up with implementation-defined behavior, but can still loosen restrictions later if an actual use case appears.

I'm OK with restricting to ASCII (though, from having done numerous

single-source ports, the cross-platform handling of Unicode/bytes seems to be a solved problem), but I don't think inequalities need to be removed. (It's easy to disable in distlib's implementation, so that's not the issue - it's whether it's a useful thing to have in the spec.) Let's see what others say. I say no to allowing text based ordered comparisons in environment markers, due to the inconsistency with the version comparison rules in dependency specifications. This also rules out chained comparisons in environment markers. "Might be useful" is not the relevant criterion at this point. For metadata 2.0, we're aiming more for "essential, given everything we know about packaging based on our own experience, that of other language communities and that of Linux distributions". That said, I can see value in supporting version specifiers (both for Python and installed distributions) as part of environment markers if a suitable syntax can be found (perhaps something that looks like a function call). It is only the use of ordinary Python string comparisons that are inconsistent with version specifiers that I seriously object to. Cheers, Nick.

Regards,

Vinay

_______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig