Mailman 3 July 2011 - Python-Dev

PEP 3101 implementation vs. documentation
by Ben Wolfson July 6, 2011

July 6, 2011

Hello, I'm writing because discussion in a bug report I submitted (<http://bugs.python.org/issue12014>) has suggested that, insofar as at least part of the issue revolves around the interpretation of PEP 3101, that aspect belonged on python-dev. In particular, I was told that the PEP, not the documentation, is authoritative. Since I'm the one who thinks something is wrong, it seems appropriate for me to be the one to bring it up. Basically, the issue is that the current behavior of str.format is at variance at least with the documentation <http://docs.python.org/library/string.html#formatstrings>, is almost certainly at variance with PEP3101 in one respect, and is in my opinion at variance with PEP3101 in another respect as well, regarding what characters can be present in what the grammar given in the documentation calls an element_index, that is, the bit between the square brackets in "{0.attr[idx]}". Both discovering the current behavior and interpreting the documentation are pretty straightforward; interpreting what the PEP actually calls for is more vexed. I'll do the first two things first. TOC for the remainder: 1. What does the current implementation do? 2. What does the documentation say? 3. What does the PEP say? [this part is long, but the PEP is not clear, and I wanted to be thorough] 4. Who cares? 1. What does the current implementation do? Suppose you have this dictionary: d = {"@": 0, "!": 1, ":": 2, "^": 3, "}": 4, "{": {"}": 5}, } Then the following expressions have the following results: (a) "{0[@]}".format(d) --> '0' (b) "{0[!]}".format(d) --> ValueError: Missing ']' in format string (c) "{0[:]}".format(d) --> ValueError: Missing ']' in format string (d) "{0[^]}".format(d) --> '3' (e) "{0[}]}".format(d) --> ValueError: Missing ']' in format string (f) "{0[{]}".format(d) --> ValueError: unmatched '{' in format (g) "{0[{][}]}".format(d) --> '5' Given (e) and (f), I think (g) should be a little surprising, though you can probably guess what's going on and it's not hard to see why it happens if you look at the source: (e) and (f) fail because MarkupIterator_next (in Objects/stringlib/string_format.h) scans through the string looking for curly braces, because it treats them as semantically significant no matter what context they occur in. So, according to MarkupIterator_next, the *first* right curly brace in (e) indicates the end of the replacement field, giving "{0[}". In (f), the second left curly brace indicates (to MarkupIterator_next) the start of a *new* replacement field, and since there's only one right curly brace, it complains. In (g), MarkupIterator_next treats the second left curly brace as starting a new replacement field and the first right curly brace as closing it. However, actually, those braces don't define new replacement fields, as indicated by the fact that the whole expression treats the element_index fields as just plain old strings. (So the current implementation is somewhat schizophrenic, acting at one point as if the braces have to be balanced because '{[]}' is a replacement field and at another point treating the braces as insignificant.) The explanation for (b) and (c) is that parse_field (same source file) treats ':' and '!' as indicating the end of the field_name section of the replacement field, regardless of whether those characters occur within square brackets or not. So, that's what the current implementation does, in those cases. 2. What does the documentation say? The documentation gives a grammar for replacement fields: """ replacement_field ::= "{" [field_name] ["!" conversion] [":" format_spec] "}" field_name ::= arg_name ("." attribute_name | "[" element_index "]")* arg_name ::= [identifier | integer] attribute_name ::= identifier element_index ::= integer | index_string index_string ::= <any source character except "]"> + conversion ::= "r" | "s" format_spec ::= <described in the next section> """ Given this definition of index_string, all of (a)--(g) should be legal, and the results should be the strings '0', '1', '2', '3', "{'}': 5}", and '5'. There is no need to exclude ':', '!', '}', or '{' from the index_string field; allowing them creates no ambiguity, because the field is delimited by square brackets. Tangent: the definition of attribute_name here also does not describe the current behavior ("{0. ;}".format(x) works fine and will call getattr(x, " ;")) and the PEP does not require the attribute_name to be an identifier; in fact it explicitly states that the attribute_name doesn't need to be a valid Python identifier. attribute_name should read (to reflect something like actual behavior, anyway) "<any source character except '[', '.', ':', '!', '{', or '}'> +". The same goes for arg_name (with the integer alternation). Observe: >>> x = lambda: None >>> setattr(x, ']]', 3) >>> "{].]]}".format(**{"]":x}) # (h) '3' One can also presently do this (hence "something like actual behavior"): >>> setattr(x, 'f}', 4) >>> "{a{s.f}}".format(**{"a{s":x}) '4' But not this: >>> "{a{s.func_name}".format(**{"a{s":x}) as it raises a ValueError, for the same reason as explains (g) above. 3. What does the PEP say? Well... It's actually hard to tell! Summary: The PEP does not contain a grammar for replacement fields, and is surprisingly nonspecific about what can appear where, at least when talking about the part of the replacement field before the format specifier. The most reasonable interpretation of the parts of the PEP that seem to be relevant favors the documentation, rather than the implementation. This can be separated into two sub-questions. A. What does the PEP say about ':' and '!'? It says two things that pertain to element_index fields. The first is this: """ The rules for parsing an item key are very simple. If it starts with a digit, then it is treated as a number, otherwise it is used as a string. Because keys are not quote-delimited, it is not possible to specify arbitrary dictionary keys (e.g., the strings "10" or ":-]") from within a format string. """ So it notes that some things can't be used: - Because anything composed entirely of digits is treated as a number, you can't get a string composed entirely of digits. Clear enough. - What's the explanation for the second example, though? Well, you can't have a right square bracket in the index_string, so that would already mean that you can't do this: "{0[:-]]}".format(...) regardless of the whether colons are legal or not. So, although the PEP gives an example of a string that can't in the element_index part of a replacement field, and that string contains a colon, that string would have been disallowed for other reasons anyway. The second is this: """ The str.format() function will have a minimalist parser which only attempts to figure out when it is "done" with an identifier (by finding a '.' or a ']', or '}', etc.). """ This requires some interpretation. For one thing, the contents of an element_index aren't identifiers. For another, it's not true that you're done with an identifier (or whatever) whenever you see *any* of those; it depends on context. When parsing this: "{0[a.b]}" the parser should not stop at the '.'; it should keep going until it reaches the ']', and that will give the element_index. And when parsing this: "{0.f]oo[bar].baz}", it's done with the identifier "foo" when it reaches the '[', not when it reaches the second '.', and not when it reaches the ']', either (recall (h)). The "minimalist parser" is, I take it, one that works like this: when parsing an arg_name you're done when you reach a '[', a ':', a '!', a '.', '{', or a '}'; the same rules apply when parsing a attribute_name; when parsing an element_index you're done when you see a ']'. Now as regards the curly braces there are some other parts of the PEP that perhaps should be taken into account (see below), but I can find no justification for excluding ':' and '!' from the element_index field; the bit quoted above about having a minimalist parser isn't a good justification for that, and it's the only part of the entire PEP that comes close to addressing the question. B. What does it say about '}' and '{'? It still doesn't say much explicitly. There's the quotation I just gave, and then these passages: """ Brace characters ('curly braces') are used to indicate a replacement field within the string: [...] Braces can be escaped by doubling: """ Taken by itself, this would suggest that wherever there's an unescaped brace, there's a replacement field. That would mean that the current implementation's behavior is correct in (e) and (f) but incorrect in (g). However, I think this is a bad interpretation; unescaped braces can indicate the presence of a replacement field without that meaning that *within* a replacement field braces have that meaning, no matter where within the replacement field they occur. Later in the PEP, talking about this example: "{0:{1}}".format(a, b) We have this: """ These 'internal' replacement fields can only occur in the format specifier part of the replacement field. Internal replacement fields cannot themselves have format specifiers. This implies also that replacement fields cannot be nested to arbitrary levels. Note that the doubled '}' at the end, which would normally be escaped, is not escaped in this case. The reason is because the '{{' and '}}' syntax for escapes is only applied when used *outside* of a format field. Within a format field, the brace characters always have their normal meaning. """ The claim "within a format field, the brace characters always have their normal meaning" might be taken to mean that within a *replacement* field, brace characters always indicate the start (or end) of a replacement field. But the PEP at this point is clearly talking about the formatting section of a replacement field---the part that follows the ':', if present. ("Format field" is nowhere defined in the PEP, but it seems reasonable to take it to mean "the format specifier of a replacement field".) However, it seems most reasonable to me to take "normal meaning" to mean "just a brace character". Note that the present implementation only kinda sorta conforms to the PEP in this respect: >>> import datetime >>> format(datetime.datetime.now(), "{{%Y") '{{2011' >>> "{0:{{%{1}}".format(datetime.datetime.now(), 'Y') # (i) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: unmatched '{' in format >>> "{0:{{%{1}}}}".format(datetime.datetime.now(), 'Y') # (j) '{2011}' Here the brace characters in (i) and (j) are treated, again in MarkupIterator_next, as indicating the start of a replacement field. In (i), this leads the function to throw an exception; since they're balanced in (j), processing proceeds further, and the doubled braces aren't treated as indicating the start or end of a replacement field---because they're escaped. Given that the format spec part of a replacement field can contain further replacement fields, this is, I think, correct behavior, but it's not easy to see how it comports with the PEP, whose language is not very exact. The call to the built-in format() bypasses the mechanism that leads to these results. The PEP is very, very nonspecific about the parts of the replacement field that precede the format specifier. I don't know what kind of discussion surrounded the drawing up of the grammar that appears in the documentation, but I think that it, and not the implementation, should be followed. The implementation only works the way it does because of parsing shortcuts: it raises ValueErrors for (b) and (c) because it generalizes something true of the attribute_name field (encountering a ':' or '!' means one has moved on to the format_spec or conversion part of the replacement field) to the element_index field. And it raises an error for (e) and (f), but lets (g) through, for the reasons already mentioned. It is, in that respect, inconsistent; it treats the curly brace as having one semantic significance at one point and an entirely different significance at another point, so that it does the right thing in the case of (g) entirely by accident. There is, I think, no way to construe the PEP so that it is reasonable to do what the present implementation does in all three cases (if "{" indicates the start of a replacement field in (f), it should do so in (g) as well); I think it's actually pretty difficult to construe the PEP in any way that makes what it does in the case of (e) and (f) correct. 4. Who cares? Well, I do. (Obviously.) I even have a use case: I want to be able to put arbitrary (or as close to arbitrary as possible) strings in the element_index field, because I've got some objects that (should!) enable me to do this: p.say("I'm warning you, {e.red.underline[don't touch that!]}") and have this written ("e" for "effects") to p.out: I'm warning you, \x1b[31m\x1b[4mdon't touch that!\x1b[0m I have a way around the square bracket problem, but it would be quite burdensome to have to deal with all of !:{} as well; enough that I would fall back on something like this: "I'm warning you, {0}".format(e.red.underline("don't touch that!")) or some similar interpolation-based strategy, which I think would be a shame, because of the way it chops up the string. But I also think the present behavior is extremely counterintuitive, unnecessarily complex, and illogical (even unpythonic!). Isn't it bizarre that (g) should work, given what (e) and (f) do? Isn't it strange that (b) and (c) should fail, given that there's no real reason for them to do so---no ambiguity that has to be avoided? And something's gotta give; the documentation and the implementation do not correspond. Beyond the counterintuitiveness of the present implementation, it is also, I think, self-inconsistent. (e) and (f) fail because the interior brace is treated as starting or ending a replacement field, even though interior replacement fields aren't allowed in that part of a replacement field. (g) succeeds because the braces are balanced: they are treated at one point as if they were part of a replacement field, and at another (correctly) as if they are not. But this makes the failure of (e) and (f) unaccountable. It would not have been impossible for the PEP to allow interior replacement fields anywhere, and not just in the format spec, in which case we might have had this: (g') "{0[{][}]}".format(range(10), **{'][':4}) --> '3' or this: (g'') "{0[{][}]}".format({'4':3}, **{'][':4}) --> '3' or something with that general flavor. As far as I can tell, the only way to consistently maintain that (e) and (f) should fail requires that one take (g') or (g'') to be correct: either the interior braces signal replacement fields (hence must be balanced) or they don't (or they're escaped). Regarding the documentation, it could of course be rendered correct by changing *it*, rather than the implementation. But wouldn't it be tremendously weird to have to explain that, in the part of the replacement field preceding the conversion, you can't employ any curly braces, unless they're balanced, and you can't employ ':' or '!' at all, even though they have no semantic significance? So these are out: {0[{].foo} {0[}{}]} {0[a:b]} But these are in: {0[{}{}]} {0[{{].foo.}}} (k) ((k) does work, if you give it an object with the right structure, though it probably should not.) And, moreover, someone would then have to actually change the documentation, whereas there's a patch already, attached to the bug report linked way up at the top of this email, that makes (a)--(g) all work, leaves (i) and (j) as they are, and has the welcome side-effect of making (k) *not* work (if any code anywhere relies on (k) or things like it working, I will be very surprised: anyway the fact that (k) works is, technically, undocumented). It is also quite simple. It doesn't effect the built-in format(), because the built-in format() is concerned only with the format-specifier part of the replacement field and not all the stuff that comes before that, telling str.format *what* object to format. Thanks for reading, -- Ben Wolfson "Human kind has used its intelligence to vary the flavour of drinks, which may be sweet, aromatic, fermented or spirit-based. ... Family and social life also offer numerous other occasions to consume drinks for pleasure." [Larousse, "Drink" entry]

5 9

Re: [Python-Dev] cpython: Remove mention of medical condition from the test suite.
by Stefan Behnel July 6, 2011

July 6, 2011

Georg Brandl, 06.07.2011 07:35: > Well, it was stated that even non-joking use of such words can offend > (the example given was "your argument is blind to (these facts)"). > I consider use in jokes to be more serious, since it's careless use. > Sorry if I overreacted here. There's a common saying that a disabled person can't be considered "normal" unless he/she can also be called an asshole from time to time. Personally, I do not consider the pun in question harmful, simply because it's so clearly just a play on words that doesn't make much sense, apart from having a double meaning. In general, however, I think it's important to make jokes with and about disabled people. That keeps them inside of our society, just like anyone else. Treating them as "untouchable" means to single them out. On that note, if anyone of you ever makes it to the beautiful village of Amersfoort (NL), don't miss the "Downey's" café. Nice and friendly place to be. Stefan

3 3

Re: [Python-Dev] [Python-checkins] cpython: Remove mention of medical condition from the test suite.
by Nick Coghlan July 6, 2011

July 6, 2011

On Wed, Jul 6, 2011 at 6:17 AM, Georg Brandl <g.brandl(a)gmx.net> wrote: > For this test string, a) I'm not a native speaker and therefore don't know of > any special treatment this pun deserves It's not an especially *good* joke, just a very old one that plays on double meanings of both "see" (as in sight and understanding) and "saw" (as in sight and a tool). Still, I'd put it in the same category as the Monty Python quotes we have scattered around the test suite - if people came across them and didn't realise they were quotes they might be puzzled, but attempting to retain that Pythonesque sense of humour is itself part of what makes the Python community what it is. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0

Re: [Python-Dev] Compiling Python 3.2 on Cygwin fails
by Dan Stromberg July 6, 2011

July 6, 2011

On Tue, Jul 5, 2011 at 7:25 AM, David Robinow <drobinow(a)gmail.com> wrote: > > Cygwin is not really a supported platform. ... > [Ultimately somebody with an > interest in cygwin will need to get active in python development. I've > been meaning to do this but life gets in the way.] > I was bitten by the lack of Cygwin support in 3.2 as well. IMO, python-dev needs continuous integration on a build farm that includes representative platforms. Most of the machines in the farm could be virtualboxes. I don't think the problem is so much that the right people haven't gotten involved, as that the currently-involved people don't know when they're breaking something for someone else due to the lack of continuous integration.

3 6

Re: [Python-Dev] [Python-checkins] cpython: Remove mention of medical condition from the test suite.
by Guido van Rossum July 6, 2011

July 6, 2011

It's not a bug and shouldn't be "fixed". We leave lots of minor infractions in the code because the code churn of fixing them all would be too distracting. On Jul 3, 2011 10:22 AM, "georg.brandl" <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/76452b892838 > changeset: 71146:76452b892838 > parent: 71144:ce52310f61a0 > user: Georg Brandl <georg(a)python.org> > date: Sun Jul 03 19:22:42 2011 +0200 > summary: > Remove mention of medical condition from the test suite. > > files: > Lib/test/test_csv.py | 8 ++++---- > 1 files changed, 4 insertions(+), 4 deletions(-) > > > diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py > --- a/Lib/test/test_csv.py > +++ b/Lib/test/test_csv.py > @@ -459,20 +459,20 @@ > '5', '6']]) > > def test_quoted_quote(self): > - self.readerAssertEqual('1,2,3,"""I see,"" said the blind man","as he picked up his hammer and saw"', > + self.readerAssertEqual('1,2,3,"""I see,"" said the happy man","as he picked up his hammer and saw"', > [['1', '2', '3', > - '"I see," said the blind man', > + '"I see," said the happy man', > 'as he picked up his hammer and saw']]) > > def test_quoted_nl(self): > input = '''\ > 1,2,3,"""I see,"" > -said the blind man","as he picked up his > +said the happy man","as he picked up his > hammer and saw" > 9,8,7,6''' > self.readerAssertEqual(input, > [['1', '2', '3', > - '"I see,"\nsaid the blind man', > + '"I see,"\nsaid the happy man', > 'as he picked up his\nhammer and saw'], > ['9','8','7','6']]) > > > -- > Repository URL: http://hg.python.org/cpython

1 1

Re: [Python-Dev] cpython: Issue #12469: replace assertions by explicit if+raise
by Antoine Pitrou July 5, 2011

July 5, 2011

On Mon, 04 Jul 2011 18:06:53 +0200 victor.stinner <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/7eef821ab20d > changeset: 71197:7eef821ab20d > user: Victor Stinner <victor.stinner(a)haypocalc.com> > date: Mon Jul 04 18:06:35 2011 +0200 > summary: > Issue #12469: replace assertions by explicit if+raise Instead of generic Exception, it would be better to use AssertionError. Regards Antoine.

5 4

Re: [Python-Dev] [Python-checkins] cpython: Issue #12452: Plist and Dict are now deprecated
by Victor Stinner July 5, 2011

July 5, 2011

Le mardi 05 juillet 2011 à 07:59 -0400, Eric Smith a écrit : > On 7/4/2011 8:28 AM, victor.stinner wrote: > > http://hg.python.org/cpython/rev/4f14050a963f > > changeset: 71194:4f14050a963f > > user: Victor Stinner <victor.stinner(a)haypocalc.com> > > date: Mon Jul 04 14:28:45 2011 +0200 > > summary: > > Issue #12452: Plist and Dict are now deprecated > > > > Replace PendingDeprecationWarning warnings by DeprecationWarning. > > Shouldn't this be in MISC/News, be accompanied by documentation changes, > and have tests? Plist and Dict were never documented (in Doc/library/plistlib.rst). These classes have no test. You mean that I should add an entry to Misc/NEWS saying that these classe are now deprecated? Should I also mention the deprecation to the "What's new in Python 3.3?" document? Victor

2 1

Re: [Python-Dev] [Python-checkins] cpython: Remove mention of medical condition from the test suite.
by Jim Jewett July 5, 2011

July 5, 2011

If you're going to get rid of the pun, you might as well change the whole sentence... On Sun, Jul 3, 2011 at 1:22 PM, georg.brandl <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/76452b892838 > changeset: 71146:76452b892838 > parent: 71144:ce52310f61a0 > user: Georg Brandl <georg(a)python.org> > date: Sun Jul 03 19:22:42 2011 +0200 > summary: > Remove mention of medical condition from the test suite. > > files: > Lib/test/test_csv.py | 8 ++++---- > 1 files changed, 4 insertions(+), 4 deletions(-) > > > diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py > --- a/Lib/test/test_csv.py > +++ b/Lib/test/test_csv.py > @@ -459,20 +459,20 @@ > '5', '6']]) > > def test_quoted_quote(self): > - self.readerAssertEqual('1,2,3,"""I see,"" said the blind man","as he picked up his hammer and saw"', > + self.readerAssertEqual('1,2,3,"""I see,"" said the happy man","as he picked up his hammer and saw"', > [['1', '2', '3', > - '"I see," said the blind man', > + '"I see," said the happy man', > 'as he picked up his hammer and saw']]) > > def test_quoted_nl(self): > input = '''\ > 1,2,3,"""I see,"" > -said the blind man","as he picked up his > +said the happy man","as he picked up his > hammer and saw" > 9,8,7,6''' > self.readerAssertEqual(input, > [['1', '2', '3', > - '"I see,"\nsaid the blind man', > + '"I see,"\nsaid the happy man', > 'as he picked up his\nhammer and saw'], > ['9','8','7','6']]) > > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins(a)python.org > http://mail.python.org/mailman/listinfo/python-checkins > >

4 6

PEP 397 (Python launcher for Windows) reference implementation
by Vinay Sajip July 5, 2011

July 5, 2011

PEP 397 (Python launcher for Windows) has a reference implementation in Python. Does anyone know of a C implementation, or is planning/working on one? I realise this is the final objective, so such implementation might be premature, but perhaps someone has been experimenting ... Regards, Vinay Sajip

10 31

Re: [Python-Dev] cpython (3.2): Issue #12497: Install test/data to prevent failures of the various codecmaps
by Antoine Pitrou July 5, 2011

July 5, 2011

On Tue, 05 Jul 2011 04:11:45 +0200 ned.deily <python-checkins(a)python.org> wrote: > LIBSUBDIRS= tkinter tkinter/test tkinter/test/test_tkinter \ > tkinter/test/test_ttk site-packages test \ > - test/capath \ > + test/capath test/data \ > test/cjkencodings test/decimaltestdata test/xmltestdata test/subprocessdata \ > test/tracedmodules test/encoded_modules \ > concurrent concurrent/futures encodings \ Shouldn't we have something less dumb than a hardcoded list of directories? :) Regards Antoine.

2 1