Mailman 3 June 2005 - Python-Dev

Generator enhancements patch available
by Phillip J. Eby June 18, 2005

June 18, 2005

I've just submitted patch 1223381 (http://python.org/sf/1223381), which implements code and test changes for: * yield expressions * bare yield (short for yield None) * yield in try/finally * generator.send(value) (send value into generator; substituted for PEP 342's next(arg)) * generator.throw(typ[,val[,tb]]) (raise error in generator) * generator.close() * GeneratorExit built-in exception type * generator.__del__ (well, the C equivalent) * All necessary mods to the compiler, parser module, and Python 'compiler' package to support these changes. It was necessary to change a small part of the eval loop (well, the initialization, not the loop) and the gc module's has_finalizer() logic in order to support a C equivalent to __del__. Specialists in these areas should probably scrutinize this patch! There is one additional implementation detail that was not contemplated in either PEP. in order to prevent used-up generators from retaining unnecessary references to their frame's contents, I set the generator's gi_frame member to None whenever the generator finishes normally or with an error. Thus, an exhausted generator cannot be part of a cycle, and it releases its frame object sooner than in previous Python versions. For generators used only in a direct "for" loop, this makes no difference, but for generators used with the iterator protocol (i.e. "gen.next()") from Python, this avoids stranding the generator's frame in a traceback cycle. Anyway, your comments/questions/feedback/bug reports are welcome.

1 0

Multiple expression eval in compound if statement?
by Skip Montanaro June 18, 2005

June 18, 2005

I'm horsing around with recognizing switch-like if statements like: if x == 1: print 1 elif x == 2: print 2 else: print "unknown" in the compiler and generating O(1) code. "x" can be any expression, but must be precisely the same in each elif clause, the comparison operator must be "==" and the RHS of the test must evaluate to a simple hashable constant (string, float, integer - still working on None, constant tuples may be allowed later). I can currently recognize such constructs and am working on code generation. This would represent a semantic change to the Python runtime, because x would only be evaluated once, whereas in existing usage it can potentially be evaluated many times. If evaluating x has side-effects, code like the above that currently works would break. In reading the language reference manual, it says: It selects exactly one of the suites by evaluating the expressions one by one until one is found to be true (see section 5.10 for the definition of true and false); then that suite is executed (and no other part of the if statement is executed or evaluated). If all expressions are false, the suite of the else clause, if present, is executed. It says nothing about possibly caching values, which is what I'm doing effectively, but it seems pretty clear that each full test expression will be evaluated until one evaluates to true. I can't imagine anyone relying on a side-effect when evaluating x in this context, and if someone did, their code would be truly fragile. Still, any thought as to whether or not such a semantic change to the language might be acceptable? Skip

6 7

Re: [Python-Dev] Implementing PEP 342 (was Re: Withdrawn PEP 288 and thoughts on PEP 342)
by Phillip J. Eby June 18, 2005

June 18, 2005

At 04:55 PM 6/18/2005 +0300, Oren Tirosh wrote: >On 6/18/05, Phillip J. Eby <pje(a)telecommunity.com> wrote: > > At 08:03 PM 6/16/2005 -0700, Guido van Rossum wrote: > > It turns out that making 'next(EXPR)' work is a bit tricky; I was going to > > use METH_COEXIST and METH_VARARGS, but then it occurred to me that > > METH_VARARGS adds overhead to normal Python calls to 'next()', so I > > implemented a separate 'send(EXPR)' method instead, and left 'next()' a > > no-argument call. > >Please use the name "feed", not "send". That would make the enhanced >generators already compatible "out of the box" with existing code >expecting the de-facto consumer interface (see >http://effbot.org/zone/consumer.htm). The other part of the consumer >interface (the close() method) is already being added in PEP 343. Hm. Do you want reset() as well? :) More seriously, I'm not sure that PEP 343's close() does something desirable for the consumer interface. Overall, this sounds like something that should be added to the PEPs though. I hadn't really thought of using inbound generator communication for parsing; it's an interesting use case. However, looking more closely at the consumer interface, it seems to me the desired semantics for feed() are different than for send(), because of the "just-started generator can't receive data" problem. Also, the consumer interface doesn't include handling for StopIteration. Maybe feed() should prime the generator if it's just started, and throw away the yield result as long as it's None? Maybe it should ignore StopIteration? Perhaps it should raise an error if the generator yields anything but None in response? These seem like questions worth discussing.

1 0

Withdrawn PEP 288 and thoughts on PEP 342
by Raymond Hettinger June 18, 2005

June 18, 2005

PEP 288 is now withdrawn. The generator exceptions portion is subsumed by PEP 343, and the generator attributes portion never garnered any support. The fate of generator attributes is interesting vís-a-vís PEP 342. The motivation was always related to supporting advanced generator uses such as emulating coroutines and writing generator based data consumer functions. At the time, Guido and everyone else found those use cases to be less than persuasive. Also, people countered that that functionality could be easily simulated with class based iterators, global variables, or passing a mutable argument to a generator. Amazingly, none of those objections seem to be directed toward 342 which somehow seems on the verge of acceptance even without use cases, clear motivation, examples, or a draft implementation. Looking back at the history of 288, generator attributes surfaced only in later drafts. In the earlier drafts, the idea for passing arguments to and from running generators used an argument to next() and a return value for yield. If this sounds familiar, it is because it is not much different from the new PEP 342 proposal. However, generator argument passing via next() was shot down early-on. The insurmountable concept flaw was an off-by-one issue. The very first call to next() does not correspond to a yield statement; instead, it corresponds to the first lines of a generator (those run *before* the first yield). All of the proposed use cases needed to have the data passed in earlier. With the death of that idea, generator attributes were born as a way of being able to pass in data before the first yield was encountered and to receive data after the yield. This was workable and satisfied the use cases. Coroutine simulations such as those in Dr Mertz's articles were easily expressed with generator attributes. As a further benefit, using attributes was a natural approach because that same technique has long been used with classes (so no new syntax was needed and the learning curve was zero). In contrast to PEP 288's low impact approach, PEP 342 changes the implementation of the for-loop, alters the semantics of "continue", introduces new and old-style iterators, and creates a new magic method. Meanwhile, it hasn't promised any advantages over the dead PEP 288 proposals. IOW, I don't follow how 342 got this far, how 342 intends to overcome the off-by-one issue, how it addresses all of the other objections leveled at the now dead PEP 288, and why no one appears concerned about introducing yet another new-style/old-style issue that will live in perpetuity. Raymond Sidenote: generator attributes also failed because generators lacked a sufficiently elegant way to refer to running instances of themselves (there is no self argument so we would need an access function or have a dynamic function attribute accessible only from within a running generator).

9 16

Propose rejection of PEPs 239 and 240 -- a builtin rational type and rational literals
by Raymond Hettinger June 17, 2005

June 17, 2005

These PEPs are four years old. Nothing is intrinsically wrong with them, but they have garnered little enthusiasm, discussion, or support, suggesting that the original need was somewhat modest. In addition, the principal (but not only) use cases for a builtin rational type and corresponding rational literal have already been addressed by the decimal module (and the expected future integration of decimal literals). From rationale section of the PEPs: """ Rational numbers are useful for exact and unsurprising arithmetic. They give the correct results people have been taught in various math classes. Making the "obvious" non-integer type one with more predictable semantics will surprise new programmers less than using floating point numbers. As quite a few posts on c.l.py and on tutor(a)python.org have shown, people often get bit by strange semantics of floating point numbers: for example, round(0.98, 2) still gives 0.97999999999999998. """ The future direction of the decimal module likely entails literals in the form of 123.45d with binary floats continuing to have the form 123.45. This conflicts with the rational literal proposal of having 123.45 interpreted as 123 + 45/100. There may still be a use for a rational module in the standard library, but builtin support is no longer needed or desirable. The PEPs also touch on division ambiguities which were largely resolved by the acceptance of PEP 238 which introduced the floor division operator and from __future__ import division. The rational proposal also has an intrinsic weakness shared with Java's prior versions of BigDecimal which they found to be unworkable in practice. The weakness was that repeated operations caused the internal number of digits to grow beyond reason. For this reason, the PEP proposes some arbitrary level of truncation which conflicts with the goals of having "obvious and exact" arithmetic. The proposed truncation methodology was undeveloped and made no proposal for the same fine level of control as its counterpart in the decimal module where issues of variable precision, multiple contexts, and alternate rounding modes have been fully thought out. Raymond

3 2

Request to rewrite PEP 206
by A.M. Kuchling June 17, 2005

June 17, 2005

Just a note, sparked by Raymond's recent work cleaning up old PEPs: I'd like to take over PEP 206, the "Batteries Included" PEP, and rewrite it to describe a "Python Advanced Library", a set of third-party packages to complement the standard library. I've written to Moshe, the original author, to check that this is OK with him, and am waiting for a response. --amk

2 1

Propose rejection of PEP 303 -- Extend divmod() for Multiple Divisors
by Raymond Hettinger June 17, 2005

June 17, 2005

This PEP has been open for two and half years without generating discussion or support. Its primary case (converting cumulative seconds into a tuple days, hours, minutes, and seconds) is a bit wanting because it doesn't generalize to months and years. That need is already met in a more robust and flexible way by date and time specific modules. The use case is also somewhat unique. Recalling 25 years of programming, almost every real case of repeated divmod() calls have been in a loop with a single constant denominator (i.e. dividing by a base in a radix conversion). The PEP does suggest other applications but they are all narrow offerings (gallon/quart/pint/ounce, miles/yards/feet, pound/shilling/pence) that are extremely rare in real apps. More importantly, the gain in succinctness is offset by a loss of clarity. Consider: dist, inches = divmod(dist, 12) yards, feet = divmod(dist, 3) versus a proposed: yards, feet, inches = divmod(dist, 3, 12) The latter form makes it difficult to visually confirm the correct number of target variables. Likewise, it is not at all obvious that the order of the 3 and 12 are correct. Users from other languages will tend to automatically understand the current form and be mystified by the second (especially given how infrequently it will arise). The PEP draws its inspiration from an APL operator. APL imbues many of its operators with scalar/scalar, scalar/array, and array/array capabilities. But in Python (not numeric) we tend to leave the operators in simple form and abstract the vectorization into operator independent primitives (i.e. map and reduce). Mathematica takes a similar approach by offering functions like NestList(). So, instead of a vectorized divmod(), it is wiser for someone to post a single accumulation recipe that would work with a variety of binary operators. Executive summary: cute, but unpersuasive and unnecessary, not worth the time to code, test, document, maintain, and explain. Raymond

4 3

Propose to reject PEP 336 -- Make None Callable
by Raymond Hettinger June 17, 2005

June 17, 2005

After nine months, no support has grown beyond the original poster. The PEP did however generate some negative responses when brought-up on comp.lang.python (it made some people's stomach churn). The PEP fails the tests of obviousness and necessity. The PEP's switch example is easily handled by replacing the callable None with a simple, null lambda: def __call__(self, input): return {1 : self.a, 2 : self.b, 3 : self.c }.get(input, lambda *args: 0)(input) The PEP does not address conflicts with other uses of None (such as default arguments or indicating values that are not applicable). Mysteriously, the PEP consumes only *args but not **kwargs. It also fails with respect to clarity and explicitness. Defining a short, explicit default function is a hands-down winner in these two important measures of merit. Raymond

4 3

Re: [Python-Dev] Propose rejection of PEP 303 -- Extend divmod() for Multiple Divisors
by Nick Coghlan June 17, 2005

June 17, 2005

Raymond Hettinger wrote: >>Plus, it fails the "not every 3-line function has to be a builtin" >>guideline: > > > Not to pick, but I hope this doesn't become a recurring refrain. That > isn't a real guideline, it's more of a snipe. It also runs counter to > Zen about proposals not being complex and being easy to explain. I guess I really mean "the use case is obscure enough that simply writing the 5-line function is better than cluttering the API of a builtin", but that doesn't have quite the same ring to it ;) > There > are tons of exceptions. Guido's new any() and all() replace only a > single line. The sorted() builtin was very successful and it only > replaced a couple of lines. The key= argument is also successful but > replaces a simple, three-line Schwarzian transform. Reading DictMixin > shows that most dictionary methods are trivially expressed in terms of a > few primitives; however, we've learned that the mapping API is > excellent, expressive, and useful (except for setdefault which I've > grown to hate ;-). IOW, the quip is food for thought but not > necessarily either a positive or negative point about a proposal. That's basically what I meant - and what I take the phrase to mean when someone else uses it. If something is simple to write, but the use case is obscure, then that's an argument *against* making it a builtin, since half the time you'll have the function written before you remember there's a builtin for it. On the other hand, if the use case is common enough, rewriting it every time you need it is just a pain. The 'not every . . .' comment just tries to say all that using only ten words. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com

1 0

Re: [Python-Dev] PEP 304 "Controlling Generation of Bytecode Files" - patch updated
by Skip Montanaro June 17, 2005

June 17, 2005

Skip> http://python.org/sf/677103 Thomas> There's no patch attached. *sigh* Thanks for noticing the problem. Apparently, since I last updated the patch, SF implemented a 250kbyte limit on file uploads. This one is big because it includes a suitably modified configure script that was generated with a slightly different version of autoconf than what's in the current Python distro so people don't need to have autoconf installed to work with the patch. I've attached a gzipped version of the patch (bcb.diffs.gz). Skip

1 0