Mailman 3 October 2012 - Python-ideas

Cofunctions PEP - Revision 4
by Greg Ewing Oct. 13, 2012

Oct. 13, 2012

Here's an updated version of the PEP reflecting my recent suggestions on how to eliminate 'codef'. PEP: XXX Title: Cofunctions Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg.ewing(a)canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for defining and calling a special type of generator called a 'cofunction'. It is designed to provide a streamlined way of writing generator-based coroutines, and allow the early detection of certain kinds of error that are easily made when writing such code, which otherwise tend to cause hard-to-diagnose symptoms. This proposal builds on the 'yield from' mechanism described in PEP 380, and describes some of the semantics of cofunctions in terms of it. However, it would be possible to define and implement cofunctions independently of PEP 380 if so desired. Specification ============= Cofunction definitions ---------------------- A cofunction is a special kind of generator, distinguished by the presence of the keyword ``cocall`` (defined below) at least once in its body. It may also contain ``yield`` and/or ``yield from`` expressions, which behave as they do in other generators. From the outside, the distinguishing feature of a cofunction is that it cannot be called the same way as an ordinary function. An exception is raised if an ordinary call to a cofunction is attempted. Cocalls ------- Calls from one cofunction to another are made by marking the call with a new keyword ``cocall``. The expression :: cocall f(*args, **kwds) is evaluated by first checking whether the object ``f`` implements a ``__cocall__`` method. If it does, the cocall expression is equivalent to :: yield from f.__cocall__(*args, **kwds) except that the object returned by __cocall__ is expected to be an iterator, so the step of calling iter() on it is skipped. If ``f`` does not have a ``__cocall__`` method, or the ``__cocall__`` method returns ``NotImplemented``, then the cocall expression is treated as an ordinary call, and the ``__call__`` method of ``f`` is invoked. Objects which implement __cocall__ are expected to return an object obeying the iterator protocol. Cofunctions respond to __cocall__ the same way as ordinary generator functions respond to __call__, i.e. by returning a generator-iterator. Certain objects that wrap other callable objects, notably bound methods, will be given __cocall__ implementations that delegate to the underlying object. Grammar ------- The full syntax of a cocall expression is described by the following grammar lines: :: atom: cocall | <existing alternatives for atom> cocall: 'cocall' atom cotrailer* '(' [arglist] ')' cotrailer: '[' subscriptlist ']' | '.' NAME Note that this syntax allows cocalls to methods and elements of sequences or mappings to be expressed naturally. For example, the following are valid: :: y = cocall self.foo(x) y = cocall funcdict[key](x) y = cocall a.b.c[i].d(x) Also note that the final calling parentheses are mandatory, so that for example the following is invalid syntax: :: y = cocall f # INVALID New builtins, attributes and C API functions -------------------------------------------- To facilitate interfacing cofunctions with non-coroutine code, there will be a built-in function ``costart`` whose definition is equivalent to :: def costart(obj, *args, **kwds): try: m = obj.__cocall__ except AttributeError: result = NotImplemented else: result = m(*args, **kwds) if result is NotImplemented: raise TypeError("Object does not support cocall") return result There will also be a corresponding C API function :: PyObject *PyObject_CoCall(PyObject *obj, PyObject *args, PyObject *kwds) It is left unspecified for now whether a cofunction is a distinct type of object or, like a generator function, is simply a specially-marked function instance. If the latter, a read-only boolean attribute ``__iscofunction__`` should be provided to allow testing whether a given function object is a cofunction. Motivation and Rationale ======================== The ``yield from`` syntax is reasonably self-explanatory when used for the purpose of delegating part of the work of a generator to another function. It can also be used to good effect in the implementation of generator-based coroutines, but it reads somewhat awkwardly when used for that purpose, and tends to obscure the true intent of the code. Furthermore, using generators as coroutines is somewhat error-prone. If one forgets to use ``yield from`` when it should have been used, or uses it when it shouldn't have, the symptoms that result can be extremely obscure and confusing. Finally, sometimes there is a need for a function to be a coroutine even though it does not yield anything, and in these cases it is necessary to resort to kludges such as ``if 0: yield`` to force it to be a generator. The ``cocall`` construct address the first issue by making the syntax directly reflect the intent, that is, that the function being called forms part of a coroutine. The second issue is addressed by making it impossible to mix coroutine and non-coroutine code in ways that don't make sense. If the rules are violated, an exception is raised that points out exactly what and where the problem is. Lastly, the need for dummy yields is eliminated by making it possible for a cofunction to call both cofunctions and ordinary functions with the same syntax, so that an ordinary function can be used in place of a cofunction that yields zero times. Record of Discussion ==================== An earlier version of this proposal required a special keyword ``codef`` to be used in place of ``def`` when defining a cofunction, and disallowed calling an ordinary function using ``cocall``. However, it became evident that these features were not necessary, and the ``codef`` keyword was dropped in the interests of minimising the number of new keywords required. The use of a decorator instead of ``codef`` was also suggested, but the current proposal makes this unnecessary as well. It has been questioned whether some combination of decorators and functions could be used instead of a dedicated ``cocall`` syntax. While this might be possible, to achieve equivalent error-detecting power it would be necessary to write cofunction calls as something like :: yield from cocall(f)(args) making them even more verbose and inelegant than an unadorned ``yield from``. It is also not clear whether it is possible to achieve all of the benefits of the cocall syntax using this kind of approach. Prototype Implementation ======================== An implementation of an earlier version of this proposal in the form of patches to Python 3.1.2 can be found here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cofunctions.h… If this version of the proposal is received favourably, the implementation will be updated to match. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

11 29

Re: [Python-ideas] Floating point contexts in Python core
by Steven D'Aprano Oct. 12, 2012

Oct. 12, 2012

On 12/10/12 11:04, Mark Adam wrote: > On Thu, Oct 11, 2012 at 6:35 AM, Steven D'Aprano<steve(a)pearwood.info> wrote: >> On 11/10/12 16:45, Greg Ewing wrote: >>> Are you sure there would be any point in this? People who >>> specifically *want* base-2 floats are probably quite happy >>> with the current float type, and wouldn't appreciate having >>> it slowed down, even by a small amount. >> >> I would gladly give up a small amount of speed for better control >> over floats, such as whether 1/0.0 raised an exception or >> returned infinity. > > Umm, you would be giving up a *lot* of speed. Native floating point > happens right in the processor, so if you want special behavior, you'd > have to take the floating point out of hardware and into "user space". Any half-decent processor supports the IEEE-754 standard. If it doesn't, it's broken by design. Even in user-space, you're not giving up that much speed in practical terms, at least not for my needs. The new decimal module in Python 3.3 is less than a factor of 10 times slower than Python's floats, which makes it pretty much instantaneous to my mind :) numpy supports configurable numeric contexts, and I don't hear that many complaints that numpy is slower than standard Python. -- Steven > > mark >

3 2

checking for identity before comparing built-in objects
by Max Moroz Oct. 12, 2012

Oct. 12, 2012

It seems that built-in classes do not short-circuit `__eq__` method when the objects are identical, at least in CPython: f = frozenset(range(200000000)) f1 = f f1 == f # this operation will take about 1 sec on my machine Is there any disadvantage to checking whether the equality was called with the same object, and if it was, return `True` right away? I noticed this when trying to memoize a function that has large frozenset arguments. While hashing of a large argument is very fast after it's done once (hash value is presumably cached), the equality comparison is always slow even against itself. So when the same large argument is provided over and over, memoization is slow. Of course, there's a workaround: subclass frozenset, and redefine __eq__ to check id() first. And arguably, for this particular use case, I should redefine both __hash__ and __eq__, to make them only look exclusively at id(), since it's not worth wasting memoizer time trying to compare two non-identical large arguments that are highly unlikely to compare equal anyway. So if there's any reason for the current implementation, I don't have a strong argument against it.

25 85

Re: [Python-ideas] Is there a good reason to use * for multiplication?
by Ram Rachum Oct. 12, 2012

Oct. 12, 2012

On Fri, Oct 12, 2012 at 10:34 PM, Mike Graham <mikegraham(a)gmail.com> wrote: > On Fri, Oct 12, 2012 at 4:27 PM, Ram Rachum <ram.rachum(a)gmail.com> wrote: > > Hi everybody, > > > > Today a funny thought occurred to me. Ever since I've learned to program > > when I was a child, I've taken for granted that when programming, the > sign > > used for multiplication is *. But now that I think about it, why? Now > that > > we have Unicode, why not use · ? > > > > Do you think that we can make Python support · in addition to *? > > > > I can think of a couple of problems, but none of them seem like > > deal-breakers: > > > > - Backward compatibility: Python already uses *, but I don't see a > backward > > compatibility problem with supporting · additionally. Let people use > > whichever they want, like spaces and tabs. > > - Input methods: I personally use an IDE that could be easily set to > > automatically convert * to · where appropriate and to allow manual input > of > > ·. People on Linux can type Alt-. . Anyone else can set up a script > that'll > > let them type · using whichever keyboard combination they want. I admit > this > > is pretty annoying, but since you can always use * if you want to, I > figure > > that anyone who cares enough about using · instead of * (I bet that > people > > in scientific computing would like that) would be willing to take the > time > > to set it up. > > > > > > What do you think? > > > > > > Ram > > Python should not expect characters that are hard for most people to > type. No one will be forced to type it. If you can't type it, use *. > Python should not expect characters that are still hard to > display on many common platforms. > We allow people to have unicode variable names, if they wish, don't we? So why not allow them to use unicode operator, if they wish, as a completely optional thing? > > I think you'll find strong opposition to adding any non-ASCII > characters or characters that don't occur on almost all keyboards as > part of the language. > > Mike >

2 1

Re: [Python-ideas] asyncore: included batteries don't fit
by Ben Darnell Oct. 12, 2012

Oct. 12, 2012

Hi python-ideas, I'm jumping in to this thread on behalf of Tornado. I think there are actually two separate issues here and it's important to keep them distinct: at a low level, there is a need for a standardized event loop, while at a higher level there is a question of what asynchronous code should look like. This thread so far has been more about the latter, but the need for standardization is more acute for the core event loop. I've written a bridge between Tornado and Twisted so libraries written for both event loops can coexist, but obviously that wouldn't scale if there were a proliferation of event loop implementations out there. I'd be in favor of a simple event loop interface in the standard library, with reference implementation(s) (select, epoll, kqueue, iocp) and some means of configuring the global (or thread-local) singleton. My preference is to keep the interface fairly low-level and close to the underlying mechanisms (i.e. like IReactorFDSet instead of IReactor{TCP,UDP,SSL,etc}), so that different interfaces like Tornado's IOStream or Twisted's protocols can be built on top of it. As for the higher-level question of what asynchronous code should look like, there's a lot more room for spirited debate, and I don't think there's enough consensus to declare a One True Way. Personally, I'm -1 on greenlets as a general solution (what if you have to call MySQLdb or getaddrinfo?), although they can be useful in particular cases to convert well-behaved synchronous code into async (as in Motor: http://emptysquare.net/blog/introducing-motor-an-asynchronous-mongodb-drive…). I like Futures, though, and I find that they work well in asynchronous code. The use of the result() method to encapsulate both successful responses and exceptions is especially nice with generator coroutines. FWIW, here's the interface I'm moving towards for async code. From the caller's perspective, asynchronous functions return a Future (the future has to be constructed by hand since there is no Executor involved), and also take an optional callback argument (mainly for consistency with currently-prevailing patterns for async code; if the callback is given it is simply added to the Future with add_done_callback). In Tornado the Future is created by a decorator and hidden from the asynchronous function (it just sees the callback), although this relies on some Tornado-specific magic for exception handling. In a coroutine, the decorator recognizes Futures and resumes execution when the future is done. With these decorators asynchronous code looks almost like synchronous code, except for the "yield" keyword before each asynchronous call. -Ben

12 36

Floating point contexts in Python core
by Oscar Benjamin Oct. 12, 2012

Oct. 12, 2012

On 9 October 2012 02:07, Guido van Rossum <guido(a)python.org> wrote: > On Mon, Oct 8, 2012 at 5:32 PM, Oscar Benjamin > <oscar.j.benjamin(a)gmail.com> wrote: >> On 9 October 2012 01:11, Guido van Rossum <guido(a)python.org> wrote: >>> On Mon, Oct 8, 2012 at 5:02 PM, Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: >>>> >>>> So the question that really needs to be answered, I think, is >>>> not "Why is NaN == NaN false?", but "Why doesn't NaN == anything >>>> raise an exception, when it would make so much more sense to >>>> do so?" >>> >>> Because == raising an exception is really unpleasant. We had this in >>> Python 2 for unicode/str comparisons and it was very awkward. >>> >>> Nobody arguing against the status quo seems to care at all about >>> numerical algorithms though. I propose that you go find some numerical >>> mathematicians and ask them. >> >> The main purpose of quiet NaNs is to propagate through computation >> ruining everything they touch. In a programming language like C that >> lacks exceptions this is important as it allows you to avoid checking >> all the time for invalid values, whilst still being able to know if >> the end result of your computation was ever affected by an invalid >> numerical operation. The reasons for NaNs to compare unequal are no >> doubt related to this purpose. >> >> It is of course arguable whether the same reasoning applies to a >> language like Python that has a very good system of exceptions but I >> agree with Guido that raising an exception on == would be unfortunate. >> How many people would forget that they needed to catch those >> exceptions? How awkward could your code be if you did remember to >> catch all those exceptions? In an exception handling language it's >> important to know that there are some operations that you can trust. > > If we want to do *anything* I think we should first introduce a > floating point context similar to the Decimal context. Then we can > talk. The other thread has gone on for ages now and isn't going anywhere. Guido's suggestion here is much more interesting (to me) so I want to start a new thread on this subject. Python's default handling of floating point operations is IEEE-754 compliant which in my opinion is the obvious and right thing to do. However, Python is a much more versatile language than some of the other languages for which IEEE-754 was designed. Python offers the possibility of a very rich approach to the control and verification of the accuracy of numeric operations on both a function by function and code block by code block basis. This kind of functionality is already implemented in the decimal module [1] as well as numpy [2], gmpy [3], sympy [4] and no doubt other numerical modules that I'm not aware of. It would be a real blessing to numerical Python programmers if either/both of the following were to occur: 1) Support for calculation contexts with floats 2) A generic kind of calculation context manager that was recognised widely by the builtin/stdlib types and also by third party numerical packages. Oscar References: [1] http://docs.python.org/library/decimal.html#context-objects [2] http://docs.scipy.org/doc/numpy/reference/generated/numpy.seterr.html#numpy… [3] https://gmpy2.readthedocs.org/en/latest/mpfr.html [4] http://docs.sympy.org/dev/modules/mpmath/contexts.html

11 26

Make undefined escape sequences have SyntaxWarnings
by Mike Graham Oct. 11, 2012

Oct. 11, 2012

The literal"\c" should be an error but in practice means "\\c". It's probably too late to make this invalid syntax as it out to be, but I wonder if a warning isn't in order, especially with the theoretical potential of adding new string escapes in the future.

7 15

Propagating StopIteration value
by Serhiy Storchaka Oct. 10, 2012

Oct. 10, 2012

As StopIteration now have value, this value is lost when using functions which works with iterators/generators (map, filter, itertools). Therefore, wrapping the iterator, which preserved its semantics in versions before 3.3, no longer preserves it: map(lambda x: x, iterator) filter(lambda x: True, iterator) itertools.accumulate(iterator, lambda x, y: y) itertools.chain(iterator) itertools.compress(iterator, itertools.cycle([True])) itertools.dropwhile(lambda x: False, iterator) itertools.filterfalse(lambda x: False, iterator) next(itertools.groupby(iterator, lambda x: None))[1] itertools.takewhile(lambda x: True, iterator) itertools.tee(iterator, 1)[0] Perhaps it would be worth to propagate original exception (or at least it's value) in functions for which it makes sense.

9 36

Make "is" checks on non-singleton literals errors
by Mike Graham Oct. 10, 2012

Oct. 10, 2012

I regularly see learners using "is" to check for string equality and sometimes other equality. Due to optimizations, they often come away thinking it worked for them. There are no cases where if x is "foo": or if x is 4: is actually the code someone intended to write. Although this has no benefit to anyone but new learners, it also doesn't really do any harm. Mike

14 28

History stepping in interactive session?
by Andy Buckley Oct. 9, 2012

Oct. 9, 2012

A couple of weeks ago I posted a question on superuser.com about whether there is a way to get the same *very* convenient stepping-through-command-history behaviour in an interactive Python interpreter session as is possible in (at least) the bash shell with the Ctrl-o keybinding: http://superuser.com/questions/477997/key-binding-to-interactively-execute-… I was spurred to ask this question by a painful development experience full of Up Up Up Up Up Enter Up Up Up Up Up Enter ... keypresses to repeat a previous set of Python commands/statements that weren't worth putting in a script file, or which I wanted to make very minor changes to on each iteration. As you might have noticed, I didn't get any answers, which either means that I'm the only person in the world to think this is an issue worth getting bothered about, or that there is no such behaviour available. Perhaps both -- but my feeling is that if this behaviour were available and well-known, it would become heavily used and very popular. As many other readline behaviours *do* work, this one would be really nice to have -- any chance that it could be added to a future release? (if it's not already there via some secret binding) Thanks! Andy

9 17