Python-ideas August 2010

python-ideas@python.org

36 participants
15 discussions

i18n and Python tracebacks
by Andre Roberge 28 Oct '12

28 Oct '12

I think it would be a good idea if Python tracebacks could be translated into languages other than English - and it would set a good example. For example, using French as my default local language, instead of >>> 1/0 Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: integer division or modulo by zero I might get something like >>> 1/0 Suivi d'erreur (appel le plus récent en dernier) : Fichier "<stdin>", à la ligne 1, dans <module> ZeroDivisionError: division entière ou modulo par zéro André

9 15

Cofunctions PEP - Revision 4
by Greg Ewing 13 Oct '12

13 Oct '12

Here's an updated version of the PEP reflecting my recent suggestions on how to eliminate 'codef'. PEP: XXX Title: Cofunctions Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg.ewing(a)canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for defining and calling a special type of generator called a 'cofunction'. It is designed to provide a streamlined way of writing generator-based coroutines, and allow the early detection of certain kinds of error that are easily made when writing such code, which otherwise tend to cause hard-to-diagnose symptoms. This proposal builds on the 'yield from' mechanism described in PEP 380, and describes some of the semantics of cofunctions in terms of it. However, it would be possible to define and implement cofunctions independently of PEP 380 if so desired. Specification ============= Cofunction definitions ---------------------- A cofunction is a special kind of generator, distinguished by the presence of the keyword ``cocall`` (defined below) at least once in its body. It may also contain ``yield`` and/or ``yield from`` expressions, which behave as they do in other generators. From the outside, the distinguishing feature of a cofunction is that it cannot be called the same way as an ordinary function. An exception is raised if an ordinary call to a cofunction is attempted. Cocalls ------- Calls from one cofunction to another are made by marking the call with a new keyword ``cocall``. The expression :: cocall f(*args, **kwds) is evaluated by first checking whether the object ``f`` implements a ``__cocall__`` method. If it does, the cocall expression is equivalent to :: yield from f.__cocall__(*args, **kwds) except that the object returned by __cocall__ is expected to be an iterator, so the step of calling iter() on it is skipped. If ``f`` does not have a ``__cocall__`` method, or the ``__cocall__`` method returns ``NotImplemented``, then the cocall expression is treated as an ordinary call, and the ``__call__`` method of ``f`` is invoked. Objects which implement __cocall__ are expected to return an object obeying the iterator protocol. Cofunctions respond to __cocall__ the same way as ordinary generator functions respond to __call__, i.e. by returning a generator-iterator. Certain objects that wrap other callable objects, notably bound methods, will be given __cocall__ implementations that delegate to the underlying object. Grammar ------- The full syntax of a cocall expression is described by the following grammar lines: :: atom: cocall | <existing alternatives for atom> cocall: 'cocall' atom cotrailer* '(' [arglist] ')' cotrailer: '[' subscriptlist ']' | '.' NAME Note that this syntax allows cocalls to methods and elements of sequences or mappings to be expressed naturally. For example, the following are valid: :: y = cocall self.foo(x) y = cocall funcdict[key](x) y = cocall a.b.c[i].d(x) Also note that the final calling parentheses are mandatory, so that for example the following is invalid syntax: :: y = cocall f # INVALID New builtins, attributes and C API functions -------------------------------------------- To facilitate interfacing cofunctions with non-coroutine code, there will be a built-in function ``costart`` whose definition is equivalent to :: def costart(obj, *args, **kwds): try: m = obj.__cocall__ except AttributeError: result = NotImplemented else: result = m(*args, **kwds) if result is NotImplemented: raise TypeError("Object does not support cocall") return result There will also be a corresponding C API function :: PyObject *PyObject_CoCall(PyObject *obj, PyObject *args, PyObject *kwds) It is left unspecified for now whether a cofunction is a distinct type of object or, like a generator function, is simply a specially-marked function instance. If the latter, a read-only boolean attribute ``__iscofunction__`` should be provided to allow testing whether a given function object is a cofunction. Motivation and Rationale ======================== The ``yield from`` syntax is reasonably self-explanatory when used for the purpose of delegating part of the work of a generator to another function. It can also be used to good effect in the implementation of generator-based coroutines, but it reads somewhat awkwardly when used for that purpose, and tends to obscure the true intent of the code. Furthermore, using generators as coroutines is somewhat error-prone. If one forgets to use ``yield from`` when it should have been used, or uses it when it shouldn't have, the symptoms that result can be extremely obscure and confusing. Finally, sometimes there is a need for a function to be a coroutine even though it does not yield anything, and in these cases it is necessary to resort to kludges such as ``if 0: yield`` to force it to be a generator. The ``cocall`` construct address the first issue by making the syntax directly reflect the intent, that is, that the function being called forms part of a coroutine. The second issue is addressed by making it impossible to mix coroutine and non-coroutine code in ways that don't make sense. If the rules are violated, an exception is raised that points out exactly what and where the problem is. Lastly, the need for dummy yields is eliminated by making it possible for a cofunction to call both cofunctions and ordinary functions with the same syntax, so that an ordinary function can be used in place of a cofunction that yields zero times. Record of Discussion ==================== An earlier version of this proposal required a special keyword ``codef`` to be used in place of ``def`` when defining a cofunction, and disallowed calling an ordinary function using ``cocall``. However, it became evident that these features were not necessary, and the ``codef`` keyword was dropped in the interests of minimising the number of new keywords required. The use of a decorator instead of ``codef`` was also suggested, but the current proposal makes this unnecessary as well. It has been questioned whether some combination of decorators and functions could be used instead of a dedicated ``cocall`` syntax. While this might be possible, to achieve equivalent error-detecting power it would be necessary to write cofunction calls as something like :: yield from cocall(f)(args) making them even more verbose and inelegant than an unadorned ``yield from``. It is also not clear whether it is possible to achieve all of the benefits of the cocall syntax using this kind of approach. Prototype Implementation ======================== An implementation of an earlier version of this proposal in the form of patches to Python 3.1.2 can be found here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cofunctions.h… If this version of the proposal is received favourably, the implementation will be updated to match. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

11 29

Cofunctions: It's alive! Its alive!
by Greg Ewing 17 Sep '10

17 Sep '10

I've been doing some more hacking, and I now have a working implementation of cofunctions, which I'll upload somewhere soon. I have also translated my yield-from examples to use cofunctions. In the course of doing this, the additional restrictions that cofunctions impose have already proved their worth -- I forgot a cocall, and it clearly told me so and pointed out exactly where it had to go! -- Greg

11 24

Keyword to disambiguate python version
by Jonny 27 Aug '10

27 Aug '10

I would propose that an idiomatic way is created, for instance a pragma or statement, which allows one to disambiguate the used python version in a manner that is both obvious for the human reader, and as well allows python to reject the script, should the wrong version of the interpreter be present. I think this is a quite common problem[1][2], which could be solved in a fairly easy and pragmatic way. Being a python beginner, I'm not well-adversed in the ways of idiomatic python, so feel free to reject or improve on my syntactic examples, these are merely meant to demonstrate my point. Building on [3], I would suggest some syntax like this: #!/usr/local/bin/python # version: python-3 import os, sys ... Alternatively, some syntax could be used that allows one to use basic comparison operators, a range or even simple chained logical statements. #!/usr/local/bin/python # version: [2.4 .. 3.0] and not 2.6.1 # or multiple clauses # version: >= 2.4.2 # version: < 3.0 # version: not 2.6.1 # jpython-version: ... # or multiple keys # min-version: 2.4.2 # min-jpython-version: 2.4.4 # max-version: 2.6.1 import os, sys ... This way it should be fairly obvious to the casual reader which python version the program is intended to run under, and the python interpreter can simply reject to parse/execute the program, should it encounter an incompatible requirement. All comments, thoughts and suggestions welcome. References [1] http://stackoverflow.com/questions/446052/python-best-way-to-check-for-pyth… [2] http://stackoverflow.com/questions/1093322/how-do-i-check-what-version-of-p… [3] http://www.python.org/dev/peps/pep-0263/

6 9

Mutable default function parameter warning
by Sérgio Surkamp 23 Aug '10

23 Aug '10

Hello list, The documentation states that the default value of function parameter, if mutable, can change it's default value at runtime due to be evaluated only once on function object creation. I would like to suggest the inclusion of an default language warning when this kind of construction is used, as it's Python specific behavior and can lead to "strange behavior" or misuse by programmers that are migrating from other languages to Python. This proposal was first open as a suggestion issue in bug track, but, as a request from Mr. Peterson, I'm rewriting it to this list. http://bugs.python.org/issue9646 Regards, -- .:''''':. .:' ` Sérgio Surkamp | Gerente de Rede :: ........ sergio(a)gruposinternet.com.br `:. .:' `:, ,.:' *Grupos Internet S.A.* `: :' R. Lauro Linhares, 2123 Torre B - Sala 201 : : Trindade - Florianópolis - SC :.' :: +55 48 3234-4109 : ' http://www.gruposinternet.com.br

6 5

@unittest.skip_others decorator
by Giampaolo Rodolà 17 Aug '10

17 Aug '10

Hello, when working on a test suite, no matter if writing new test cases from scratch or modifying an existing one, the need of temporarily excluding other tests and focus on that particular one is very common (at least for me). This is especially true when working on a test suite which contains a lot of tests producing a very verbose output result or which takes a lot to complete. What I usually do in such cases is to scroll down until test_main() function, manually change support.run_unittest() ilke this: - support.run_unittest(TestCase1, TestCase2, TestCase3, TestCase4) + support.run_unittest(TestCase4) ...and then comment out all the test cases in TestCase4 class except the one I'm interested in. This is obviously not very flexible, then I thought that maybe unittest (or test/support module, I'm not sure) could provide a specific decorator for temporarily running only one specific test class(es) or method(s). The decorator could be used in 3 ways: == use case #1 == @unittest.exclude_others class TestCase1: ... All TestCase1 test cases are run, TestCase2, TestCase3 and TestCase4 classes are excluded. == use case #2 == class TestCase1: @unittest.exclude_others def test_something(self): .... All TestCase1.* tests are excluded except "test_something". TestCase2, TestCase3 and TestCase4 are run. == use case #3 == @unittest.exclude_others class TestCase1: @unittest.exclude_others def test_something(self): .... Only TestCase1.test_something() is run (this is the most common use case). Thoughts? Kindest regards, --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/

5 4

Curly braces expansion in shell-like matcher modules
by Mathieu Bridon 17 Aug '10

17 Aug '10

Hi, A few days ago I submitted a feature request for the fnmatch module to allow curly braces shell-like expansions: http://bugs.python.org/issue9584 The patch I submitted was found to be incorrect, and I'm working on it (I already have a patch that seems to be working and I need to test it extensively). However, some concerns were raised about the fnmatch module not being the correct place to do that, if the goal is to mimic shell behavior. Expansion would have to be done before, generating the list of patterns that would then be given to fnmatch. It was suggested that I took this issue on Python-ideas, so here we go. What do people think? Should it be done in fnmatch, expanding the curly braces to the regex? Should it be done before that? If the latter, what would be an appropriate module to expand the braces? glob? Another module? FWIW (i.e not much), my opinion is leaning towards implementing it in glob.py: - add '{' to the magic_check regex - in glob1 (which is called when the pattern 'has_magic'), expand the braces and then call fnmatch.filter() on each resulting pattern That would respect more what is done in shells like Bash, and it makes it also more straight-forward to implement. Cheers, -- Mathieu

5 6

Cofunctions/yield from -> fibers
by Ivan Pozdeev 16 Aug '10

16 Aug '10

The original proposal for introducing 'yield from' (which is still in PEP 380's name) was to delegate a part of generator's work to another generator. However, in later discussions, the focus totally shifted to cooperative multitasking. In the example code Greg has given in http://mail.python.org/pipermail/python-ideas/2010-August/007927.html , there's not a single use case for delegating! 'yield from's essentially replace normal function calls! All Greg uses this stuff for is to manually switch between `threads' simulating individual customers! The _programming_logic_ is plain function calls, yields/cocalls are just a crutch (and quite an ugly one) to release a time slot. So, in fact, this all `yield from/coroutine' effort is an attempt to unseparatably mix two very loosely-related subjects: - code delegation (sequence in which sequential/closely-related code is executed) - execution scheduling (sequence in which parallel/loosely-related/unrelated code is executed) in the form 'a function call _may_be_ a scheduling event at the same time'! That's why it all feels so `clumsy/weird/twisted/fallacious' ! Cooperative threads must be switched somehow but choosing such a quirky and implicit technique for that is completely against Python Zen (violates about half of the items :^) )! Instead, if it's cooperative multitasking you play with, switching must be independent from other activity and as explicit as possible. There's a technique just for that called 'fibers' (MS fiber API: http://msdn.microsoft.com/en-us/library/ms682661.aspx ). In short: - ConvertThreadToFiber() decorates current thread as an initial fiber - CreateFiber (*function, *parameter)->ID creates a new fiber - SwitchToFiber(ID) switches execution to another fiber. - ExitFiber() exits a fiber. In python, the function may just return as in threads As the goal of fibers is the same as threads, it's reasonable to derive knowledge from there too, maybe duplicate the interface where applicable. And as cofunctions, these Fibers are just what http://en.wikipedia.org/wiki/Coroutine describes. With interacting coroutines, there is no execution stack anymore: there are 'calls' but no 'returns'! It's essentially like passing messages back and forth as in win32k. `Yield from's are still valid. But only as code delegation technique, i.e. a shortcut to `for i in subgen(): yield i'. The latter statement looks brief enough for me to discard the proposal altogether. Send() and stuff doesn't look to fit in what generators were originally intended to do - producing sequences with arbitrary or unknown length.

5 14

Re: [Python-ideas] Cofunctions: It's alive! Its alive!
by Bruce Leban 11 Aug '10

11 Aug '10

On Aug 10, 2010 6:18 PM, "Greg Ewing" <greg.ewing(a)canterbury.ac.nz> wrote: > Bruce Leban wrote: >> Is there a pythonesque alternative? > > It's generally considered unpythonic to assign arbitary meanings > to randomly chosen punctuation (decorator syntax notwithstanding!) ... I quite agree. I hate random punctuation. Suppose hypothetically that cocall is just one of a class of execution modifiers that we might ultimately want to add. In that case maybe there's a syntax (analogous to adding decorator syntax) that doesn't require new keywords. There's yield, yield from and cocall. Are there more? (I don't know. I'm just wondering.) --- Bruce (via android)

2 1

Cofunctions without codef
by Greg Ewing 11 Aug '10

11 Aug '10

I think I may have found a way to eliminate the need for the 'codef' keyword. I realised that although it's important to prevent making a regular call to a cofunction, there's no need to prevent making a cocall to an ordinary function. In that case, it can just be treated like a normal call. So I suggest the following changes: 1. The presence of 'cocall' inside a function causes it to be a cofunction. There is no 'codef' keyword. 2. The 'cocall' construct checks whether the function supports cocalls, and if so, proceeds as previously specified. Otherwise, it calls the function normally and returns the result. (To allow for objects such as bound methods to wrap things which could be cofunctions or not, the __cocall__ method will be permitted to return NotImplemented as a way of signalling that cocalls are not supported.) Does this scheme sound any better? -- Greg

2 1

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

Python-ideas August 2010