Mailman 3 January 2012 - Python-Dev

"Documenting Python" is moving to devguide
by Sandro Tosi Jan. 15, 2012

Jan. 15, 2012

Hi all, (another) heads-up about my current work: I've just pushed the "Documenting Python" doc section (ftr: http://docs.python.org/documenting/index.html) to devguide. That was possibile now that we use the same sphinx version on all the active branches. It was not a re-editing of the content, that might still be outdated and in need of work, but just a brutal cut & paste of the current files. Now that we have a central place, additional editing will be much more easy. The section is … [View More]

2 2

Re: [Python-Dev] cpython: Implement PEP 380 - 'yield from' (closes #11682)
by Nick Coghlan Jan. 14, 2012

Jan. 14, 2012

On Jan 14, 2012 5:56 PM, "Georg Brandl" <g.brandl(a)gmx.net> wrote: > > On 01/14/2012 07:53 AM, Nick Coghlan wrote: > > I agree, but it's one of the challenges of a long-lived branch like > > the PEP 380 one (I believe some of these cosmetic changes started life > > in Greg's original patch and separating them out would have been quite > > a pain). Anyone that wants to see the gory details of the branch > > history can take a look at my bitbucket repo: … [View More]

1 0

Re: [Python-Dev] cpython: Implement PEP 380 - 'yield from' (closes #11682)
by Georg Brandl Jan. 14, 2012

Jan. 14, 2012

Caution, long review ahead. On 01/13/2012 12:43 PM, nick.coghlan wrote: > http://hg.python.org/cpython/rev/d64ac9ab4cd0 > changeset: 74356:d64ac9ab4cd0 > user: Nick Coghlan <ncoghlan(a)gmail.com> > date: Fri Jan 13 21:43:40 2012 +1000 > summary: > Implement PEP 380 - 'yield from' (closes #11682) > diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst > --- a/Doc/reference/expressions.rst > +++ b/Doc/reference/expressions.… [View More]rst > @@ -318,7 +318,7 @@ There should probably be a "versionadded" somewhere on this page. > .. productionlist:: > yield_atom: "(" `yield_expression` ")" > - yield_expression: "yield" [`expression_list`] > + yield_expression: "yield" [`expression_list` | "from" `expression`] > > The :keyword:`yield` expression is only used when defining a generator function, > and can only be used in the body of a function definition. Using a > @@ -336,7 +336,10 @@ > the generator's methods, the function can proceed exactly as if the > :keyword:`yield` expression was just another external call. The value of the > :keyword:`yield` expression after resuming depends on the method which resumed > -the execution. > +the execution. If :meth:`__next__` is used (typically via either a > +:keyword:`for` or the :func:`next` builtin) then the result is :const:`None`, > +otherwise, if :meth:`send` is used, then the result will be the value passed > +in to that method. > > .. index:: single: coroutine > > @@ -346,12 +349,29 @@ > where should the execution continue after it yields; the control is always > transferred to the generator's caller. > > -The :keyword:`yield` statement is allowed in the :keyword:`try` clause of a > +:keyword:`yield` expressions are allowed in the :keyword:`try` clause of a > :keyword:`try` ... :keyword:`finally` construct. If the generator is not > resumed before it is finalized (by reaching a zero reference count or by being > garbage collected), the generator-iterator's :meth:`close` method will be > called, allowing any pending :keyword:`finally` clauses to execute. > > +When ``yield from expression`` is used, it treats the supplied expression as > +a subiterator. All values produced by that subiterator are passed directly > +to the caller of the current generator's methods. Any values passed in with > +:meth:`send` and any exceptions passed in with :meth:`throw` are passed to > +the underlying iterator if it has the appropriate methods. If this is not the > +case, then :meth:`send` will raise :exc:`AttributeError` or :exc:`TypeError`, > +while :meth:`throw` will just raise the passed in exception immediately. > + > +When the underlying iterator is complete, the :attr:`~StopIteration.value` > +attribute of the raised :exc:`StopIteration` instance becomes the value of > +the yield expression. It can be either set explicitly when raising > +:exc:`StopIteration`, or automatically when the sub-iterator is a generator > +(by returning a value from the sub-generator). > + > +The parentheses can be omitted when the :keyword:`yield` expression is the > +sole expression on the right hand side of an assignment statement. > + > .. index:: object: generator > > The following generator's methods can be used to control the execution of a > @@ -444,6 +464,10 @@ > The proposal to enhance the API and syntax of generators, making them > usable as simple coroutines. > > + :pep:`0380` - Syntax for Delegating to a Subgenerator > + The proposal to introduce the :token:`yield_from` syntax, making delegation > + to sub-generators easy. > + > > .. _primaries: > > PEP 3155: Qualified name for classes and functions > ================================================== > > @@ -208,7 +224,6 @@ > how they might be accessible from the global scope. > > Example with (non-bound) methods:: > - > >>> class C: > ... def meth(self): > ... pass This looks like a spurious (and syntax-breaking) change. > diff --git a/Grammar/Grammar b/Grammar/Grammar > --- a/Grammar/Grammar > +++ b/Grammar/Grammar > @@ -121,7 +121,7 @@ > |'**' test) > # The reason that keywords are test nodes instead of NAME is that using NAME > # results in an ambiguity. ast.c makes sure it's a NAME. > -argument: test [comp_for] | test '=' test # Really [keyword '='] test > +argument: (test) [comp_for] | test '=' test # Really [keyword '='] test This looks like a change without effect? > diff --git a/Include/genobject.h b/Include/genobject.h > --- a/Include/genobject.h > +++ b/Include/genobject.h > @@ -11,20 +11,20 @@ > struct _frame; /* Avoid including frameobject.h */ > > typedef struct { > - PyObject_HEAD > - /* The gi_ prefix is intended to remind of generator-iterator. */ > + PyObject_HEAD > + /* The gi_ prefix is intended to remind of generator-iterator. */ > > - /* Note: gi_frame can be NULL if the generator is "finished" */ > - struct _frame *gi_frame; > + /* Note: gi_frame can be NULL if the generator is "finished" */ > + struct _frame *gi_frame; > > - /* True if generator is being executed. */ > - int gi_running; > + /* True if generator is being executed. */ > + int gi_running; > > - /* The code object backing the generator */ > - PyObject *gi_code; > + /* The code object backing the generator */ > + PyObject *gi_code; > > - /* List of weak reference. */ > - PyObject *gi_weakreflist; > + /* List of weak reference. */ > + PyObject *gi_weakreflist; > } PyGenObject; While these change tabs into spaces, it should be 4 spaces, not 8. > @@ -34,6 +34,7 @@ > > PyAPI_FUNC(PyObject *) PyGen_New(struct _frame *); > PyAPI_FUNC(int) PyGen_NeedsFinalizing(PyGenObject *); > +PyAPI_FUNC(int) PyGen_FetchStopIterationValue(PyObject **); > > #ifdef __cplusplus > } Does this API need to be public? If yes, it needs to be documented. > diff --git a/Include/opcode.h b/Include/opcode.h > --- a/Include/opcode.h > +++ b/Include/opcode.h > @@ -7,116 +7,117 @@ > > /* Instruction opcodes for compiled code */ > > -#define POP_TOP 1 > -#define ROT_TWO 2 > -#define ROT_THREE 3 > -#define DUP_TOP 4 > +#define POP_TOP 1 > +#define ROT_TWO 2 > +#define ROT_THREE 3 > +#define DUP_TOP 4 > #define DUP_TOP_TWO 5 > -#define NOP 9 > +#define NOP 9 > > -#define UNARY_POSITIVE 10 > -#define UNARY_NEGATIVE 11 > -#define UNARY_NOT 12 > +#define UNARY_POSITIVE 10 > +#define UNARY_NEGATIVE 11 > +#define UNARY_NOT 12 > > -#define UNARY_INVERT 15 > +#define UNARY_INVERT 15 > > -#define BINARY_POWER 19 > +#define BINARY_POWER 19 > > -#define BINARY_MULTIPLY 20 > +#define BINARY_MULTIPLY 20 > > -#define BINARY_MODULO 22 > -#define BINARY_ADD 23 > -#define BINARY_SUBTRACT 24 > -#define BINARY_SUBSCR 25 > +#define BINARY_MODULO 22 > +#define BINARY_ADD 23 > +#define BINARY_SUBTRACT 24 > +#define BINARY_SUBSCR 25 > #define BINARY_FLOOR_DIVIDE 26 > #define BINARY_TRUE_DIVIDE 27 > #define INPLACE_FLOOR_DIVIDE 28 > #define INPLACE_TRUE_DIVIDE 29 > > -#define STORE_MAP 54 > -#define INPLACE_ADD 55 > -#define INPLACE_SUBTRACT 56 > -#define INPLACE_MULTIPLY 57 > +#define STORE_MAP 54 > +#define INPLACE_ADD 55 > +#define INPLACE_SUBTRACT 56 > +#define INPLACE_MULTIPLY 57 > > -#define INPLACE_MODULO 59 > -#define STORE_SUBSCR 60 > -#define DELETE_SUBSCR 61 > +#define INPLACE_MODULO 59 > +#define STORE_SUBSCR 60 > +#define DELETE_SUBSCR 61 > > -#define BINARY_LSHIFT 62 > -#define BINARY_RSHIFT 63 > -#define BINARY_AND 64 > -#define BINARY_XOR 65 > -#define BINARY_OR 66 > -#define INPLACE_POWER 67 > -#define GET_ITER 68 > -#define STORE_LOCALS 69 > -#define PRINT_EXPR 70 > +#define BINARY_LSHIFT 62 > +#define BINARY_RSHIFT 63 > +#define BINARY_AND 64 > +#define BINARY_XOR 65 > +#define BINARY_OR 66 > +#define INPLACE_POWER 67 > +#define GET_ITER 68 > +#define STORE_LOCALS 69 > +#define PRINT_EXPR 70 > #define LOAD_BUILD_CLASS 71 > +#define YIELD_FROM 72 > > -#define INPLACE_LSHIFT 75 > -#define INPLACE_RSHIFT 76 > -#define INPLACE_AND 77 > -#define INPLACE_XOR 78 > -#define INPLACE_OR 79 > -#define BREAK_LOOP 80 > +#define INPLACE_LSHIFT 75 > +#define INPLACE_RSHIFT 76 > +#define INPLACE_AND 77 > +#define INPLACE_XOR 78 > +#define INPLACE_OR 79 > +#define BREAK_LOOP 80 > #define WITH_CLEANUP 81 > > -#define RETURN_VALUE 83 > -#define IMPORT_STAR 84 > +#define RETURN_VALUE 83 > +#define IMPORT_STAR 84 > > -#define YIELD_VALUE 86 > -#define POP_BLOCK 87 > -#define END_FINALLY 88 > -#define POP_EXCEPT 89 > +#define YIELD_VALUE 86 > +#define POP_BLOCK 87 > +#define END_FINALLY 88 > +#define POP_EXCEPT 89 > > -#define HAVE_ARGUMENT 90 /* Opcodes from here have an argument: */ > +#define HAVE_ARGUMENT 90 /* Opcodes from here have an argument: */ > > -#define STORE_NAME 90 /* Index in name list */ > -#define DELETE_NAME 91 /* "" */ > -#define UNPACK_SEQUENCE 92 /* Number of sequence items */ > -#define FOR_ITER 93 > +#define STORE_NAME 90 /* Index in name list */ > +#define DELETE_NAME 91 /* "" */ > +#define UNPACK_SEQUENCE 92 /* Number of sequence items */ > +#define FOR_ITER 93 > #define UNPACK_EX 94 /* Num items before variable part + > (Num items after variable part << 8) */ > > -#define STORE_ATTR 95 /* Index in name list */ > -#define DELETE_ATTR 96 /* "" */ > -#define STORE_GLOBAL 97 /* "" */ > -#define DELETE_GLOBAL 98 /* "" */ > +#define STORE_ATTR 95 /* Index in name list */ > +#define DELETE_ATTR 96 /* "" */ > +#define STORE_GLOBAL 97 /* "" */ > +#define DELETE_GLOBAL 98 /* "" */ > > -#define LOAD_CONST 100 /* Index in const list */ > -#define LOAD_NAME 101 /* Index in name list */ > -#define BUILD_TUPLE 102 /* Number of tuple items */ > -#define BUILD_LIST 103 /* Number of list items */ > -#define BUILD_SET 104 /* Number of set items */ > -#define BUILD_MAP 105 /* Always zero for now */ > -#define LOAD_ATTR 106 /* Index in name list */ > -#define COMPARE_OP 107 /* Comparison operator */ > -#define IMPORT_NAME 108 /* Index in name list */ > -#define IMPORT_FROM 109 /* Index in name list */ > +#define LOAD_CONST 100 /* Index in const list */ > +#define LOAD_NAME 101 /* Index in name list */ > +#define BUILD_TUPLE 102 /* Number of tuple items */ > +#define BUILD_LIST 103 /* Number of list items */ > +#define BUILD_SET 104 /* Number of set items */ > +#define BUILD_MAP 105 /* Always zero for now */ > +#define LOAD_ATTR 106 /* Index in name list */ > +#define COMPARE_OP 107 /* Comparison operator */ > +#define IMPORT_NAME 108 /* Index in name list */ > +#define IMPORT_FROM 109 /* Index in name list */ > > -#define JUMP_FORWARD 110 /* Number of bytes to skip */ > -#define JUMP_IF_FALSE_OR_POP 111 /* Target byte offset from beginning of code */ > -#define JUMP_IF_TRUE_OR_POP 112 /* "" */ > -#define JUMP_ABSOLUTE 113 /* "" */ > -#define POP_JUMP_IF_FALSE 114 /* "" */ > -#define POP_JUMP_IF_TRUE 115 /* "" */ > +#define JUMP_FORWARD 110 /* Number of bytes to skip */ > +#define JUMP_IF_FALSE_OR_POP 111 /* Target byte offset from beginning of code */ > +#define JUMP_IF_TRUE_OR_POP 112 /* "" */ > +#define JUMP_ABSOLUTE 113 /* "" */ > +#define POP_JUMP_IF_FALSE 114 /* "" */ > +#define POP_JUMP_IF_TRUE 115 /* "" */ > > -#define LOAD_GLOBAL 116 /* Index in name list */ > +#define LOAD_GLOBAL 116 /* Index in name list */ > > -#define CONTINUE_LOOP 119 /* Start of loop (absolute) */ > -#define SETUP_LOOP 120 /* Target address (relative) */ > -#define SETUP_EXCEPT 121 /* "" */ > -#define SETUP_FINALLY 122 /* "" */ > +#define CONTINUE_LOOP 119 /* Start of loop (absolute) */ > +#define SETUP_LOOP 120 /* Target address (relative) */ > +#define SETUP_EXCEPT 121 /* "" */ > +#define SETUP_FINALLY 122 /* "" */ > > -#define LOAD_FAST 124 /* Local variable number */ > -#define STORE_FAST 125 /* Local variable number */ > -#define DELETE_FAST 126 /* Local variable number */ > +#define LOAD_FAST 124 /* Local variable number */ > +#define STORE_FAST 125 /* Local variable number */ > +#define DELETE_FAST 126 /* Local variable number */ > > -#define RAISE_VARARGS 130 /* Number of raise arguments (1, 2 or 3) */ > +#define RAISE_VARARGS 130 /* Number of raise arguments (1, 2 or 3) */ > /* CALL_FUNCTION_XXX opcodes defined below depend on this definition */ > -#define CALL_FUNCTION 131 /* #args + (#kwargs<<8) */ > -#define MAKE_FUNCTION 132 /* #defaults + #kwdefaults<<8 + #annotations<<16 */ > -#define BUILD_SLICE 133 /* Number of items */ > +#define CALL_FUNCTION 131 /* #args + (#kwargs<<8) */ > +#define MAKE_FUNCTION 132 /* #defaults + #kwdefaults<<8 + #annotations<<16 */ > +#define BUILD_SLICE 133 /* Number of items */ > > #define MAKE_CLOSURE 134 /* same as MAKE_FUNCTION */ > #define LOAD_CLOSURE 135 /* Load free variable from closure */ Not sure putting these and all the other cosmetic changes into an already big patch is such a good idea... > diff --git a/Include/pyerrors.h b/Include/pyerrors.h > --- a/Include/pyerrors.h > +++ b/Include/pyerrors.h > @@ -51,6 +51,11 @@ > Py_ssize_t written; /* only for BlockingIOError, -1 otherwise */ > } PyOSErrorObject; > > +typedef struct { > + PyException_HEAD > + PyObject *value; > +} PyStopIterationObject; > + > /* Compatibility typedefs */ > typedef PyOSErrorObject PyEnvironmentErrorObject; > #ifdef MS_WINDOWS > @@ -380,6 +385,8 @@ > const char *reason /* UTF-8 encoded string */ > ); > > +/* create a StopIteration exception with the given value */ > +PyAPI_FUNC(PyObject *) PyStopIteration_Create(PyObject *); About this API see below. > diff --git a/Objects/abstract.c b/Objects/abstract.c > --- a/Objects/abstract.c > +++ b/Objects/abstract.c > @@ -2267,7 +2267,6 @@ > > func = PyObject_GetAttrString(o, name); > if (func == NULL) { > - PyErr_SetString(PyExc_AttributeError, name); > return 0; > } > > @@ -2311,7 +2310,6 @@ > > func = PyObject_GetAttrString(o, name); > if (func == NULL) { > - PyErr_SetString(PyExc_AttributeError, name); > return 0; > } > va_start(va, format); These two changes also look suspiciously unrelated? > +PyObject * > +PyStopIteration_Create(PyObject *value) > +{ > + return PyObject_CallFunctionObjArgs(PyExc_StopIteration, value, NULL); > +} I think this function is rather questionable. It is only used once at all. If kept, it should rather be named _PyE{rr,xc}_CreateStopIteration. But since it's so trivial, it should be removed altogether. > diff --git a/Objects/genobject.c b/Objects/genobject.c > --- a/Objects/genobject.c > +++ b/Objects/genobject.c > @@ -5,6 +5,9 @@ > #include "structmember.h" > #include "opcode.h" > > +static PyObject *gen_close(PyGenObject *gen, PyObject *args); > +static void gen_undelegate(PyGenObject *gen); > + > static int > gen_traverse(PyGenObject *gen, visitproc visit, void *arg) > { > @@ -90,12 +93,18 @@ > > /* If the generator just returned (as opposed to yielding), signal > * that the generator is exhausted. */ > - if (result == Py_None && f->f_stacktop == NULL) { > - Py_DECREF(result); > - result = NULL; > - /* Set exception if not called by gen_iternext() */ > - if (arg) > + if (result && f->f_stacktop == NULL) { > + if (result == Py_None) { > + /* Delay exception instantiation if we can */ > PyErr_SetNone(PyExc_StopIteration); > + } else { > + PyObject *e = PyStopIteration_Create(result); > + if (e != NULL) { > + PyErr_SetObject(PyExc_StopIteration, e); > + Py_DECREF(e); > + } Wouldn't PyErr_SetObject(PyExc_StopIteration, value) suffice here anyway? > +/* > + * If StopIteration exception is set, fetches its 'value' > + * attribute if any, otherwise sets pvalue to None. > + * > + * Returns 0 if no exception or StopIteration is set. > + * If any other exception is set, returns -1 and leaves > + * pvalue unchanged. > + */ > + > +int > +PyGen_FetchStopIterationValue(PyObject **pvalue) { > + PyObject *et, *ev, *tb; > + PyObject *value = NULL; > + > + if (PyErr_ExceptionMatches(PyExc_StopIteration)) { > + PyErr_Fetch(&et, &ev, &tb); > + Py_XDECREF(et); > + Py_XDECREF(tb); > + if (ev) { > + value = ((PyStopIterationObject *)ev)->value; > + Py_DECREF(ev); > + } PyErr_Fetch without PyErr_Restore clears the exception, that should be mentioned in the docstring. Georg [View Less]

2 2

Re: [Python-Dev] [Python-checkins] cpython: add test, which was missing from d64ac9ab4cd0
by Nick Coghlan Jan. 13, 2012

Jan. 13, 2012

On Sat, Jan 14, 2012 at 5:39 AM, benjamin.peterson <python-checkins(a)python.org> wrote: > http://hg.python.org/cpython/rev/be85914b611c > changeset: 74363:be85914b611c > parent: 74361:609482c6710e > user: Benjamin Peterson <benjamin(a)python.org> > date: Fri Jan 13 14:39:38 2012 -0500 > summary: > add test, which was missing from d64ac9ab4cd0 Ah, that's where that came from, … [View More]

1 0

Summary of Python tracker Issues
by Python tracker Jan. 13, 2012

Jan. 13, 2012

ACTIVITY SUMMARY (2012-01-06 - 2012-01-13) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 3210 (+30) closed 22352 (+30) total 25562 (+60) Open issues with patches: 1384 Issues opened (42) ================== #6774: socket.shutdown documentation: on some platforms, closing one http://bugs.python.org/issue6774 reopened by neologix #13721: ssl.… [View More]

1 0

Proposed PEP on concurrent programming support
by Mike Meyer Jan. 12, 2012

Jan. 12, 2012

PEP: XXX Title: Interpreter support for concurrent programming Version: $Revision$ Last-Modified: $Date$ Author: Mike Meyer <mike(a)mired.org> Status: Draft Type: Informational Content-Type: text/x-rst Created: 11-Nov-2011 Post-History: Abstract ======== The purpose of this PEP is to explore strategies for making concurrent programming in Python easier by allowing the interpreter to detect and notify the user about possible bugs in concurrent access. The reason for doing so is that "… [View More]Errors should never pass silently". Such bugs are caused by allowing objects to be accessed simultaneously from another thread of execution while they are being modified. Currently, python systems provide no support for such bugs, falling back on the underlying platform facilities and some tools built on top of those. While these tools allow prevention of such modification if the programmer is aware of the need for them, there are no facilities to detect that such a need might exist and warn the programmer of it. The goal is not to prevent such bugs, as that depends on the programmer getting the logic of the interactions correct, which the interpreter can't judge. Nor is the goal to warn the programmer about any such modifications - the goal is to catch standard idioms making unsafe modifications. If the programmer starts tinkering with Python's internals, it's assumed they are aware of these issues. Rationale ========= Concurrency bugs are among the hardest bugs to locate and fix. They result in corrupt data being generated or used in a computation. Like most such bugs, the corruption may not become evident until much later and far away in the program. Minor changes in the code can cause the bugs to fail to manifest. They may even fail to manifest from run to run, depending on external factors beyond the control of the programmer. Therefore any help in locating and dealing with such bugs is valuable. If the interpreter is to provide such help, it must be aware of when things are safe to modify and when they are not. This means it will almost certainly cause incompatible changes in Python, and may impose costs so high for non-concurrent operations as to make it untenable. As such, the final options discussed are destined for Python version 4 or later, and may never be implemented in any mainstream implementation of Python. Terminology =========== The word "thread" is used throughout to mean "concurrent thread of execution". Nominally, this means a platform thread. However, it is intended to include any threading mechanism that allows the interpreter to change threads between or in the middle of a statement without the programmer specifically allowing this to happen. Similarly, the word "interpreter" means any system that processes and executes Python language files. While this normally means cPython, the changes discussed here should be amenable to other implementations. Concept ======= Locking object -------------- The idea is that the interpreter should indicate an error anytime an unlocked object is mutated. For mutable types, this would mean changing the value of the type. For Python class instances, this would mean changing the binding of an attribute. Mutating an object bound to such an attribute isn't a change in the object the attribute belongs to, and so wouldn't indicate an error unless the object bound to the attribute was unlocked. Locking by name --------------- It's also been suggested that locking "names" would be useful. That is, to prevent a specific attribute of an object from being rebound, or a key/index entry in a mapping object. This provides a finer grained locking than just locking the object, as you could lock a specific attribute or set of attributes of an object, without locking all of them. Unfortunately, this isn't sufficient: a set may need to be locked to prevent deletions for some period, or a dictionary to prevent adding a key, or a list to prevent changing a slice, etc. So some other locking mechanism is required. If that needs to specify objects, some way of distinguishing between locking a name and locking the object bound to the name needs to be invented, or there needs to be two different locking mechanisms. It's not clear that the finer grained locking is worth adding yet another language mechanism. Alternatives ============ Explicit locking ---------------- These alternatives requires that the programmer explicitly name anything that is going to be changed to lock it before changing it. This lets the interpreter gets involved, but makes a number of errors possible based on the order that locks are applied. Platform locks '''''''''''''' The current tool set uses platform locks via a C extension. The problem with these is that the interpreter has no knowledge of them, and so can't do anything about detecting the mutation of unlocked objects. A ``locking`` keyword ''''''''''''''''''''' Adding a statement to tell the interpreter to lock objects for the attached suite would let the interpreter know which objects are locked. To help prevent deadlocks, such a keyword needs to imply an order for locking objects, such that two objects locked by the a locking statement will lock the two objects in the same order during a single execution of the program. This can be achieved by sorting objects by the ``id`` of the object, since the requirements for ``id`` are sufficient for this. While the locking order requirement is sufficient to prevent deadlocks from non-nested locking statements, it's not sufficient if locking statements are allowed to nest. So either nested locking statements need to be disallowed, or the outer statement must lock everything that's going to need to be locked. Either requirement is sufficiently onerous that alternatives need to be considered. Implicit locking ---------------- In this alternative, the interpter uses one or more heuristics to decide when things should need locking. Software Transactional Memory (STM) ''''''''''''''''''''''''''''''''''' STM is a relatively new technology being experimented with in newer languages, and in a number of 3rd party libraries (both Peak [#Peak]_ and Kamaelia [#Kamaelia]_ provide STM facilities). A suite is marked as a `transaction`, and then when an unlocked object is modified, instead of indicating an error, a locked copy of it is created to be used through the rest of the transaction. If any of the originals are modified during the execution of the suite, the suite is rerun from the beginning. If it completes, the locked copies are copied back to the originals in an atomic manner. This causes the changes seen by any threads not running the transaction to be atomic. If two threads are updating the same object in transactions, the one that finishes second will be restarted with values set by the one that finished first. The advantage of an STM is that the programmer doesn't have to worry about what is locked, and there's no overhead for using locked objects (after locking them, of course). The disadvantage is that any code in a transaction must be safe to run multiple times. This forbids any kind of I/O. Compiler support '''''''''''''''' Since the point is to get the interpreter involved, we might as well let it be involved in figuring out which things are safe and don't need to be locked. This could potentially eliminate a lot of locking. Each object - whether a Python class instance or builtin type - is created with no way to access it until it is bound. So it is inherently safe to modify. Being bound to a local (or nonlocal?) variable doesn't change this. Being bound to a global, class or instance variable or stored in a container does change this, as the object may now be accessed from other threads via the module or container. Since this analysis is being done at compile time, being passed to another function - including methods of the object - makes it unsafe. Likewise, yielding an object makes it unsafe for future use. Returning it doesn't change anything, since our execution is over and we lose access to the object. Unfortunately, objects returned from functions must be treated as unsafe. Interpreter support ''''''''''''''''''' If we instead track whether or not objects require locking in the interpreter, then we can improve the analsysis. The only thing that definitely makes an object unsafe is binding to a global variable or a variable known to be unsafe. Passing objects to C routines exposes them to concurrent modification, since there's no way to know what will happen inside the C routine. Adding some way of marking C routines - or possibly the objects passed to them - as not exposing things to concurrent modification would help with this, allowing C modules to be called without requiring locking everything passed to them. Binding to class and instance variables, or adding them to a container, is an interesting issue. If the object in question is safe, then anything added to it is also safe. However, this would mean that when an object is flagged as unsafe, all objects accessible through it would also have to be flagged as unsafe. This type of tracking also means that objects effectively have three states: locked, unlocked, and safe. Both locked and safe objects can safely be modified without a problem. Locking and unlocking safe objects is a nop. Interpreter threading --------------------- One alternative is replacing the current threading tools - which are wrappers around the OS-provided threading - with threading support in the interpreter. This would allow the interpreter to control whether or not objects are shared between threads, which isn't possible today. The full implications of this approach have as yet to be worked out. Mixed solutions --------------- Most likely, any real implementation would use a number of the techniques above, since all of them have serious shortcomings. For instance, combining STM with explicit locking would allow explicit locking when IO was required, but complex multi-object changes could be handled by STM, thus avoiding the nested locking issues. Likewise, interpreter or compiler support could be mixed with most other solutions to relax the requirement of locking for part of the objects used in a program. The implications of mixing these things together also needs to be explored more thoroughly. Change proposal =============== This is 'strawman' proposal to provide a starting point for discussion. The proposal is to add an STM support to the python interpreter. A new suite type - the ``transaction`` will be added to the language. The suite will have the semantics discussed above: modifying an object in the suite will trigger creation of a thread-local shallow copy to be used in the Transaction. Further modifications of the original will cause all existing copies to be discarded and the transaction to be restarted. At the end of the transaction, the originals of all the copies are locked and then updated to the state of the copy. Further work ============ Requiring further investigation: - The interpreter providing it's own threading. - How various solutions interact when mixed. There are also a couple tools that might be useful to build, or at least investigate building: - A static concurrency safety analyzer, that handled the AST of a function to determine which variables are safe. - A dynamic concurrency safety analyzer, similar to coverage [#coverage]_. Implementation Notes ==================== Not significantly impacting the performance of single-threaded code must be of paramount importance to any implementation. One implementation technique arose that could help with this. Instead of keeping track of the objects state and having methods check that state and modify their behavior based on it, change the methods as the object changes state. So in safe or locked mode, the objects methods could freely modify the object without having to check it's mode. In unlocked mode, an attempt to do so would raise an error or warning. Unfortunately, this doesn't work if some global or thread state must be checked instead of just object-local state. References ========== .. [#Peak] "Peak, the Python Enterprise Application Kit", http://peak.telecommunity.com/ .. [#Kamaelia] "Kamaelia - Concurrency made useful, fun", http://www.kamaelia.org/ .. [#coverage] "Code coverage measurement for Python", http://pypi.python.org/pypi/coverage Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

3 4

os.walk() with followlinks=False
by Nick Coghlan Jan. 12, 2012

Jan. 12, 2012

When discussing http://bugs.python.org/issue13734, Charles-François noted that when os.walk() is called with "followlinks=False", symlinks to directories are still included in the "subdirs" list rather than the "files" list. This seems rather odd to me, so I'm asking here to see if there's a specific rationale for it, or if it's just an artifact of the implementation. If it's the latter... could we change it for 3.3, or is that too significant a breach of backwards compatibility? Even if we … [View More]

2 2

Python C API: Problem sending tuple to a method of a python Class
by pasparis＠noos.fr Jan. 11, 2012

Jan. 11, 2012

8 10

shutil.copy() and hard links
by Christian Heimes Jan. 11, 2012

Jan. 11, 2012

Hello, here is another fun fact about links, this time hard links and the shutil.copy() function. The shutil.copy() functions behaves like the Unix cp(1) command. Both don't unlink the destination file if it already exists. As a consequence all hard links point to the updated file data. This behavior may surprise some users. Perhaps the docs should point out how shutil.copy() works when hard links join the party. It might be worth to add a function that works similar to install(1). The … [View More]

1 0

Re: [Python-Dev] devguide: Backporting is obsolete. Add details that I had to learn.
by Antoine Pitrou Jan. 10, 2012

Jan. 10, 2012

On Mon, 09 Jan 2012 21:58:29 +0100 terry.reedy <python-checkins(a)python.org> wrote: > > -Different branches are used at a time to represent different *minor versions* > -in which development is made. All development should be done **first** in the > -:ref:`in-development <indevbranch>` branch, and selectively backported > -to other branches when necessary. > +There is a branch for each *minor version*. Development is done separately > +for Python 2 and … [View More]

8 10