Mailman 3 October 2005 - Python-Dev

Proposal for 2.5: Returning values from PEP 342 enhanced generators
by Piet Delport Oct. 7, 2005

Oct. 7, 2005

PEP 255 ("Simple Generators") closes with: > Q. Then why not allow an expression on "return" too? > > A. Perhaps we will someday. In Icon, "return expr" means both "I'm > done", and "but I have one final useful value to return too, and > this is it". At the start, and in the absence of compelling uses > for "return expr", it's simply cleaner to use "yield" exclusively > for delivering values. Now that Python 2.5 gained enhanced generators (multitudes rejoice!… [View More]

2 1

Sandboxed Threads in Python
by Adam Olsen Oct. 7, 2005

Oct. 7, 2005

Okay, basic principal first. You start with a sandboxed thread that has access to nothing. No modules, no builtins, *nothing*. This means it can run without the GIL but it can't do any work. To make it do something useful we need to give it two things: first, immutable types that can be safely accessed without locks, and second a thread-safe queue to coordinate. With those you can bring modules and builtins back into the picture, either by making them immutable or using a proxy that … [View More]handles all the methods in a single thread. Unfortunately python has a problem with immutable types. For the most part it uses an honor system, trusting programmers not to make a class that claims to be immutable yet changes state anyway. We need more than that, and "freezing" a dict would work well enough, so it's not the problem. The problem is the reference counting, and even if we do it "safely" all the memory writes just kill performance so we need to avoid it completely. Turns out it's quite easy and it doesn't harm performance of existing code or require modification (but a recompile is necessary). The idea is to only use a cyclic garbage collector for cleaning them up, which means we need to disable the reference counting. That requires we modify Py_INCREF and Py_DECREF to be a no-op if ob_refcnt is set to a magic constant (probably a negative value). That's all it takes. Modify Py_INCREF and Py_DECREFs to check for a magic constant. Ahh, but the performance? See for yourself. Normal Py_INCREF/Py_DECREF rhamph@factor:~/src/Python-2.4.1$ ./python Lib/test/pystone.py 500000 Pystone(1.1) time for 500000 passes = 13.34 This machine benchmarks at 37481.3 pystones/second Modified Py_INCREF/Py_DECREF with magic constant rhamph@factor:~/src/Python-2.4.1-sandbox$ ./python Lib/test/pystone.py 500000 Pystone(1.1) time for 500000 passes = 13.38 This machine benchmarks at 37369.2 pystones/second The numbers aren't significantly different. In fact the second one is often slightly faster, which shows the difference is smaller than the statistical noise. So to sum up, by prohibiting mutable objects from being transferred between sandboxes we can achieve scalability on multiple CPUs, making threaded programming easier and more reliable, as a bonus get secure sandboxes[1], and do that all while maintaining single-threaded performance and requiring minimal changes to existing C modules (recompiling). A "proof of concept" patch to Py_INCREF/Py_DECREF (only demonstrates performance effects, does not create or utilize any new functionality) can be found here: https://sourceforge.net/tracker/index.php?func=detail&aid=1316653&group_id=… [1] We need to remove any backdoor methods of getting to mutable objects outside of your sandbox, which gets us most of the way towards a restricted execution environment. -- Adam Olsen, aka Rhamphoryncus [View Less]

1 0

Proposed changes to PEP 343
by Nick Coghlan Oct. 7, 2005

Oct. 7, 2005

Based on Jason's comments regarding decimal.Context, and to explicitly cover the terminology agreed on during the documentation discussion back in July, I'm proposing a number of changes to PEP 343. I'll be updating the checked in PEP assuming there aren't any objections in the next week or so (and assuming I get CVS access sorted out ;). The idea of dropping __enter__/__exit__ and defining the with statement solely in terms of coroutines is *not* included in the suggested changes, but I … [View More]added a new item under "Resolved Open Issues" to cover some of the reasons why. Cheers, Nick. 1. Amend the statement specification such that: with EXPR as VAR: BLOCK is translated as: abc = (EXPR).__with__() exc = (None, None, None) VAR = abc.__enter__() try: try: BLOCK except: exc = sys.exc_info() raise finally: abc.__exit__(*exc) 2. Add the following to the subsequent explanation: The call to the __with__ method serves a similar purpose to the __iter__ method for iterables and iterators. An object such as threading.Lock may provide its own __enter__ and __exit__ methods, and simply return 'self' from its __with__ method. A more complex object such as decimal.Context may return a distinct context manager which takes care of setting and restoring the appropriate decimal context in the thread. 3. Update ContextWrapper in the "Generator Decorator" section to include: def __with__(self): return self 4. Add a paragraph to the end of the "Generator Decorator" section: By applying the @contextmanager decorator to a context's __with__ method, it is as easy to write a generator-based context manager for the context as it is to write a generator-based iterator for an iterable (see the decimal.Context example below). 5. Add three items under "Resolved Open Issues": 2. After this PEP was originally approved, a subsequent discussion on python-dev [4] settled on the term "context manager" for objects which provide __enter__ and __exit__ methods, and "context management protocol" for the protocol itself. With the addition of the __with__ method to the protocol, a natural extension is to call objects which provide only a __with__ method "contexts" (or "manageable contexts" in situations where the general term "context" would be ambiguous). The distinction between a context and a context manager is very similar to the distinction between an iterable and an iterator. 3. The originally approved version of this PEP did not include a __with__ method - the method was only added to the PEP after Jason Orendorff pointed out the difficulty of writing appropriate __enter__ and __exit__ methods for decimal.Context [5]. This approach allows a class to use the @contextmanager decorator to defines a native context manager using generator syntax. It also allows a class to use an existing independent context manager as its native context manager by applying the independent context manager to 'self' in its __with__ method. It even allows a class written in C to use a coroutine based context manager written in Python. The __with__ method parallels the __iter__ method which forms part of the iterator protocol. 4. The suggestion was made by Jason Orendorff that the __enter__ and __exit__ methods could be removed from the context management protocol, and the protocol instead defined directly in terms of the coroutine interface described in PEP 342 (or a cleaner version of that interface with start() and finish() convenience methods) [6]. Guido rejected this idea [7]. The following are some of benefits of keeping the __enter__ and __exit__ methods: - it makes it easy to implement a simple context manager in C without having to rely on a separate coroutine builder - it makes it easy to provide a low-overhead implementation for context managers which don't need to maintain any special state between the __enter__ and __exit__ methods (having to use a coroutine for these would impose unnecessary overhead without any compensating benefit) - it makes it possible to understand how the with statement works without having to first understand the concept of a coroutine 6. Add new references: [4] http://mail.python.org/pipermail/python-dev/2005-July/054658.html [5] http://mail.python.org/pipermail/python-dev/2005-October/056947.html [6] http://mail.python.org/pipermail/python-dev/2005-October/056969.html [7] http://mail.python.org/pipermail/python-dev/2005-October/057018.html 7. Update Example 4 to include a __with__ method: def __with__(self): return self 8. Replace Example 9 with the following example: 9. Here's a proposed native context manager for decimal.Context: # This would be a new decimal.Context method @contextmanager def __with__(self): # We set the thread context to a copy of this context # to ensure that changes within the block are kept # local to the block. This also gives us thread safety # and supports nested usage of a given context. newctx = self.copy() oldctx = decimal.getcontext() decimal.setcontext(newctx) try: yield newctx finally: decimal.setcontext(oldctx) Sample usage: def sin(x): with decimal.getcontext() as ctx: ctx.prec += 2 # Rest of sin calculation algorithm # uses a precision 2 greater than normal return +s # Convert result to normal precision def sin(x): with decimal.ExtendedContext: # Rest of sin calculation algorithm # uses the Extended Context from the # General Decimal Arithmetic Specification return +s # Convert result to normal context -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com [View Less]

5 9

__doc__ behavior in class definitions
by Martin Maly Oct. 7, 2005

Oct. 7, 2005

Hello Python-Dev, My name is Martin Maly and I am a developer at Microsoft, working on the IronPython project with Jim Hugunin. I am spending lot of time making IronPython compatible with Python to the extent possible. I came across a case which I am not sure if by design or a bug in Python (Python 2.4.1 (#65, Mar 30 2005, 09:13:57)). Consider following Python module: # module begin "module doc" class c: print __doc__ __doc__ = "class doc" (1) print __doc__ print c.__doc__ # … [View More]

5 4

Re: [Python-Dev] __doc__ behavior in class definitions
by Phillip J. Eby Oct. 7, 2005

Oct. 7, 2005

At 12:15 PM 10/7/2005 -0700, Martin Maly wrote: >Based on the binding rules described in the Python documentation, I >would expect the code to throw because binding created on the line (1) >is local to the class block and all the other __doc__ uses should >reference that binding. Apparently, it is not the case. Correct - the scoping rules about local bindings causing a symbol to be local only apply to *function* scopes. Class scopes are able to refer to module-level names until … [View More]

1 0

Re: [Python-Dev] Python 2.5 and ast-branch
by Phillip J. Eby Oct. 7, 2005

Oct. 7, 2005

At 07:34 PM 10/6/2005 -0700, Guido van Rossum wrote: >How does this sound to the non-AST-branch developers who have to >suffer the inevitable post-merge instability? I think it's now or >never -- waiting longer isn't going to make this thing easier (not >with several more language changes approved: with-statement, extended >import, what else...) Do the AST branch changes affect the interface of the "parser" module? Or do they just add new functionality?

2 1

PyObject_Init documentation
by "Martin v. Löwis" Oct. 7, 2005

Oct. 7, 2005

says If type indicates that the object participates in the cyclic garbage detector, it is added to the detector's set of observed objects. Is this really correct? I thought you need to invoke PyObject_GC_TRACK explicitly? Regards, Martin

1 0

Re: [Python-Dev] Static builds on Windows (continued)
by Marvin Oct. 6, 2005

Oct. 6, 2005

> Date: Wed, 05 Oct 2005 00:21:20 +0200 > From: "Martin v. L?wis" <martin(a)v.loewis.de> > Subject: Re: [Python-Dev] Static builds on Windows (continued) > Cc: python-dev(a)python.org > Message-ID: <43430060.6070909(a)v.loewis.de> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Marvin wrote: > >>I built pythoncore and python. The resulting python.exe worked fine, but did >>indeed fail when I tried to dynamically load … [View More]

1 0

Lexical analysis and NEWLINE tokens
by Matthew F. Barnes Oct. 6, 2005

Oct. 6, 2005

I posted this question to python-help, but I think I have a better chance of getting the answer here. I'm looking for clarification on when NEWLINE tokens are generated during lexical analysis of Python source code. In particular, I'm confused about some of the top-level components in Python's grammar (file_input, interactive_input, and eval_input). Section 2.1.7 of the reference manual states that blank lines (lines consisting only of whitespace and possibly a comment) do not generate … [View More]

4 3

Unicode charmap decoders slow
by Tony Nelson Oct. 6, 2005

Oct. 6, 2005

Is there a faster way to transcode from 8-bit chars (charmaps) to utf-8 than going through unicode()? I'm writing a small card-file program. As a test, I use a 53 MB MBox file, in mac-roman encoding. My program reads and parses the file into messages in about 3 to 5 seconds (Wow! Go Python!), but takes about 14 seconds to iterate over the cards and convert them to utf-8: for i in xrange(len(cards)): u = unicode(cards[i], encoding) cards[i] = u.encode('utf-8') The time is … [View More]

7 33