Mailman 3 May 2005 - Python-Dev

Re: [Python-Dev] my first post: asking about a "decorator" module
by Raymond Hettinger May 6, 2005

May 6, 2005

> > Ultimately, some of these will likely end-up in the library. For the > > time being, I think it best that these get posted and evolve either as > > Wiki entries or as ASPN entries. The best practices and proven winners > > have yet to emerge. Solidifying first attempts is likely not a good > > idea. Putting tools in the standard library should be the last > > evolutionary step, not the first. > > Yes, of course. I just wanted to know it there was interest on the > subject. Yes, there has been quite a bit of interest including several ASPN recipes and a wiki: http://www.python.org/moin/PythonDecoratorLibrary Raymond

3 2

PEP 340: Non-looping version (aka PEP 310 redux)
by Nick Coghlan May 6, 2005

May 6, 2005

The discussion on the meaning of break when nesting a PEP 340 block statement inside a for loop has given me some real reasons to prefer PEP 310's single pass semantics for user defined statements (more on that at the end). The suggestion below is my latest attempt at combining the ideas of the two PEP's. For the keyword, I've used the abbreviation 'stmt' (for statement). I find it reads pretty well, and the fact that it *isn't* a real word makes it easier for me to track to the next item on the line to find out the actual statement name (I think this might be similar to the effect of 'def' not being a complete word making it easier for me to pick out the function name). I consequently use 'user statement' or 'user defined statement' to describe what PEP 340 calls anonymous block statements. I'm still fine with the concept of not using a keyword at all, though. Cheers, Nick. == User Defined Statement Usage Syntax == stmt EXPR1 [as VAR1]: BLOCK1 == User Defined Statement Semantics == the_stmt = EXPR1 terminated = False try: stmt_enter = the_stmt.__enter__ stmt_exit = the_stmt.__exit__ except AttributeError: raise TypeError("User statement required") try: VAR1 = stmt_enter() # Omit 'VAR1 =' if no 'as' clause except TerminateBlock: pass # Block is not entered at all in this case # If an else clause were to be permitted, the # associated block would be executed here else: try: try: BLOCK1 except: exc = sys.exc_info() terminated = True try: stmt_exit(*exc) except TerminateBlock: pass finally: if not terminated: try: stmt_exit(TerminateBlock) except TerminateBlock: pass Key points: * The supplied expression must have both __enter__ and __exit__ methods. * The result of the __enter__ method is assigned to VAR1 if VAR1 is given. * BLOCK1 is not executed if __enter__ raises an exception * A new exception, TerminateBlock, is used to signal statement completion * The __exit__ method is called with the exception tuple if an exception occurs * Otherwise it is called with TerminateBlock as the argument * The __exit__ method can suppress an exception by converting it to TerminateBlock or by returning without reraising the exception * return, break, continue and raise StopIteration are all OK inside BLOCK1. They affect the surrounding scope, and are in no way tampered with by the user defined statement machinery (some user defined statements may choose to suppress the raising of StopIteration, but the basic machinery doesn't do that) * Decouples user defined statements from yield expressions, the enhanced continue statement and generator finalisation. == New Builtin: statement == def statement(factory): try: factory.__enter__ factory.__exit__ # Supplied factory is already a user statement factory return factory except AttributeError: # Assume supplied factory is an iterable factory # Use it to create a user statement factory class stmt_factory(object): def __init__(*args, **kwds) self = args[0] self.itr = iter(factory(*args[1:], **kwds)) def __enter__(self): try: return self.itr.next() except StopIteration: raise TerminateBlock def __exit__(self, *exc_info): try: stmt_exit = self.itr.__exit__ except AttributeError: try: self.itr.next() except StopIteration: pass raise *exc_info # i.e. re-raise the supplied exception else: try: stmt_exit(*exc_info) except StopIteration: raise TerminateBlock Key points: * The supplied factory is returned unchanged if it supports the statement API (such as a class with both __enter__ and __exit__ methods) * An iterable factory (such as a generator, or class with an __iter__ method) is converted to a block statement factory * Either way, the result is a callable whose results can be used as EXPR1 in a user defined statement. * For statements constructed from iterators, the iterator's next() method is called once when entering the statement, and the result is assigned to VAR1 * If the iterator has an __exit__ method, it is invoked when the statement is exited. The __exit__ method is passed the exception information (which may indicate that no exception occurred). * If the iterator does not have an __exit__ method, it's next() method is invoked a second time instead * When an iterator is used to drive a user defined statement, StopIteration is translated to TerminateBlock * Main intended use is as a generator decorator * Decouples user defined statements from yield expressions, the enhanced continue statement and generator finalisation. == Justification for non-looping semantics == For most use cases, the effect PEP 340 block statements have on break and continue statements is both surprising and undesirable. This is highlighted by the major semantic difference between the following two cases: stmt locking(lock): for item in items: if handle(item): break for item in items: stmt locking(lock): if handle(item): break Instead of simply acquiring and releasing the lock on each iteration, as one would legitimately expect, the latter piece of code actually processes all of the items, instead of breaking out of the loop once one of the items is handled. With non-looping user defined statements, the above code works in the obvious fashion (the break statement ends the for loop, not the lock acquisition). With non-looping semantics, the implementation of the examples in PEP 340 is essentially identical - just add an invocation of @statement to the start of the generators. It also becomes significantly easier to write user defined statements manually as there is no need to track state: class locking: def __init__(self, lock): self.lock = lock def __enter__(self): self.lock.acquire() def __exit__(self, exc_type, value=None, traceback=None): self.lock.release() if type is not None: raise exc_type, value, traceback The one identified use case for a user-defined loop was PJE's auto_retry. We already have user-defined loops in the form of custom iterators, and there is nothing stopping an iterator from returning user defined statements like this: for attempt in auto_retry(3, IOError): stmt attempt: # Do something! # Including break to give up early # Or continue to try again without raising IOError The implementation of auto-retry is messier than it is with all user defined statement being loops, but I think the benefits of non-looping semantics justify that sacrifice. Besides, it really isn't all that bad: class auto_retry(3, IOError): def __init__(self, times, exc=Exception): self.times = xrange(times-1) self.exc = exc self.succeeded = False def __iter__(self): attempt = self.attempt for i in self.times: yield attempt() if self.succeeded: break else: yield self.last_attempt() @statement def attempt(self): try: yield None self.succeeded = True except self.exc: pass @statement def last_attempt(self): yield None (Third time lucky! One day I'll remember that Python has these things called classes designed to elegantly share state between a collection of related functions and generators. . .) The above code for auto_retry assumes that generators supply an __exit__ method as described in PEP 340 - without that, auto_retry.attempt would need to be written as a class since it needs to know if an exception was thrown or not: class auto_retry(3, IOError): def __init__(self, times, exc=Exception): self.times = xrange(times-1) self.exc = exc self.succeeded = False def __iter__(self): attempt = self.attempt for i in self.times: yield attempt(self) if self.succeeded: break else: yield self.last_attempt() class attempt(object): def __init__(self, outer): self.outer = outer def __enter__(self): pass def __exit__(self, exc_type, value=None, traceback=None): if exc_type is None: self.outer.succeeded = true elif exc_type not in self.outer.exc raise exc_type, value, traceback @statement def last_attempt(self): yield None -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

3 2

PEP 340: Examples as class's.
by Ron Adam May 6, 2005

May 6, 2005

Eric Nieuwland wrote: > Ron Adam wrote: > >> Eric Nieuwland wrote: >> >>> This is linear. No looping whatsoever. And easily translated to a >>> simple language construct and a protocol: >>> >>> class resource(object): >>> def __init__(self,...): >>> # store resource parameters >>> def __acquire__(self): >>> # whatever it takes to grab the resource >>> def __release__(self): >>> # free the resource >> >> >> >> >> I wanted to see what the examples in PEP340 would look like written with >> standard class's using object inheritance and overriding to define >> resource managers. If anyone's interested I can post it. > > > I'm interested. > >> My block class is non-looping as I found in most cases looping isn't >> required, and looping complicates things because you have to pass around >> a loop expression due to not all loops will want to behave that same way. >> >> The solution was to put the loop in the body method and call a >> repeat_body method (the repeated body section) which is added to the >> class when needed. > > > Show us your stuff! ;-) > > --eric > Ok, :) These probably can be improved on, and renamed, and I don't know how many problems there may be with it, but they work in these examples. This is just one way to do it, so critique, correct, and improve as needed. Just no flames please. ;-) Cheers, Ron_Adam # --- start --- ## blockclass.py ## A base class that manages resource acquire and release. class Block(object): """A manager class for working with resources.""" def __init__(self,*args): self._cleanup = False self.__err__ = None try: self.start(*args) self._cleanup = True except: raise def __call__(self, *args): self.block(*args) self.__del__() def __del__(self): if self._cleanup: try: self.final() self._cleanup = False except: raise if self.__err__: raise self.__err__ def start(self,*args): """Override to add initialization code""" pass def final(self): """Override to add finalization code""" pass def block(self,*args): """Override to add main block body""" pass ## A dummy lock for test calls only class mylock: def acquire(self): print "Lock acquired" def release(self): print "Lock released" ## 1. A template for ensuring that a lock, acquired at the start of a ## block, is released when the block is left: class Get_Lock(Block): def start(self, lock): self.lock = lock self.lock.acquire() def final(self): self.lock.release() def block(self): pass class Lock_It(Get_Lock): def block(self): print "Do stuff while locked" Lock_It(mylock())() ## 2. A template for opening a file that ensures the file is closed: class File_Opener(Block): def start(self, filename, mode='r'): self.filename = filename self.mode = mode self.f = open(filename, mode) def final(self): self.f.close() def block(self): pass class Line_Printer(File_Opener): def block(self): n = 0 for line in self.f: print n, line.rstrip() n += 1 Line_Printer('blockclass.py')() ### 3. A template for committing or rolling back a database: #def transactional(db): # try: # yield # except: # db.rollback() # raise # else: # db.commit() """I'm not exactly sure how this one should work, so maybe some one else can give an example using the block class above.""" ## 4. A template that tries something up to n times: import urllib class Auto_Retry(Block): def start(self, n=3, exc=Exception): self.n = n self.exc = exc def block(self, *args): while self.n: try: self.repeat_block(*args) break except self.exc, self.__err__: self.n -= 1 def repeat_block(self, *args): """Try this block n times""" pass class Retry_Url(Auto_Retry): def repeat_block(self, url): f = urllib.urlopen(url) print f.read() # This could be slow, so wait for it. try: Retry_Url(3, IOError)("http://cantfind.com/this.html") except IOError, err: print err Retry_Url(3, IOError)("http://python.org/peps/pep-0340.html") ## 5. It is possible to nest blocks and combine templates: class Lockit(Get_Lock): def block(self, job): job() Lockit(mylock())(Line_Printer("blockclass.py")) ## 7. Redirect stdout temporarily: import sys class Redirect_Stdout(Block): def start(self, handle=sys.stdout): self.save_stdout = sys.stdout sys.stdout = handle def final(self): sys.stdout.close() sys.stdout = self.save_stdout def block(self): pass class New_Out(Redirect_Stdout): def block(self): print "Writing to redirected std_out" New_Out(open('newfile.txt','w'))() ## 8. A variant on opening() that also returns an ## error condition: class Append_To(File_Opener): def start(self, filename): self.f = open(filename, 'a') def block(self, string): self.f.write(string) def append_w_error(file,sting): try: Append_To(file)(string) except IOError, err: print "IOError:", err append_w_error("/etc/passwd", "guido::0:0::/:/bin/sh\n") # --- end ---

1 1

Re: [Python-Dev] PEP 340 - For loop cleanup, and feature separation
by Delaney, Timothy C (Timothy) May 6, 2005

May 6, 2005

Greg Ewing wrote: > I'm still bothered by the idea of for-loops not participating > in the new generator finalization protocol. I agree - that's always been nagging at me too. The problem with it is that then you either: 1. Have a guarantee that an iterator will be exhausted when the for loop exits (as per block statements); OR 2. Rely on garbage collection (reference counting, etc) to ensure finalisation. Personally, I'm of the opinion that we should make a significant break (no pun intended ;) and have for-loops attempt to ensure that iterators are exhausted. An iterator could be specifically designed to prevent that if needed, but in the vast majority of cases an iterator is never used after the for-loop. An example of an iterator that was specifically designed to continue working after its initial for-loop would be:: def gen(): try: setup1() try: yield 1 finally: cleanup1() except StopIteration: pass setup2() try: yield 2 finally: cleanup2() This allows cleanup, but then continued usage. > So I think that, in the interests of least surprise, > for-loops should provide the same finalization promises > as block statements. Agreed. > If that is done, it becomes easier to decide whether > the block statement should loop. The answer is probably > no: If you're writing a loop, you use a for-statement; > if you're not, you use a block-statement. This helps > to clearly differentiate the two, and provide > justification for having two statements. Agreed. > I'm also starting to like the idea of having a > completely separate protocol for block-statements +1 > and an adaptor of some kind for using generators to > implement them. My preference would be direct syntactic support... Tim Delaney

1 0

Re: [Python-Dev] PEP 340: Breaking out.
by Delaney, Timothy C (Timothy) May 6, 2005

May 6, 2005

Greg Ewing wrote: > Seems to me it should be up to the block iterator whether > a break statement gets caught or propagated, since it's > up to the block iterator whether the construct behaves > like a loop or not. > > This could be achieved by having a separate exception > for breaks, as originally proposed. > > If the iterator propagates the Break exception back out, > the block statement should break any enclosing loop. > If the iterator wants to behave like a loop, it can > catch the Break exception and raise StopIteration > instead. In this scenario (and I'm not saying I approve or disapprove) I think BreakIteration should inherit from StopIteration (thus retaining the existing PEP 340 semantics if uncaught):: Iteration | +- ContinueIteration | +- StopIteration | +- BreakIteration I think no matter what it would be useful to be able to separate a break from a stop sometimes, but there are many cases where they are the same thing. Tim Delaney

2 1

PEP 340: propose to get rid of 'as' keyword
by Gustavo J. A. M. Carneiro May 6, 2005

May 6, 2005

I have not read every email about this subject, so sorry if this has already been mentioned. In PEP 340 I read: block EXPR1 as VAR1: BLOCK1 I think it would be much clearer this (plus you save one keyword): block VAR1 = EXPR1: BLOCK1 Regards. -- Gustavo J. A. M. Carneiro <gjc(a)inescporto.pt> <gustavo(a)users.sourceforge.net> The universe is always one step beyond logic.

5 7

Re: [Python-Dev] PEP 340: Breaking out.
by Josiah Carlson May 5, 2005

May 5, 2005

Ka-Ping Yee <yahoo(a)zesty.ca> wrote: > > On Thu, 5 May 2005, Josiah Carlson wrote: > > Ka-Ping Yee <python-dev(a)zesty.ca> wrote: > > > continue with 2 > > > > There is something about <action> <which> <level> that I just don't like. > > Just to clarify: if by <level> you mean "nesting level", did it appear > that the 2 in my example was a count of block levels? I didn't > mean it that way -- the 2 is a value passed in to the generator > that appears as the value of the "yield" expression, as in PEP 340. I remember reading that, but I seem to have forgotten it when I was composing my reply. Thankfully, sleeping on it has helped me discover what I really don't like. With the 'passing value' semantic, the <action> [ <which> ] [ <value> ] is only useful for the deepest loop of a particular type. Take for example... for ...: for ...: for ...: break/continue [for] That break or continue can only affect that last for loop. It doesn't make any easier the use of nested fors, nested whiles, or even nested blocks. It only really helps you if you mix and match all possible looping constructs, and even then, only gives the granularity of the most recent block of a particular type. In that sense, I think it is a nonstarter, because it doesn't really add functionality in common uses of for and while statements. If one allowed <action> [<which>] [<value>] , [<level>], then one could jump to arbitrary loops. Now, I'm not condoning this, and I don't even like it. Sure, it allows breaking or continuing to any for, while, or block statement in the current scope, but the <level> argument is as equivalently ambiguous as package-relative imports using a leading integer (http://python.org/peps/pep-0328.html). Now, one can remove ambiguity if we were able to 'label' while loops and for loops producing <action> [ <label> ] , [ <value> ], but at that point we are getting into the discussion of a loop-aware goto with loop/block cleanup, and a syntax for labeling loops. Ick. - Josiah

1 0

problems with memory management
by Carlos Garcia May 5, 2005

May 5, 2005

Hi All, I do hava a problem with python and it is that it raise an outofmemory (i comment lines in Py.java to avoid system.exit, to debug), i try to debug this issue with jprobe and realize that i get the exception even although the java heap is not in the limit, i book 64- 256M and the java heap was less than 60 M. The program is a command line that receive a line that python parse an call some java classes ti execute the appropiate command, any idea? Thansk, ========================================================== Carlos García Phone : +34 91 714 8796 Lucent Technologies e-mail : cgarciaf(a)lucent.com Avenida de Bruselas , 8 - 28108 Alcobendas (Madrid) ==========================================================

2 1

Re: [Python-Dev] PEP 340 -- concept clarification
by Delaney, Timothy C (Timothy) May 5, 2005

May 5, 2005

Nick Coghlan wrote: > Ah, someone else did post this idea first :) I knew I was standing on the shoulders of others :) > To deal with the generator issue, one option would be to follow up on > Phillip's idea of a decorator to convert a generator (or perhaps any > standard iterator) into a block iterator. > > I think this would also do wonders for emphasising the difference > between for loops and block statements. I think if we are going to emphasise the difference, a decorator does not go far enough. To use a decorator, this *must* be valid syntax:: def gen(): try: yield finally: print 'Done!' However, that generator cannot be properly used in a for-loop. So it's only realistically valid with the decorator, and used in a block statement (resource suite ;) My feeling is that the above should be a SyntaxError, as it currently is, and that a new keyword is needed which explicitly allows the above, and creates an object conforming to the resource protocal (as I called it). Tim Delaney

2 1

Adding DBL_MANTISSA and such to Python
by Edward C. Jones May 5, 2005

May 5, 2005

Recently I needed some information about the floating point numbers on my machine. So I wrote a tiny C99 program with the line printf("%a\n", DBL_EPSILON); The answer was "0x1p-52". A search of comp.lang.python shows that I was not alone. Here are some ideas. 1. Add to Python the constants in "float.h" and "limits.h". 2. Add the C99 "%a" format to the "%" operator for strings and allow it in floating point literals. 3. Add full "tostring" and "fromstring" capabilities for Python numeric types. "tostring(x)" would return a string containing the binary representation of x. For example, if x is a Python float, "tostring(x)" would have eight characters. "fromstring(s, atype)" does the reserve, so fromstring(tostring(x), type(x)) == x 4. Add some functions that process floating point types at a low level. I suggest borrowing from C (mantissa, exponent) = frexp(x) where mantissa is a float and exponent is an int. The mantissa can be 0.0 or 0.5 <= mantissa < 1.0. Also x = mamtissa * 2**exponent. If x == 0.0, the function returns (0.0, 0). (This is almost a quote from Harbison and Steele.) 5. Add the C99 constants and functions involving special floating point values: "FP_INFINITE", "FP_NAN", "FP_NORMAL", "FP_SUBNORMAL", "FP_ZERO", "fpclassify", "isfinite", "isinf", "isnan", "isnormal", "signbit", "copysign", "nan", "nextafter", and "nexttoward". There has been controversy about these in the past, but I am in favor of them. The documentation should discuss portability.

4 4