Mailman 3 May 2005 - Python-Dev

anonymous blocks
by Guido van Rossum May 4, 2005

May 4, 2005

I've been thinking about this a lot, but haven't made much progess. Here's a brain dump. I've been thinking about integrating PEP 325 (Resource-Release Support for Generators) into the for-loop code, so that you could replace the_lock.acquire() try: BODY finally: the_lock.release() with for dummy in synchronized(the_lock): BODY or perhaps even (making "for VAR" optional in the for-loop syntax) with in synchronized(the_lock): BODY Then synchronized() could be written cleanly as follows: def synchronized(lock): lock.acquire() try: yield None finally: lock.release() But then every for-loop would have to contain an extra try-finally clause; the translation of for VAR in EXPR: BODY would become __it = iter(EXPR) try: while True: try: VAR = __it.next() except StopIteration: break BODY finally: if hasattr(__it, "close"): __it.close() which I don't particularly like: most for-loops DON'T need this, since they don't use a generator but some other form of iterator, or even if they use a generator, not all generators have a try/finally loop. But the bytecode compiler can't know that, so it will always have to generate this code. It also changes the semantics of using a generator in a for-loop slightly: if you break out of the for-loop before the generator is exhausted you will still get the close() call. It's also a bit funny to see this approach used with the only other use case for try/finally we've looked at, which requires passing a variable into the block: the "with_file" use case. We now can write with_file as a nice and clean generator: def with_file(filename): f = open(filename) try: yield f finally: f.close() but the use looks very odd because it is syntactically a for-loop but there's only one iteration: for f in with_file("/etc/passwd"): for line in f: print line[:line.find(":")] Seeing this example makes me cringe -- why two nested for loops to loop over the lines of one file??? So I think that this is probably not the right thing to pursue, and we might be better off with something along the lines of PEP 310. The authors of PEP 310 agree; under Open Issues they wrote: There are some simiralities in concept between 'with ...' blocks and generators, which have led to proposals that for loops could implement the with block functionality[3]. While neat on some levels, we think that for loops should stick to being loops. (Footnote [3] references the tread that originated PEP 325.) Perhaps the most important lesson we've learned in this thread is that the 'with' keyword proposed in PEP 310 is redundant -- the syntax could just be [VAR '=']* EXPR ':' BODY IOW the regular assignment / expression statement gets an optional colon-plus-suite at the end. So now let's assume we accept PEP 310 with this change. Does this leave any use cases for anonymous blocks uncovered? Ruby's each() pattern is covered by generators; personally I prefer Python's for var in seq: ... over Ruby's much-touted seq.each() {|var| ...} The try/finally use case is covered by PEP 310. (If you want to combine this with a for-loop in a single operation, you'll need PEP 325.) The use cases where the block actually returns a value are probably callbacks for things like sort() or map(); I have to admit that I'd rather keep lambda for these (and use named functions for longer blocks) than introduce an anonymous block syntax that can return values! I also note that if you *already* have a comparison function, Ruby's Array sort method doesn't let you pass it in as a function argument; you have to give it a block that calls the comparison function, because blocks are not the same as callables (and I'm not sure that Ruby even *has* callables -- everything seems to be a block). My tentative conclusion remains: Python doesn't need Ruby blocks. Brian Sabbey ought to come up with more examples rather than arguments why his preferred syntax and semantics are best. --Guido van Rossum (home page: http://www.python.org/~guido/)

30 144

2 words keyword for block
by Gheorghe Milas May 3, 2005

May 3, 2005

I'm not really in position to speak but since I only saw people trying to come up with a keyword only using one word and without much success I would venture to suggest the possibility of making a keyword out of two words. Would there be a huge problem to use 2 words to make up a keyword? like for example <in template> or <intemplate> if using space is a real problem in template thread_safe(lock): in template redirected_stdout(stream): in template use_and_close_file(path) as file: in template as_transaction(): in template auto_retry(times=3, failas=IOError):

2 1

Re: [Python-Dev] PEP 340 -- concept clarification
by Delaney, Timothy C (Timothy) May 3, 2005

May 3, 2005

Delaney, Timothy C (Timothy) wrote: > Guido van Rossum wrote: > >> I'd like the block statement to be defined exclusively in terms of >> __exit__() though. > > 1. If an iterator declares __exit__, it cannot be used in a for-loop. > For-loops do not guarantee resource cleanup. > > 2. If an iterator does not declare __exit__, it cannot be used in a > block-statement. > Block-statements guarantee resource cleanup. Now some thoughts have solidified in my mind ... I'd like to define some terminology that may be useful. resource protocol: __next__ __exit__ Note: __iter__ is explicitly *not* required. resource: An object that conforms to the resource protocol. resource generator: A generator function that produces a resource. resource usage statement/suite: A suite that uses a resource. With this conceptual framework, I think the following makes sense: - Keyword 'resource' for defining a resource generator. - Keyword 'use' for using a resource. e.g. :: resource locker (lock): lock.acquire() try: yield finally: lock.release() use locker(lock): # do stuff Tim Delaney

1 0

Re: [Python-Dev] PEP 340 -- loose ends
by Delaney, Timothy C (Timothy) May 3, 2005

May 3, 2005

Another loose end (which can partially explain why I still thought __next__ took an exception ;) In "Specification: Generator Exit Handling":: "When __next__() is called with an argument that is not None, the yield-expression that it resumes will return the value attribute of the argument." I think this should read:: "When __next__() is called with an argument that is not None, the yield-expression that it resumes will return the argument." Tim Delaney

1 0

PEP 340 -- concept clarification
by Jim Jewett May 3, 2005

May 3, 2005

[Raymond Hettinger] >> Likewise, is it correct that "yield" is anti-parallel to the current >> meaning? Inside a generator, it returns control upwards to the caller. >> But inside a block-iterator, it pushes control downwards (?inwards) to >> the block it controls. Guido: > I have a hard time visualizing the difference. In a normal generator, someone makes a call to establish the generator, which then becomes a little island -- anyone can call the generator, and it returns control back to whoever made the last call. With the block, every yield returns to a single designated callback. This callback had to be established at the same time the block was created, and must be textually inside it. (An indented suite to the "block XXX:" line.) >> Are there some good use cases that do not involve resource locking? > Decorators don't need @synchronized as a motivating use case; > there are plenty of other use cases. But are there plenty of other use cases for PEP 340? If not, then why do we need PEP 340? Are decorators not strong enough, or is it just that people aren't comfortable yet? If it is a matter of comfort or recipies, then the new construct might have just as much trouble. (So this one is not a loop, and you can tell the difference because ... uh, just skip that advanced stuff.) > Anyway, @synchronized was mostly a demonstration toy; whole > method calls are rarely the right granularity of locking. That is an important difference -- though I'm not sure that the critical part *shouldn't* be broken out into a separate method. >> I've scanned through the code base looking for some places >> to apply the idea and have come up empty handed. > I presume you mentally discarded the resource allocation use > cases where the try/finally statement was the outermost statement > in the function body, since those would be helped by @synchronized; > but look more closely at Queue, and you'll find that the two such > methods use different locks! qsize, empty, and full could be done with a lockself decorator. Effectively, they *are* lockself decorators for the _xxx functions that subclasses are told to override. If you're talking about put and get, decorators don't help as much, but I'm not sure blocks are much better. You can't replace the outermost try ... finally with a common decorator because the locks are self variables. A block, by being inside a method, would be delayed until self exists -- but that outer lock is only a tiny fraction of the boilerplate. It doesn't help with if not block: if self._STATE(): raise STATEException elif timeout is None: while self._STATE(): self.not_STATE.wait() else: if timeout < 0: raise ValueError("'timeout' must be a positive number") endtime = _time() + timeout while self._STATE(): remaining = endtime - _time() if remaining <= 0.0: raise STATEException self.not_STATE.wait(remaining) val = self._RealMethod(item) # OK, the put optimizes out this and the return self.not_OTHERSTATE.notify() return val I wouldn't object to a helper method, but using a block just to get rid of four lines (two of which are the literals "try:" and "finally:") seems barely worth doing, let alone with special new syntax. > Also the use case for closing a file upon leaving a block, while > clearly a resource allocation use case, doesn't work well with a > decorator. def autoclose(fn): def outer(filename, *args, **kwargs): f = open(filename) val = fn(f, *args, **kwargs) f.close() return val return outer @autoclose def f1(f): for line in f: print line > I just came across another use case that is fairly common in the > standard library: redirecting sys.stdout. This is just a beauty (in > fact I'll add it to the PEP): > def saving_stdout(f): > save_stdout = sys.stdout > try: > sys.stdout = f > yield > finally: > sys.stdout = save_stdout Why does this need a yield? Why not just a regular call to the function? If you're trying to generalize the redirector, then this also works as a decorator. The nested functions (and the *args, **kwargs, if you don't inherit from a standard dedcorator) is a bit of an annoyance, but I'm not sure the new iterator form will be any easier to explain. def saving_stdout(f): import sys # Just in case... def captured_stream(fn): def redirect(*args, **kwargs): save_stdout = sys.stdout try: sys.stdout = f return fn (*args, **kwargs) finally: sys.stdout = save_stdout return redirect return captured_stream o=StringIO() @saving_stdout(o) ...

3 4

Re: [Python-Dev] PEP 340 -- loose ends
by Delaney, Timothy C (Timothy) May 3, 2005

May 3, 2005

Guido van Rossum wrote: > Oops. Read the most recent version of the PEP again. __next__() > doesn't take an exception argument, it only takes a value. Maybe this > removes your concern? Actually, I misinterpreted it, assuming that the value passed in was an exception instance because the previous versions worked that way. This has been going on too long ;) > Ah, here we see the other misconception (caused by not reading the > most recent version of the PEP). __exit__() shouldn't call __next__() > -- it should just raise the exception passed in unless it has > something special to do. Ah - I think this needs to be explained better. In particular, in the specification of the __next__ and __exit__ methods it should state what exceptions are expected to be raised under what circumstances - in particular, that __exit__ is expected to raise the passed in exception or StopIteration. This is only explained in the Generator Exception Handling specification, but it's applicable to all iterators. >> Finally, I think there is another loose end that hasn't been >> addressed:: >> >> When __next__() is called with an argument that is not None, the >> yield-expression that it resumes will return the value attribute >> of the argument. If it resumes a yield-statement, the value is >> ignored (or should this be considered an error?). When the >> *initial* call to __next__() receives an argument that is not >> None, the generator's execution is started normally; the >> argument's value attribute is ignored (or should this be >> considered an error?). When __next__() is called without an >> argument or with None as argument, and a yield-expression is >> resumed, the yield-expression returns None. > > Good catch. > >> My opinion is that each of these should be an error. > > Personally, I think not using the value passed into __next__() should > not be an error; that's about the same as not using the value returned > by a function you call. Now that I understand that the parameter to __next__ is not an exception, I agree. > I agree that calling the initial __next__() of a generator with a > non-None argument should be considered an error; this is likely caused > by some kind of logic error; it can never happen when the generator is > called by a block statement. Cheers. Tim Delaney

1 0

Weekly Python Patch/Bug Summary
by Kurt B. Kaiser May 3, 2005

May 3, 2005

Patch / Bug Summary ___________________ Patches : 322 open ( +6) / 2832 closed ( +1) / 3154 total ( +7) Bugs : 920 open (+12) / 4952 closed (+11) / 5872 total (+23) RFE : 186 open ( +8) / 156 closed ( +3) / 342 total (+11) New / Reopened Patches ______________________ Info Associated with Merge to AST (2005-01-07) http://python.org/sf/1097671 reopened by kbk Minimal cleanup of run.py (2005-04-26) http://python.org/sf/1190163 opened by Michiel de Hoon socketmodule.c's recvfrom on OSF/1 4.0 (2005-04-27) http://python.org/sf/1191065 opened by Marcel Martin Simplify logic in random.py (2005-04-28) CLOSED http://python.org/sf/1191489 opened by Raymond Hettinger wrong offsets in bpprint() (2005-04-28) http://python.org/sf/1191700 opened by Pechkinzzz about shift key (2005-04-28) http://python.org/sf/1191726 opened by yang debugger ``condition`` and ``ignore`` exception handling (2005-04-29) http://python.org/sf/1192590 opened by Jeremy Jones Use GCC4 ELF symbol visibility (2005-04-30) http://python.org/sf/1192789 opened by James Henstridge Patches Closed ______________ type conversion methods and subclasses (2005-01-25) http://python.org/sf/1109424 closed by bcannon Don't assume all exceptions are SyntaxError's (2005-04-24) http://python.org/sf/1189210 closed by bcannon Automatically build fpectl module from setup.py (2005-04-19) http://python.org/sf/1185529 closed by mwh Simplify logic in random.py (2005-04-28) http://python.org/sf/1191489 closed by rhettinger New / Reopened Bugs ___________________ [AST] assignment to None allowed (2005-04-25) CLOSED http://python.org/sf/1190010 opened by Brett Cannon [AST] distinct code objects not created (2005-04-25) http://python.org/sf/1190011 opened by Brett Cannon ``from sys import stdin,`` doesn't raise a SyntaxError (2005-04-25) http://python.org/sf/1190012 opened by Brett Cannon 3.29 site is confusing re site-packages on Windows (2005-04-26) http://python.org/sf/1190204 opened by Kent Johnson 6.9 First sentence is confusing (2005-04-26) CLOSED http://python.org/sf/1190451 opened by Nicolas Grilly os.waitpid docs don't specify return value for WNOHANG (2005-04-26) http://python.org/sf/1190563 opened by jls SimpleHTTPServer sends wrong c-length and locks up client (2005-04-26) http://python.org/sf/1190580 opened by Alexander Schremmer calendar._firstweekday is too hard-wired (2005-04-26) http://python.org/sf/1190596 opened by Tres Seaver dir() docs show incorrect output (2005-04-26) CLOSED http://python.org/sf/1190599 opened by Martin Chase bz2 RuntimeError when decompressing file (2005-04-27) http://python.org/sf/1191043 opened by Chris AtLee Warning ``error`` filter action is ignored. (2005-04-27) CLOSED http://python.org/sf/1191104 opened by Ivan Vilata i Balaguer [AST] Failing tests (2005-04-27) http://python.org/sf/1191458 opened by Brett Cannon 'clear -1' in pdb (2005-04-29) http://python.org/sf/1192315 opened by Pechkinzzz doctest's ELLIPSIS and multiline statements (2005-04-29) CLOSED http://python.org/sf/1192554 opened by Sébastien Boisgérault docstring error (2005-04-29) CLOSED http://python.org/sf/1192777 opened by Christopher Smith Notation (2005-04-30) http://python.org/sf/1193001 opened by Mythril Python and Turkish Locale (2005-04-30) http://python.org/sf/1193061 opened by S.Çağlar Onur Embedded python thread crashes (2005-04-30) http://python.org/sf/1193099 opened by ugodiggi Strange os.path.exists() results with invalid chars (2005-04-30) http://python.org/sf/1193180 opened by Daniele Varrazzo MACOSX_DEPLOYMENT_TARGET checked incorrectly (2005-04-30) http://python.org/sf/1193190 opened by Bob Ippolito os.path.expanduser documentation wrt. empty $HOME (2005-05-02) http://python.org/sf/1193849 opened by Wummel calendar.weekheader not found in __all__ (2005-05-03) http://python.org/sf/1193890 opened by George Yoshida Weakref types documentation bugs (2005-05-02) http://python.org/sf/1193966 opened by Barry A. Warsaw bz2.BZ2File doesn't handle modes correctly (2005-05-02) http://python.org/sf/1194181 opened by Bob Ippolito Error in section 4.2 of Python Tutorial (2005-05-03) http://python.org/sf/1194209 opened by Andrina Kelly Bugs Closed ___________ [ast branch] fatal error when compiling test_bool.py (2005-03-19) http://python.org/sf/1166714 closed by bcannon [AST] assert failure on ``eval("u'\Ufffffffe'")`` (2005-04-19) http://python.org/sf/1186345 closed by bcannon "Atuple containing default argument values ..." (2005-04-25) http://python.org/sf/1189819 closed by rhettinger [AST] assignment to None allowed (2005-04-25) http://python.org/sf/1190010 closed by bcannon 6.9 First sentence is confusing (2005-04-26) http://python.org/sf/1190451 closed by rhettinger Python 2.4 Not Recognized by Any Programs (2005-04-23) http://python.org/sf/1188637 closed by tjreedy Variable.__init__ uses self.set(), blocking specialization (2005-04-07) http://python.org/sf/1178872 closed by tjreedy Compiler generates relative filenames (2001-04-11) http://python.org/sf/415492 closed by tjreedy smtplib crashes Windows Kernal. (2003-06-09) http://python.org/sf/751612 closed by tjreedy AssertionError from urllib.retrieve / httplib (2003-06-15) http://python.org/sf/755080 closed by tjreedy dir() docs show incorrect output (2005-04-26) http://python.org/sf/1190599 closed by mwh Warning ``error`` filter action is ignored. (2005-04-27) http://python.org/sf/1191104 closed by vsajip doctest's ELLIPSIS and multiline statements (2005-04-29) http://python.org/sf/1192554 closed by boisgerault docstring error (2005-04-29) http://python.org/sf/1192777 closed by bcannon New / Reopened RFE __________________ The array module and the buffer interface (2005-04-25) http://python.org/sf/1190033 opened by Josiah Carlson logging module '.' behavior (2005-04-26) http://python.org/sf/1190689 opened by Christopher Dunn Add 'before' and 'after' methods to Strings (2005-04-26) CLOSED http://python.org/sf/1190701 opened by Christopher Dunn cStringIO has reset(), but StringIO does not (2005-04-27) CLOSED http://python.org/sf/1191420 opened by Christopher Dunn logging module root logger name (2005-04-26) http://python.org/sf/1190689 reopened by cxdunn slice indices different than integers (2005-04-28) CLOSED http://python.org/sf/1191697 opened by Sebastien de Menten make slices pickable (2005-04-28) http://python.org/sf/1191699 opened by Sebastien de Menten asynchronous Subprocess (2005-04-28) http://python.org/sf/1191964 opened by Josiah Carlson "replace" function should accept lists. (2005-04-17) http://python.org/sf/1184678 reopened by poromenos 'str'.translate(None) => identity translation (2005-04-30) http://python.org/sf/1193128 opened by Bengt Richter add server.shutdown() method and daemon arg to SocketServer (2005-05-02) http://python.org/sf/1193577 opened by paul rubin Expat Parser to supply document locator in incremental parse (2005-05-02) http://python.org/sf/1193610 opened by GaryD RFE Closed __________ Add 'before' and 'after' methods to Strings (2005-04-26) http://python.org/sf/1190701 closed by rhettinger cStringIO has reset(), but StringIO does not (2005-04-27) http://python.org/sf/1191420 closed by rhettinger logging module root logger name (2005-04-27) http://python.org/sf/1190689 closed by vsajip logging module documentation (2003-01-16) http://python.org/sf/668905 closed by vsajip slice indices different than integers (2005-04-28) http://python.org/sf/1191697 closed by mwh

1 0

Re: [Python-Dev] PEP 340 -- loose ends
by Delaney, Timothy C (Timothy) May 3, 2005

May 3, 2005

Guido van Rossum wrote: > [Delaney, Timothy] >> PEP 340 does not address "normal" iterators very well, but a >> properly-constructed iterator will behave correctly. > > This is by design. Yep - I agree. >> The PEP though is very generator-focussed. > > Disagree. The PEP describes most everything (e.g. the block statement > semantics) in terms of iterators, and then describes how the new APIs > behave for generators. Again, agree. What I meant is that there are no examples of how to actually implement the correct semantics for a normal iterator. Doing it right is non-trivial, especially with the __next__ and __exit__ interaction (see below). >> The issues I see for "normal" >> iterators (and that need to be addressed/stated in the PEP) are: >> >> 1. No automatic handling of parameters passed to __next__ and >> __exit__. In a generator, these will raise at the yield >> -statement or -expression. A "normal" iterator will have >> to take care of this manually. > > Not sure what you mean by this. If __next__() is defined, it is passed > the parameter; if only next() is defined, a parameter (except None) is > an error. That seems exactly right. Also, if __exit__() isn't defined, > the exception is raised, which is a very sensible default behavior > (and also what will happen to a generator that doesn't catch the > exception). What I meant is how the iterator is meant to handle the parameters passed to each method. PEP 340 deals with this by stating that exceptions will be raised at the next yield-statement or -expression. I think we need an example though of how this would translate to a "normal" iterator. Something along the lines of:: class iterator (object): def next (self): return self.__next__() def __next__(self, arg=None): value = None if isinstance(arg, ContinueIteration): value = arg.value elif arg is not None: raise arg if value is None: raise StopIteration return value def __exit__(self, type=None, value=None, traceback=None): if (type is None) and (value is None) and (traceback is None): type, value, traceback = sys.exc_info() if type is not None: try: raise type, value, traceback except type, exc: return self.__next__(exc) return self.__next__() >> As another option, it might be worthwhile creating a base iterator type >> with "correct" semantics. > Well, what would the "correct" semantics be? What would passing a > parameter to a list iterator's __next__() method mean? Sorry - I meant for user-defined iterators. And the correct semantics would be something like the example above I think. Except that I think most of it would need to be in a separate method (e.g. _next) for base classes to call - then things would change to be something like:: class iterator (object): ... def _next (self, arg): if isinstance(arg, ContinueIteration): return arg.value elif arg is not None: raise arg def __next__(self, arg=None): value = self._next(arg) if value is None: raise StopIteration return value ... Finally, I think there is another loose end that hasn't been addressed:: When __next__() is called with an argument that is not None, the yield-expression that it resumes will return the value attribute of the argument. If it resumes a yield-statement, the value is ignored (or should this be considered an error?). When the *initial* call to __next__() receives an argument that is not None, the generator's execution is started normally; the argument's value attribute is ignored (or should this be considered an error?). When __next__() is called without an argument or with None as argument, and a yield-expression is resumed, the yield-expression returns None. My opinion is that each of these should be an error. Tim Delaney

2 1

Re: [Python-Dev] PEP 340 -- loose ends
by Delaney, Timothy C (Timothy) May 2, 2005

May 2, 2005

Phillip J. Eby wrote: > Specifically, I propose that PEP 340 *not* allow the use of "normal" > iterators. Instead, the __next__ and __exit__ methods would be an > unrelated protocol. This would eliminate the need for a 'next()' > builtin, > and avoid any confusion between today's iterators and a template > function > for use with blocks. PEP 340 does not address "normal" iterators very well, but a properly-constructed iterator will behave correctly. The PEP though is very generator-focussed. The issues I see for "normal" iterators (and that need to be addressed/stated in the PEP) are: 1. No automatic handling of parameters passed to __next__ and __exit__. In a generator, these will raise at the yield-statement or -expression. A "normal" iterator will have to take care of this manually. This could be an argument to only allow generator-iterators to be used with PEP 340 semantics (i.e. continue <EXPR>, block), but I don't think it's a very compelling one. Although perhaps the initial implementation could be restricted to generator-iterators. So if a for-loop used `continue <EXPR>` it would have a check (at the start of the for loop) that the iterator is a generator-iterator. Likewise, a block-statement would always include this check. As another option, it might be worthwhile creating a base iterator type with "correct" semantics. Tim Delaney

2 1

Generating nested data structures with blocks
by Walter Dörwald May 2, 2005

May 2, 2005

Reading PEP 340, it seems to me that blocks could be used for generating nested data structures: def blist(list): def enter(parent=None): if parent: parent.append(self) yield self x = blist() block x.enter() as x: x.append(1) block blist().enter(x) as x: x.append(2) x.append(3) print x this should print [1, [2], 3] For this to work, the scope of the block variable has to end with the end of the block. Currently the PEP leaves this unspecified. Bye, Walter Dörwald

1 1