Mailman 3 April 2005 - Python-Dev

next(arg) was: Anonymous blocks: Thunks or iterators?
by Jim Jewett April 29, 2005

April 29, 2005

Guido van Rossum: > One [of many separate ideas in PEP 340] is turning generators > into more general coroutines: continue EXPR passes the expression > to the iterator's next() method ... I would have been very happy with that a week ago. Seeing the specific implementation changed my mind. The caller shouldn't know what state the generator is in, so the passed-in-message will be the same regardless of which yield accepts it. Unless I have a single-yield generator, this means I end up writing boilerplate code to accept and process the arg at each yield. I don't want more boilerplate. > Even without a block-statement, these two changes make yield look a > lot like invoking a thunk Though it feels backwards to me; yield is returning control to something that already had to coordinate the thunks itself. -jJ

2 1

PEP 340: What is "ret" in block statement semantics?
by Skip Montanaro April 29, 2005

April 29, 2005

me> It uses a variable "ret" that is always False. Gaack. Please ignore. Skip

1 0

PEP 340: What is "ret" in block statement semantics?
by Skip Montanaro April 29, 2005

April 29, 2005

PEP 340 describes the block statement translation as: itr = EXPR1 val = arg = None ret = False while True: try: VAR1 = next(itr, arg) except StopIteration: if ret: return val if val is not None: raise val break try: val = arg = None ret = False BLOCK1 except Exception, val: arg = StopIteration() It uses a variable "ret" that is always False. If it does manage to take on a True value, a return statement is executed. How does ret become True? What's meaning of return in this context? Something seems amiss. Skip

1 0

About block statement name alternative
by Luis P Caamano April 29, 2005

April 29, 2005

How about "bracket" or "bracket_with"? As in: bracket_with synchronized(lock): BLOCK bracket_with opening("/etc/passwd") as f: for line in f: print line.rstrip() bracket_with transactional(db): db.store() bracket_with auto_retry(3, IOError): f = urllib.urlopen("http://python.org/peps/pep-0340.html") print f.read() block_with synchronized_opening("/etc/passwd", myLock) as f: for line in f: print line.rstrip() def synchronized_opening(lock, filename, mode="r"): bracket_with synchronized(lock): bracket_with opening(filename) as f: yield f bracket_with synchronized_opening("/etc/passwd", myLock) as f: for line in f: print line.rstrip() Or for that matter, "block_with", as in: block_with transactional(db): db.store() -- Luis P Caamano Atlanta, GA USA

1 0

Re: [Python-Dev] Re: anonymous blocks
by Luis P Caamano April 29, 2005

April 29, 2005

On 4/29/05, python-dev-request(a)python.org <python-dev-request(a)python.org> wrote: > > Message: 2 > Date: Thu, 28 Apr 2005 21:56:42 -0600 > From: Shane Hathaway <shane(a)hathawaymix.org> > Subject: Re: [Python-Dev] Re: anonymous blocks > To: guido(a)python.org > Cc: Ka-Ping Yee <python-dev(a)zesty.ca>, Python Developers List > <python-dev(a)python.org> > Message-ID: <4271B07A.4010501(a)hathawaymix.org> > Content-Type: text/plain; charset=ISO-8859-1 > > > I think this concept can be explained clearly. I'd like to try > explaining PEP 340 to someone new to Python but not new to programming. > I'll use the term "block iterator" to refer to the new type of > iterator. This is according to my limited understanding. > > "Good programmers move commonly used code into reusable functions. > Sometimes, however, patterns arise in the structure of the functions > rather than the actual sequence of statements. For example, many > functions acquire a lock, execute some code specific to that function, > and unconditionally release the lock. Repeating the locking code in > every function that uses it is error prone and makes refactoring difficult. > > "Block statements provide a mechanism for encapsulating patterns of > structure. Code inside the block statement runs under the control of an > object called a block iterator. Simple block iterators execute code > before and after the code inside the block statement. Block iterators > also have the opportunity to execute the controlled code more than once > (or not at all), catch exceptions, or receive data from the body of the > block statement. > > "A convenient way to write block iterators is to write a generator. A > generator looks a lot like a Python function, but instead of returning a > value immediately, generators pause their execution at "yield" > statements. When a generator is used as a block iterator, the yield > statement tells the Python interpreter to suspend the block iterator, > execute the block statement body, and resume the block iterator when the > body has executed. > > "The Python interpreter behaves as follows when it encounters a block > statement based on a generator. First, the interpreter instantiates the > generator and begins executing it. The generator does setup work > appropriate to the pattern it encapsulates, such as acquiring a lock, > opening a file, starting a database transaction, or starting a loop. > Then the generator yields execution to the body of the block statement > using a yield statement. When the block statement body completes, > raises an uncaught exception, or sends data back to the generator using > a continue statement, the generator resumes. At this point, the > generator can either clean up and stop or yield again, causing the block > statement body to execute again. When the generator finishes, the > interpreter leaves the block statement." > > Is it understandable so far? > I've been skipping most of the anonymous block discussion and thus, I only had a very vague idea of what it was about until I read this explanation. Yes, it is understandable -- assuming it's correct :-) Mind you though, I'm not new to python and I've been writing system software for 20+ years. -- Luis P Caamano Atlanta, GA USA

2 1

RE: [Python-Dev] Anonymous blocks: Thunks or iterators?
by Robert Brewer April 29, 2005

April 29, 2005

> [Greg Ewing] > > Elegant as the idea behind PEP 340 is, I can't shake > > the feeling that it's an abuse of generators. It seems > > to go to a lot of trouble and complication so you > > can write a generator and pretend it's a function > > taking a block argument. [Guido] > Maybe. You're not the first one saying this and I'm not saying "no" > outright, but I'd like to defend the PEP. > > There are a number of separate ideas that all contribute to PEP 340. > One is turning generators into more general coroutines: continue EXPR > passes the expression to the iterator's next() method (renamed to > __next__() to work around a compatibility issue and because it should > have been called that in the first place), and in a generator this > value can be received as the return value of yield. Incidentally this > makes the generator *syntax* more similar to Ruby (even though Ruby > uses thunks, and consequently uses return instead of continue to pass > a value back). I'd like to have this even if I don't get the block > statement. Completely agree. Maybe we should have PEP 340 push just that, and make a PEP 341 independently for resource-cleanup (which assumes 340)? > [snip] > > The other problem with thunks is that once we think of them as the > anonymous functions they are, we're pretty much forced to say that a > return statement in a thunk returns from the thunk rather than from > the containing function. Doing it any other way would cause major > weirdness when the thunk were to survive its containing function as a > closure (perhaps continuations would help, but I'm not about to go > there :-). > > But then an IMO important use case for the resource cleanup template > pattern is lost. I routinely write code like this: > > def findSomething(self, key, default=None): > self.lock.acquire() > try: > for item in self.elements: > if item.matches(key): > return item > return default > finally: > self.lock.release() > > and I'd be bummed if I couldn't write this as > > def findSomething(self, key, default=None): > block synchronized(self.lock): > for item in self.elements: > if item.matches(key): > return item > return default Okay, you've convinced me. The only way I can think of to get the effect I've been wanting would be to recompile the template function every time that it's executed with a different block. Call it a "Python _re_processor" ;). Although you could memoize the the resultant bytecode, etc., it would still be pretty slow, and you wouldn't be able to alter (rebind) the thunk once you'd entered the caller. Even then, you'd have the cell issues you mentioned, trying to push values from the thunk's original scope. Bah. It's so tempting on the semantic level, but the implementation's a bear. Robert Brewer MIS Amor Ministries fumanchu(a)amor.org

1 0

Integrating PEP 310 with PEP 340
by Nick Coghlan April 29, 2005

April 29, 2005

This is my attempt at a coherent combination of what I like about both proposals (as opposed to my assortment of half-baked attempts scattered through the existing discussion). PEP 340 has many ideas I like: - enhanced yield statements and yield expressions - enhanced continue and break - generator finalisation - 'next' builtin and associated __next__() slot - changes to 'for' loop One restriction I don't like is the limitation to ContinueIteration and StopIteration as arguments to next(). The proposed semantics and conventions for ContinueIteration and StopIteration are fine, but I would like to be able to pass _any_ exception in to the generator, allowing the generator to decide if a given exception justifies halting the iteration. The _major_ part I don't like is that the block statement's semantics are too similar to those of a 'for' loop. I would like to see a new construct that can do things a for loop can't do, and which can be used in _conjunction_ with a for loop, to provide greater power than either construct on their own. PEP 310 forms the basis for a block construct that I _do_ like. The question then becomes whether or not generators can be used to write useful PEP 310 style block managers (I think they can, in a style very similar to that of the looping block construct from PEP 340). Block statement syntax from PEP 340: block EXPR1 [as VAR1]: BLOCK1 Proposed semantics (based on PEP 310, with some ideas stolen from PEP 340): blk_mgr = EXPR1 VAR1 = blk_mgr.__enter__() try: try: BLOCK1 except Exception, exc: blk_mgr.__except__(exc) else: blk_mgr.__else__() finally: blk_mgr.__exit__() 'blk_mgr' is a hidden variable (as per PEP 340). Note that nothing special happens to 'break', 'return' or 'continue' statements with this proposal. Generator methods to support the block manager protocol used by the block statement: def __enter__(self): try: return next(self) except StopIteration: raise RuntimeError("Generator exhausted before block statement") def __except__(self, exc): try: next(self, exc) except StopIteration: pass def __no_except__(self): try: next(self) except StopIteration: pass def __exit__(self): pass Writing simple block managers with this proposal (these should be identical to the equivalent PEP 340 block managers): def opening(name): opened = open(name) try: yield opened finally: opened.close() def logging(logger, name): logger.enter_scope(name) try: try: yield except Exception, exc: logger.log_exception(exc) finally: logger.exit_scope() def transacting(ts): ts.begin() try: yield except: ts.abort() else: ts.commit() Using simple block managers with this proposal (again, identical to PEP 340): block opening(name) as f: pass block logging(logger, name): pass block transacting(ts): pass Obviously, the more interesting block managers are those like auto_retry (which is a loop, and hence an excellent match for PEP 340), and using a single generator in multiple block statements (which PEP 340 doesn't allow at all). I'll try to get to those tomorrow (and if I can't find any good use cases for the latter trick, then this idea can be summarily discarded in favour of PEP 340). Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

4 3

noob question regarding the interpreter
by Jing Su April 29, 2005

April 29, 2005

Hello, I know this is a n00b question, so I apologize ahead of time. I've been taking a look at they python interpreter, trying to understand how it works on the compiled byte-codes. Looking through the sources of the 2.4.1 stable version, it looks like Python/ceval.c is the module that does the main dispatch. However, it looks like a switched interpreter. I just find this surprising because python seems to run pretty fast, and a switched interpreter is usually painfully slow. Is there work to change python into a direct-threaded or even JIT'ed interpreter? Has there been previous discussion on this topic? I'd greatly appreciate any pointers to discussions on this topic. Thus far my google-fu has not turned up fruitful hits. Thanks in advance for any help! -Jing

3 2

anonymous blocks as scope-collapse: detailed proposal
by Jim Jewett April 28, 2005

April 28, 2005

Based on Guido's opinion that caller and callee should both be marked, I have used keywords 'include' and 'chunk'. I therefore call them "Chunks" and "Includers". Examples are based on (1) The common case of a simple resource manager. e.g. http://mail.python.org/pipermail/python-dev/2005-April/052751.html (2) Robert Brewer's Object Relational Mapper http://mail.python.org/pipermail/python-dev/2005-April/052924.html which uses several communicating Chunks in the same Includer, and benefits from Includer inheritance. Note that several cooperating Chunks may use the same name (e.g. old_children) to refer to the same object, even though that object is never mentioned by the Includer. It is possible for the same code object to be both a Chunk and an Includer. Its own included sub-Chunks also share the top Includer's namespace. Chunks and Includers must both be written in pure python, because C frames cannot be easily manipulated. They can of course call or be called (as a unit) by extension modules. I have assumed that Chunks should not take arguments. While arguments are useful ("Which pattern should I match against on this inclusion?"), the same functionality *can* be had by binding a known name in the Includer. When that starts to get awkward, it is a sign that you should be using separate namespaces (and callbacks, or value objects). "self" and "cls" are just random names to a Chunk, though using them for any but the conventional meaning will be as foolhardy as it is in a method. Chunks are limited to statement context, as they do not return a value. Includers must provide a namespace. Therefore a single inclusion will turn the entire nearest enclosing namespace into an Includer. ? Should this be limited to nearest enclosing function or method? I can't think of a good use case for including directly from class definition or module toplevel, except registration. And even then, a metaclass might be better. Includers may only be used in a statement context, as the Chunks must be specified in a following suite. (It would be possible to skip the suite if all Chunk names are already bound, but I'm not sure that is a good habit to encourage -- so initially forbid it.) Chunks are defined without a (), in analogy to parentless classes. They are included (called) with a (), so that they can remain first class objects. Example Usage ============= def withfile(filename, mode='r'): """Close the file as soon we're done. This frees up file handles sooner. This is particularly important under Jython, or if you are using files in cyclic structures.""" openfile = open(filename, mode) try: include fileproc() # keyword 'include' prevents XXX_FAST optimization finally: openfile.close() chunk nullreader: # callee Chunk defined for reuse for line in openfile: pass withfile("testr.txt"): # Is this creation of a new block-starter a problem? fileproc=nullreader # Using an external Chunk object withfile("testw.txt", "w"): chunk fileproc: # Providing an "inline" Chunk openfile.write("Line 1") # If callers must be supported in expression context #fileproc=nullreader #withfile("tests.txt") # Resolve Chunk name from caller's default # binding, which in this case defaults back # to the current globals. # Is this just asking for trouble? class ORM(object): chunk nullchunk: # The extra processing is not always needed. pass begin=pre=post=end=nullchunk # Default to no extra processing def __set__(self, unit, value): include self.begin() if self.coerce: value = self.coerce(unit, value) oldvalue = unit._properties[self.key] if oldvalue != value: include self.pre() unit._properties[self.key] = value include self.post() include self.end() class TriggerORM(ORM): chunk pre: include super(self,TriggerORM).pre() # self was bound by __set__ old_children = self.children() # inject new variable chunk post: include super(self,TriggerORM).post() for child in self.children(): if child not in old_children: # will see pre's binding notify_somebody("New child %s" % child) As Robert Brewer said, > The above is quite ugly written with callbacks (due to > excessive argument passing), and is currently fragile > when overriding __set__ (due to duplicated code). How to Implement ---------------- The Includer cannot know which variables a Chunk will use (or inject), so the namespace must remain a dictionary. This precludes use of the XXX_FAST bytecodes. But as Robert pointed out, avoiding another frame creation/destruction will compensate somewhat. Two new bytecodes will be needed to handle the jump and return to a different bytecode string without setting up or tearing down a new frame. Position in the Includer bytecode will need to be kept in a stack, though it might make sense to use a frame variable instead of the execution stack. With those two exceptions, the Includer and Chunk are both composed entirely of valid statements that can already be compiled to ordinary bytecode. -jJ

1 0

Problem with embedded python - bug?
by Ugo Di Girolamo April 28, 2005

April 28, 2005

I have been having a few more discussions around about this, and I'm starting to think that this is a bug. My take is that, when I call Py_Finalize, the python thread should be shut down gracefully, closing the file and everything. Maybe I'm missing a call to something (?PyEval_FinalizeThreads?) but the docs seem to say that just PyFinalize should be called. The open file seems to be the issue, since if I remove all the references to the file I cannot get the program to crash. I can reproduce the same behavior on two different wxp systems, under python 2.4 and 2.4.1. Ugo -----Original Message----- From: Ugo Di Girolamo Sent: Tuesday, April 26, 2005 2:16 PM To: 'python-dev(a)python.org' Subject: Problem with embedded python I have the following code, that seems to make sense to me. However, it crashes about 1/3 of the times. My platform is Python 2.4.1 on WXP (I tried the release version from the msi and the debug version built by me, both downloaded today to have the latest version). The crash happens while the main thread is in Py_Finalize. I traced the crash to _Py_ForgetReference(op) in object.c at line 1847, where I have op->_ob_prev == NULL. What am I doing wrong? I'm definitely not too sure about the way I'm handling the GIL. Thanks in adv for any suggestion/ comment Cheers and ciao Ugo ////////////////////////// TestPyThreads.py ////////////////////////// #include <windows.h> #include "Python.h" int main() { PyEval_InitThreads(); Py_Initialize(); PyGILState_STATE main_restore_state = PyGILState_UNLOCKED; PyGILState_Release(main_restore_state); // start the thread { PyGILState_STATE state = PyGILState_Ensure(); int trash = PyRun_SimpleString( "import thread\n" "import time\n" "def foo():\n" " f = open('pippo.out', 'w', 0)\n" " i = 0;\n" " while 1:\n" " f.write('%d\\n'%i)\n" " time.sleep(0.01)\n" " i += 1\n" "t = thread.start_new_thread(foo, ())\n" ); PyGILState_Release(state); } // wait 300 ms Sleep(300); PyGILState_Ensure(); Py_Finalize(); return 0; }

1 0