For the record, I very much want PEPs 342 and 343 implemented. I haven't had the time to look at the patch and don't expect to find the time any time soon, but it's not for lack of desire to see this feature implemented. I don't like Jason's __with__ proposal and even less like his idea to drop __enter__ and __exit__ (I think this would just make it harder to provide efficient implementations in C). I'm all for adding __enter__ and __exit__ to locks. I'm even considering that it might be a good idea to add them to files. For the record, here at Elemental we write a lot of Java code that uses database connections in a pattern that would have greatly benefited from a similar construct in Java. :) --Guido On 10/3/05, Michael Hudson <mwh@python.net> wrote:

"Phillip J. Eby" <pje@telecommunity.com> writes:

...

Since the PEP is accepted and has patches for both its implementation and a good part of its documentation, a major change like this would certainly need a better rationale.

Though given the amount of interest said patch has attracted (none at all) perhaps noone cares very much and the proposal should be dropped. Which would be a shame given the time I spent on it and all the hot air here on python-dev...

Cheers, mwh (who still likes PEP 343 and doesn't particularly like Jason's suggested changes).

-- Gevalia is undrinkable low-octane see-through only slightly roasted bilge water. Compared to .us coffee it is quite drinkable. -- Måns Nilsson, asr _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (home page: http://www.python.org/~guido/)

"Phillip J. Eby" <pje@telecommunity.com> writes:

At 07:02 PM 10/3/2005 +0100, Michael Hudson wrote:

...

"Phillip J. Eby" <pje@telecommunity.com> writes:

...

Since the PEP is accepted and has patches for both its implementation and a good part of its documentation, a major change like this would certainly need a better rationale.

Though given the amount of interest said patch has attracted (none at all)

Actually, I have been reading the patch and meant to comment on it.

Oh, good.

I was perplexed by the odd stack behavior of the new opcode until I realized that it's try/finally that's weird. :)

:)

I was planning to look into whether that could be cleaned up as well, when I got distracted and didn't go back to it.

I see. I don't know whether trying to clean up the stack protocol around exceptions is worth the about of pain it causes in the head (anyone still thinking about removing the block stack?).

...

perhaps noone cares very much and the proposal should be dropped.

I care an awful lot, as 'with' is another framework-dissolving tool that makes it possible to do more things in library form, without needing to resort to template methods. It also enables more context-sensitive programming, in that "global" states can be set and restored in a structured fashion. It may take a while to feel the effects, but it's going to be a big improvement to Python, maybe as big as new-style classes, and certainly bigger than decorators.

I think 'as big as new-style classes' is probably an exaggeration, but I'm glad my troll caught a few people :) Cheers, mwh -- Those who have deviant punctuation desires should take care of their own perverted needs. -- Erik Naggum, comp.lang.lisp

On 10/4/05, Nick Coghlan <ncoghlan@gmail.com> wrote:

I was planning on looking at your patch too, but I was waiting for an answer from Guido about the fate of the ast-branch for Python 2.5. Given that we have patches for PEP 342 and PEP 343 against the trunk, but ast-branch still isn't even passing the Python 2.4 test suite, I'm wondering if it should be bumped from the feature list again.

What do you want me to say about the AST branch? It's not my branch, I haven't even checked it out, I'm just patiently waiting for the folks who started it to finally finish it. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On 10/5/05, Nick Coghlan <ncoghlan@gmail.com> wrote:

Anyway, the question is: What do we want to do with ast-branch? Finish bringing it up to Python 2.4 equivalence, make it the HEAD, and only then implement the approved PEP's (308, 342, 343) that affect the compiler? Or implement the approved PEP's on the HEAD, and move the goalposts for ast-branch to include those features as well?

I believe the latter is the safe option in terms of making sure 2.5 is a solid release, but doing it that way suggests to me that the ast compiler would need to be held over until 2.6, which would be somewhat unfortunate.

Given that I don't particularly like that answer, I'd love for someone to convince me I'm wrong ;)

Given the total lack of response, I have a different suggestion. Let's *abandon* the AST-branch. We're fooling ourselves believing that we can ever switch to that branch, no matter how theoretically better it is. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

[Brett]

To answer Nick's email here, I didn't respond to that initial email because it seemed specifically directed at Guido and not me.

Fair enough. I think I was actually misrembering the sequence of events leading up to 2.4a1, so the question was less appropriate for Guido than I thought :) [Guido]

On 10/5/05, Guido van Rossum <guido@python.org> wrote:

...

Given the total lack of response, I have a different suggestion. Let's *abandon* the AST-branch. We're fooling ourselves believing that we can ever switch to that branch, no matter how theoretically better it is.

[Brett]

Since the original people who have done the majority of the work (Jeremy, Tim, Neal, Nick, logistix, and myself) have fallen so far behind this probably is not a bad decision. Obviously I would like to see the work pan out, but since I personally just have not found the time to shuttle the branch the rest of the way I really am in no position to say much in terms of objecting to its demise.

If we kill the branch for now, then anyone that wants to bring up the idea again can write a PEP first, not only to articulate the benefits of switching to an AST compiler (Jeremy has a few notes scattered around the web on that front), but also to propose a solid migration strategy. We tried the "develop in parallel, switch when done"; it doesn't seem to have worked due to the way it split developer effort between the branches, and both the HEAD and ast-branch ended up losing out.

Maybe I can come up with a new design and get my dissertation out of it. =)

A strategy that may work out better is to develop something independent of the Python core that can: 1. Produce an ASDL based AST structure from: - Python source code - CPython 'AST' - CPython bytecode 2. Parse an ASDL based AST structure and produce: - Python source code - CPython 'AST' - CPython bytecode That is, initially develop an enhanced replacement for the compiler package, rather than aiming directly to replace the actual CPython compiler. Then the folks who want to do serious bytecode hacking can reverse compile the bytecode on the fly ;) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com

On 10/6/05, Neil Schemenauer <nas@arctrix.com> wrote:

Nick Coghlan <ncoghlan@gmail.com> wrote:

...

If we kill the branch for now, then anyone that wants to bring up the idea again can write a PEP first

I still have some (very) small hope that it can be finished. If we don't get it done soon then I fear that it will never happen. I had hoped that a SoC student would pick up the task or someone would ask for a grant from the PSF. Oh well.

...

A strategy that may work out better is [...]

Another thought I've had recently is that most of the complexity seems to be in the CST to AST translator. Perhaps having a parser that provided a nicer CST might help.

Dream on, Neil... Adding more work won't make it more likely to happen. The only alternative to abandoning it that I see is to merge it back into main NOW, using the time that remains us until the 2.5 release to make it robust. That way, everybody can help out (and it may motivate more people). Even if this is a temporary regression (e.g. PEP 342), it might be worth it -- but only if there are at least two people committed to help out quickly when there are problems. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Jeremy Hylton <jeremy@alum.mit.edu> writes:

On 10/6/05, Guido van Rossum <guido@python.org> wrote:

...

The only alternative to abandoning it that I see is to merge it back into main NOW, using the time that remains us until the 2.5 release to make it robust. That way, everybody can help out (and it may motivate more people).

Even if this is a temporary regression (e.g. PEP 342), it might be worth it -- but only if there are at least two people committed to help out quickly when there are problems.

I'm sorry I didn't respond earlier. I've been home with a new baby for the last six weeks and haven't been keeping a close eye on my email. (I didn't see Nick's earlier email until his most recent post.)

It would take a few days of work to get the branch ready to merge to the head. There are basic issues like renaming newcompile.c to compile.c and the like. I could work on that tomorrow and Monday.

Unless I'm missing something, we would need to merge HEAD to the AST branch once more to pick up the changes in MAIN since the last merge, and then make sure everything in the AST branch is passing the test suite. Otherwise we risk having MAIN broken for awhile following a merge. Finally, we can then merge the diff of HEAD to AST back into MAIN. If we try to merge the entire AST branch since its inception, we will re-apply to MAIN those changes made in MAIN which have already been merged to the AST branch and it will be difficult to sort out all the conflicts. If we try to merge the AST branch from the its last merge tag to its head we will miss the work done on AST prior to that merge. Let me know at kbk@shore.net if you want to do this. -- KBK

[Kurt]

...

Unless I'm missing something, we would need to merge HEAD to the AST branch once more to pick up the changes in MAIN since the last merge, and then make sure everything in the AST branch is passing the test suite. Otherwise we risk having MAIN broken for awhile following a merge.

[Raymond]

IMO, merging to the head is a somewhat dangerous strategy that doesn't have any benefits. Whether done on the head or in the branch, the same amount of work needs to be done.

If the stability of the head is disrupted, it may impede other maintenance efforts because it is harder to test bug fixes when the test suites are not passing.

Well, at some point it will HAVE to be merged into the head. The longer we wait the more painful it will be. If we suffer a week of instability now, I think that's acceptable, as long as all developers are suitably alerted, and as long as the AST team works towards resolving the issues ASAP. I happen to agree with Kurt that we should first merge the head into the branch; then the AST team can work on making sure the entire test suite passes; then they can merge back into the head. BUT this should only be done with a serious commitment from the AST team (I think Neil and Jeremy are offering this -- I just don't know how much time they will have available, realistically). My main point is, we should EITHER abandon the AST branch, OR force a quick resolution. I'm willing to suffer a week of instability in head now, or in a week or two -- but I'm not willing to wait again. Let's draw a line in the sand. The AST team (which includes whoever will help) has up to three weeks to het the AST branch into a position where it passes all the current unit tests merged in from the head. Then they merge it into the head after which we can accept at most a week of instability in the head. After that the AST team must remain available to resolve remaining issues quickly. How does this sound to the non-AST-branch developers who have to suffer the inevitable post-merge instability? I think it's now or never -- waiting longer isn't going to make this thing easier (not with several more language changes approved: with-statement, extended import, what else...) What does the AST team think? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido van Rossum <guido@python.org> writes:

I happen to agree with Kurt that we should first merge the head into the branch; then the AST team can work on making sure the entire test suite passes; then they can merge back into the head.

I can be available to do this again. It would involve freezing the AST branch for a day. Once the AST branch is stable, we would need to freeze everything, merge MAIN to AST one more time to pick up the last few changes in MAIN, and then merge the AST head back to MAIN. By doing these merges from MAIN to AST we would have effectively moved the AST branch point along MAIN to HEAD. So the final join is HEAD to AST, conducted from MAIN. I'll run a local experiment to verify this concept is workable. -- KBK

At 09:50 AM 10/4/2005 +0100, Michael Hudson wrote:

(anyone still thinking about removing the block stack?).

I'm not any more. My thought was that it would be good for performance, by reducing the memory allocation overhead for frames enough to allow pymalloc to be used instead of the platform malloc. After more investigation, however, I realized that was a dumb idea, because for a typical application the amortized allocation cost of frames approaches zero as the program runs and allocates as many frames as it will ever use, as large as it will ever use them, and just recycles them on the free list. And all of the ways I came up with for removing the block stack were a lot more complex than leaving it as-is. Clearly, the cost of function calls in Python lies somewhere else, and I'd probably look next at parameter tuple allocation, and other frame initialization activities. I seem to recall that Armin Rigo once supplied a patch that sped up calls at the cost of slowing down recursive or re-entrant ones, and I seem to recall that it was based on preinitializing frames, not just preallocating them: http://mail.python.org/pipermail/python-dev/2004-March/042871.html However, the patch was never applied because of its increased memory usage as well as the slowdown for recursion. Every so often, in blue-sky thinking about alternative Python VM designs, I think about making frames virtual, in the sense of not even having "real" frame objects except for generators, sys._getframe(), and tracebacks. I suspect, however, that doing this in a way that doesn't mess with the current C API is non-trivial. And for many "obvious" ways to simplify the various stacks, locals, etc., the downside could be more complexity for generators, and probably less speed as well. For example, we could use a single "stack" arena in the heap for parameters, locals, cells, and blocks, rather than doing all the various sub-allocations within the frame. But then creating a frame would involve copying data off the top of this pseudo-stack, and doing all the offset computations and perhaps some other trickery as well. And resuming a generator would have to either copy it back, or have some sane way to make calls out to a new stack arena when calling other functions - thus making those operations slower. The real problem, of course, with any of these ideas is that we are at best shaving a few percentage points here, a few points there, so it's comparatively speaking rather expensive to do the experiments to see if they help anything.

Neal Norwitz wrote:

My thoughts are to dynamically allocate the Python stack memory (e.g., void *stack = malloc(128MB)). Then all calls within each thread uses its own stack. So things would be pushed onto the stack like they are currently, but we wouldn't need to do create a tuple to pass to a method, they could just be used directly. Basically more closely simulate the way it currently works in hardware.

One issue with argument tuples on the stack (or some sort of stack) is that functions may hold onto argument tuples longer: def foo(*args): global last_args last_args = args I considered making true tuple objects (i.e. with ob_type etc.) on the stack, but this possibility breaks it. Regards, Martin

Phillip J. Eby wrote:

Clearly, the cost of function calls in Python lies somewhere else, and I'd probably look next at parameter tuple allocation,

For simple calls where there aren't any *args or other such complications, it seems like it should be possible to just copy the args from the calling frame straight into the called one. Or is this already done these days? -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg.ewing@canterbury.ac.nz +--------------------------------------+

The argument I am going to try to make is that Python coroutines need a more usable API.

Try to explain the semantics of the with statement without referring to the __enter__ and __exit__ methods, and then see if you still think they're superfluous ;)

The @contextmanager generator decorator is just syntactic sugar [...] [T]he semantics of the with statement itself can only be explained in terms of the __enter__ and __exit__ methods.

That's not true. It can certainly use the coroutine API instead. Now... as specified in PEP 342, the coroutine API can be used to implement 'with', but it's ugly. I think this is a problem with the coroutine API, not the idea of using coroutines per se. Actually I think 'with' is a pretty tame use case for coroutines. Other Python objects (dicts, lists, strings) have convenience methods that are strictly redundant but make them much easier to use. Coroutines should, too. This: with EXPR as VAR: BLOCK expands to this under PEP 342: _cm = contextmanager(EXPR) VAR = _cm.next() try: BLOCK except: try: _cm.throw(*sys.exc_info()) except: pass raise finally: try: _cm.next() except StopIteration: pass except: raise else: raise RuntimeError Blah. But it could look like this: _cm = (EXPR).__with__() VAR = _cm.start() try: BLOCK except: _cm.throw(*excinfo) else: _cm.finish() I think that looks quite nice. Here is the proposed specification for start() and finish(): class coroutine: # pseudocode ... def start(self): """ Convenience method -- exactly like next(), but assert that this coroutine hasn't already been started. """ if self.__started: raise ValueError # or whatever return self.next() def finish(self): """ Convenience method -- like next(), but expect the coroutine to complete without yielding again. """ try: self.next() except (StopIteration, GeneratorExit): pass else: raise RuntimeError("coroutine didn't finish") Why is this good? - Makes coroutines more usable for everyone, not just for implementing 'with'. - For example, if you want to feed values to a coroutine, call start() first and then send() repeatedly. Quite sensible. - Single mental model for 'with' (always uses a coroutine or lookalike object). - No need for "contextmanager" wrapper. - Harder to implement a context manager object incorrectly (it's quite easy to screw up with __begin__ and __end__). -j

On 10/4/05, Jason Orendorff <jason.orendorff@gmail.com> wrote:

Right after I sent the preceding message I got a funny feeling I'm wasting everybody's time here. I apologize. Guido's original concern about speedy C implementation for locks stands. I don't see a good way around it.

OK. Our messages crossed, so you can ignore my response. Let's spend our time implementing the PEPs as they stand, then see what else we can do with the new APIs. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On 10/4/05, Jason Orendorff <jason.orendorff@gmail.com> wrote:

This:

with EXPR as VAR: BLOCK

expands to this under PEP 342:

_cm = contextmanager(EXPR) VAR = _cm.next() try: BLOCK except: try: _cm.throw(*sys.exc_info()) except: pass raise finally: try: _cm.next() except StopIteration: pass except: raise else: raise RuntimeError

Where in the world do you get this idea? The translation is as follows, according to PEP 343: abc = EXPR exc = (None, None, None) VAR = abc.__enter__() try: try: BLOCK except: exc = sys.exc_info() raise finally: abc.__exit__(*exc) PEP 342 doesn't touch on the expansion of with-statements at all. I think I know where you're coming from, but please do us a favor and don't misrepresent the PEPs. If anything, your proposal is more complicated; it requires four new APIs instead of two, and requires an extra call to set up (__with__() followed by start()). Proposals like yours (and every other permutation) were brought up during the initial discussion. We picked one. Don't create more churn by arguing for a different variant. Spend your efforts on implementing it so you can actually use it and see how bad it is (I predict it won't be bad at all). -- --Guido van Rossum (home page: http://www.python.org/~guido/)