Mailman 3 February 2003 - Python-Dev

RE: [Python-Dev] exec/with thunk-handling proposal
by Moore, Paul Feb. 4, 2003

Feb. 4, 2003

holger krekel wrote: > Michael Hudson wrote: > > holger krekel <pyth(a)devel.trillke.net> writes: > > > I think we can may get away with only a "weak" keyword > > > and allow the aforementioned encapsulation of execution > > > events into an object like this: > > > > > > exec expr [with params]: suite > > > > Gut reaction: ugh! > > then i guess you rather prefer new keywords. I thought that there > is a general reluctance to introduce new keywords *and* many > people dislike 'exec' for its existence. So reusing it > (and we are dealing with execution aspects IMO) makes > some sense to me. I can see your point. Personally, I don't have any great problems with "exec", so I'd like to understand better how your proposal integrates this new use of exec with the existing use. And at some point, it will be necessary to confirm that the parser can be made to distinguish the various uses (it seems to me that extra lookahead will be needed). > > > where the expression is evaluated to return a > > > "thunk" handler with these optional "execution" hooks: > > > > > > def __enter__(self): > > > "before suite start" > > > > > > def __except__(self, type, value, tb): > > > "swallow given exception, reraise if neccessary" > > > > > > def __leave__(self): > > > """upon suite finish (not called if __except__ > > > exists and an exception happened) > > > """ > > > > > > The above "with" parameters (of the form name=expr, comma-separated) > > > are bound in local (or global/nested) *and* handler instance > > > namespace. The 'suite' is what we call "thunk". > > > > > > The above logic allows clean timely finalization for > > > *multiple* ressources: > > > > > > exec autoclose() with f1=open(name1), f2=open(name2, 'w'): > > > for line in f1: > > > ... > > > f2.write(...) > > > > That looks messy. > > as compared to? I think i have seen messier ideas with all kinds > of brackets and double-colons :-) I agree with you here. This looks OK to me in terms of syntax. I'm not so sure of the semantics - that may be the bit Michael thought was "messy". Binding the with parameters in two places feels odd, and in particular injecting the parameter names into the instance namespace is unusual (I know of no other construct in Python which implicitly changes an instance namespace like this). And as far as the local namespace is concerned, consider exec something() with f1=12: suite # Out of the exec now - what is the value of f1??? I assume that the assignment to f1 is still in existence, otherwise you're inventing a new way of defining a scope. But if the binding to f1 does still exist, then that looks odd, because "exec ... with f1=12" *reads* as if f1 is only in scope for the duration of the statement. You seem to lose either way... > > > which would execute as follows > > > > > > a) autoclose() instance is created and stored as the > > > "thunk"-handler > > > > We need a name for these. I've been using "monitor" for a while, but > > I'm not sure it's that apposite. > > So far i'd liked "execution handler" because 'enter/exit/except' are IMHO > execution events which you can hook into. Do you agree with calling > them 'execution events' or how would you call them? The way you've defined things, I see the logic - it's hooks into the execution model. I'm not sure I'd have thought of that way of looking at it for myself, though... > > > b) f1/f2 are stored as attributes on the autoclose instance > > > > > > c) f1/f2 are put into the local/global namespace (and nested ones > > > if rebinding is allowed) > > > > > > d) thunk executes (for line ...) > > > > > > e) autoclose 'leave' hook is called (with or without exception) > > > and is implemented like this: > > > > > > def __leave__(self): > > > for obj in self.__dict__.values(): > > > obj.close() But what if the object was more complex, and needed its own state? How could it tell which were "its own" variables and which had been injected? (Other than by hard coding a list of "its own" variables to ignore, which is utterly horrible). > > > f) thunk handler is removed > > > > "Too much magic!" > > Hmmm. That seems a bit unfair to me as many other proposals didn't care > to elaborate the exact sequence. You probably are refering to the > "namespace interactions" as the other stuff is probably the same > with any other proposal so far. No, I agree with Michael here. Just because other proposals handwave and don't give proper semantics doesn't mean that they don't also have too much magic. In my book "unclear semantics" is a worse accusation than "too much magic". But both are bad :-) > > > Because computing 'f1' may succeed but 'f2' can subsequently > > > fail the assignments *have to* execute within "autoclose" > > > control. Hmm. In the equivalent try...finally construct, the idiom is to open the file *outside* the try statement, so that exceptions when opening the file don't get accidentally caught. You need to address this - particularly in the case of multiple variables. > Would the following be the correct way to achieve this with your patch? > > f1=open(inputfn) > with autoclose(f1): > f2 = open(outputfn, 'w') > with autoclose(f2): > for line in f1: > ... > f2.write(line) > > I think there should be a better solution for multiple ressources. In many ways, I agree with you. I originally wanted multiple expressions, and assignments within the "with" statement. But people argued against them (look back over the thread if you want the details). I honestly don't believe that the with statement will ever please everyone - the best we can hope for is to let it handle the simple cases well, with clean, well-defined semantics with as few surprises as possible, and leave the remaining cases to the existing try statement (or to a more ambitious thunk-based mechanism). There is a real risk of slimming the with statement down to uselessness by this philosophy. But this may simply indicate that the need is perceived rather than real :-) > > > Now on to the usage of the except hook. Nice use cases might be > > > > > > exec retry(maxretry=3, on=IOError): > > > # do network io-stuff > > > or > > > exec skip_on(AttributeError, TypeError): > > > some_object.notify_hook() > > > > > > but i am sure there are more. Exception handling is often > > > ugly when inlined with the code. I think that stating > > > 'exception behaviour' up-front allows to write nice > > > readable constructs. > > > > I am *still* not convinced that an __except__ hook is worth the pain. > > Can you implement those for me? > > Maybe, i haven't looked at your implementation yet and there are > some other projects pending :-) There's no implementation of any form of except hook in existence yet (excluding the IEXEC patch). The key point here is that your explanation of semantics above didn't cover __except__ hooks. I'd like to see __except__ integrated into the semantics you describe, and then with that as a basis I's like to see definitions of retry() and skip_on(). The idea looks interesting, but I'd like to see concrete code. > > yields already can't go in blocks with finally statements, right? > > right. > > > Would you propose calling the __enter__ method each time the generator > > resumed? > > yes, neccessary for locking (as the above example tried to indicate). But not for files. It's arguable that you need __enter__, __exit__, __yield__, and __resume__ hooks for full generality. On the other hand, YAGNI makes more sense to me... > > > If there is interest i can probably modify my patch > > > to allow all of the proposed syntax so that you could > > > play around with it. Working code for *any* of these ideas could only be a good thing. User code which benefits from the new features is also good. The auto-closing file example has been worked to death, and its main effect for me so far has been to make me wonder whether we need *any* of these mechanisms :-) The lock example is better, but still pretty borderline. I'd really, really like to see a substantial, non-toy use case. Actually, my main experience is C/C++ based, where blocks with many memory allocations and the occasional open file is the canonical example. Freeing and closing in reverse order, under all error conditions, is a *pain*. But for this, the C++ "resource acquisition is initialisation" idiom works brilliantly. It's a shame it can't be made to work in Python. Paul.

2 1

Re: native code compiler? (or, OCaml vs. Python)
by Graham Guttocks Feb. 4, 2003

Feb. 4, 2003

Greg Ewing <greg(a)cosc.canterbury.ac.nz> wrote: > >> one glaring weakness is the absence of a high-performance native >> code compiler. Are there any plans to develop this? > > This is asked fairly frequently, and the usual answer is "No, but > you're welcome to volunteer." :-) Really? It surprises me that after 10 years, this isn't something that has been given more priority. Is the problem simply too difficult? > There are some projects attacking parts of the problem, however, > e.g. Psyco. I've looked at Psyco, but it seems treated like an ad-hoc red-headed stepchild. Why isn't something this made part of Python? It would be nice to use Python for more serious projects, but it isn't fast enough currently. ===== Regards, Graham http://movies.yahoo.com.au - Yahoo! Movies - What's on at your local cinema?

27 95

RE: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax)
by Moore, Paul Feb. 4, 2003

Feb. 4, 2003

From: Bernhard Herzog [mailto:bh@intevation.de] > IMO if you want multiple controllers for a with statement you > should just nest the withs: > > with var1 = expr1: > with var2 = expr2: > suite To some extent, an important point of the with expression is to simplify the overall control construct, and to reduce the level of nesting needed. Combining the variables was intended to help with this. > This makes the order in which the methods of the varN will be > called obvious. It should be fairly obvious from the combined form too - enter in the order specified, and leave in reverse order. > > In this, any varN can be omitted and a dummy hidden variable > > is used. > > Meaning something like > > with var1, , var3 = expr1, expr2, expr3: > suite > > I guess. Urgs. Agreed. I didn't mean that. Originally, I'd specified this as with var1 = expr1, var2 = expr2, ...: but I switched to parallel the normal multiple assignment idiom. How about with [ var1, var2, ... = ] expr1, expr2, ... so that you can omit all or none? That's basically just like a normal tuple assignment. > Nested withs handle this in a cleaner way as well, IMO. Yes, I guess they do. I'll make sure I have this as at least an option in the PEP. I still like the combined form, just because it reduces the tendency to drift towards the right margin. But I'm not convinced it will be important particularly often... Thanks for the comments, Paul.

4 3

database APIs
by Luke Kenneth Casson Leighton Feb. 4, 2003

Feb. 4, 2003

has anyone considered doing database helper functions based off of the DB 2.0 api? the question is inspired by the amount of work that is needed to create a useable suite of database queries in the shortest period of time - something that is important when both developing databases and also when doing lots of database applications one after the other. also for some of the background please see the advogato article http://advogato.org/article/600.html particularly the link to jmg's code which contains some very simple to use but obscure coding methods to wrap SQL databases in objects. the reason i ask is because the DB 2.0 api is great in that it makes things easy-ish to add in different databases (e.g. MySQL, MS-SQL, PostgreSQL) if you can deal with the syntactical differences, but it doesn't help you do things like: - make a query that gets you some results - modify one of the results - write the modified results back. i would _expect_ that the cursor class would have a function called save() which performed the last of these operations. e.g. as follows: cur = db.execute("select name from table where id=20") cur[0] = "fred's new name" cur.save() where the save() will AUTOMATICALLY create a database statement: update table set name="fred's new name" where id=20 now, i realise that the above idea has quite a lot of SQL-specific dependence: that can be dealt with by providing base classes where the analysis or creation of the query and also the save() function can be over-ridden. e.g. a class named DBgeneratorBase and therefore an example class can be created e.g. DBgeneratorSQL. i was therefore wondering about two things: - what do people on the python-dev list think of this concept? - is it something appropriate to be included in the python libs, given that the python db API at present is more of a convention than something that is enforced as part of the python library suite. -- ---------------------------------------------------------- this message is private, confidential, and is intented for the specified recipients only. if you received in error, altered, deleted, modified, destroyed or interfered with the contents of this message, in whole or in part, please inform the sender (that's me), immediately. if you, the recipient, reply to this message, and do not then receive a response, please consider your reply to have been lost or deliberately destroyed: i *always* acknowledge personal email received. please therefore take appropriate action and use appropriate protocols to ensure effective communication. thank you.

4 5

RE: [Python-Dev] Acquire/release functionality
by Moore, Paul Feb. 3, 2003

Feb. 3, 2003

Paul Moore <lists(a)morpheus.demon.co.uk> writes: > As far as the acquire/release case is concerned, I think that there is > definitely a strong case for syntactic support for this. The > equivalent idiom in C++ is "resource acquisition is initialisation" > which, while not having syntax support, does rely on the existence of > deterministic destructors, and on the ability to introduce a new scope > at will. Python has neither of these, and the try...finally construct > is verbose enough to make something better worth having. In trying to draw a parallel between "with" and the C++ RAII idiom, I hit a funny, best illustrated by an example: class autoclose(file): __exit__ = close with f = autoclose("blah.txt", "r"): # Let's get sneaky g = f # Main code here # File now gets closed with g: # Who's a naughty boy then - g is a closed file # And the g.__exit__() call goes boom here... OK, it's contrived. The equivalent C++ risk is that of creating a dangling pointer to a local variable which is deleted, which is also rare. I don't know if this is a significant issue - after all, I can write code which closes a closed file object in normal Python. But I do think that we should remember the weird edge cases when discussing this sort of thing. If nothing else, it strikes me as a mild argument in favour of allowing the assignment to be part of the "with" clause, just so that people don't get used to seeing a bare "with g" (which doesn't include any obvious clues that g *needs* auto-cleanup). Paul.

5 4

Idea for avoiding exception masking
by Raymond Hettinger Feb. 3, 2003

Feb. 3, 2003

[Raymond] > > So, here's the bright idea. Add a function, > > PyErr_FormatAppend, that leaves the original message > > intact but allows additional information to be added > > (like the name of the called function, identification > > of which argument triggered the error, a clue as > > to how many iterations had passed, or anything else > > that makes the traceback more informative). > > > > Python code has a number of cases where a higher > > level routine traps an exception and reraises it with > > new information and losing the lower level error > > detail in the process. [GvR] > Was a conclusion reached in this thread? It seems a good idea. We ended-up with two ways to go: 1. The most lightweight, least invasive approach was one the one first proposed. It allows extra information to be tacked onto the error message string. All other mechanisms work the same way. 2. The more involved approach is keep generate an exception as keep the original one as an attribute, "the root cause". This allows more information to be retained but raises a number of issues about how to display the result, whether the original exception is catchable, whether we want to intercept all exceptions, etc. The first approach is the "simplest thing that could possibly work". It is easy to implement, understand, and use. The second one would need to be teased out a lot more than it has been. Raymond Hettinger

1 0

Idea for avoiding exception masking
by Raymond Hettinger Feb. 3, 2003

Feb. 3, 2003

When working on the sets module, a bug was found where trapping an exception (a TypeError for a mutable argument passed to a dictionary) resulted in masking other errors that should have been passed through (potential TypeErrors in the called iterator for example). Now, Walter is working on a bug for map(), zip(), and reduce() where errors in the getiter() call are being trapped, reported as TypeError (for non-iterability), but potentially masking other real errors in a __iter__ routine. The current proposed solution is to remove the PyErr_Format call so that the underlying error message gets propagated up its original form. The downside of that approach is that it loses information about which argument caused the error. So, here's the bright idea. Add a function, PyErr_FormatAppend, that leaves the original message intact but allows additional information to be added (like the name of the called function, identification of which argument triggered the error, a clue as to how many iterations had passed, or anything else that makes the traceback more informative). Python code has a number of cases where a higher level routine traps an exception and reraises it with new information and losing the lower level error detail in the process. Raymond Hettinger

10 10

exec/with thunk-handling proposal (on a new thread eventually)
by holger krekel Feb. 3, 2003

Feb. 3, 2003

[sorry that this posting may double but i failed to post this on a new thread and even 'mutt' with 120 columns doesn't show the subject anymore] Hello, I'll try to take the discussion about a "unified approach" towards 'code block' or thunk handling into a fresh thread and present what Michael Hudson truthfully calls "a friendly competing approach". Preliminaries ------------- We have some use cases for a construct that allows encapsulating try/except/finally hooks into an object. Especially for timely finalization (of files, locks or other resources) we don't want to scatter try-finally constructs all over the place. Please understand that the following proposal has nothing to do with "function-level" thunk-manipulation ala classmethod but only with "inline" thunks (which don't and shouldn't have their own scope). Syntax and semantic proposal ---------------------------- I think we can may get away with only a "weak" keyword and allow the aforementioned encapsulation of execution events into an object like this: exec expr [with params]: suite where the expression is evaluated to return a "thunk" handler with these optional "execution" hooks: def __enter__(self): "before suite start" def __except__(self, type, value, tb): "swallow given exception, reraise if neccessary" def __leave__(self): """upon suite finish (not called if __except__ exists and an exception happened) """ The above "with" parameters (of the form name=expr, comma-separated) are bound in local (or global/nested) *and* handler instance namespace. The 'suite' is what we call "thunk". The above logic allows clean timely finalization for multiple ressources: exec autoclose() with f1=open(name1), f2=open(name2, 'w'): for line in f1: ... f2.write(...) which would execute as follows a) autoclose() instance is created and stored as the "thunk"-handler b) f1/f2 are stored as attributes on the autoclose instance c) f1/f2 are put into the local/global namespace (and nested ones if rebinding is allowed) d) thunk executes (for line ...) e) autoclose 'leave' hook is called (with or without exception) and is implemented like this: def __leave__(self): for obj in self.__dict__.values(): obj.close() f) thunk handler is removed Because computing 'f1' may succeed but 'f2' can subsequently fail the assignments *have to* execute within "autoclose" control. Now on to the usage of the except hook. Nice use cases might be exec retry(maxretry=3, on=IOError): # do network io-stuff or exec skip_on(AttributeError, TypeError): some_object.notify_hook() but i am sure there are more. Exception handling is often ugly when inlined with the code. I think that stating 'exception behaviour' up-front allows to write nice readable constructs. __exit__ versus __leave__ --------------------------- One remark (mainly to Michael as he does that other patch) about the hook-name __leave__ versus __exit__. we may want to eventually allow 'yield' within the thunk and then '__exit__' would be misleading. Here is the use case: exec self.mylock: # would lock/unlock on entering/leaving # the generator ... for whatever in something: yield whatever ... Or do you think that this (future) use case warrants yet another hook? If there is interest i can probably modify my patch to allow all of the proposed syntax so that you could play around with it. the next additional idea is not essential for my so-far proposal (but hopefully interesting, nonetheless). regards and interested in comments, holger ------------- Catching values ala Quixote ------------------- With my former patch there was another "execution" event: the catching of "dropped" values ala Quixote. For values that are not assigned to a name and would be discarded otherwise the execution handler can implement another hook: def __catch__(self, value): "catch dropped value from thunk" this allows anonymous (un-named) objects to be passed *within the thunk* to the execution handler. some example use-cases: exec debug: if condition: "something questionable happenend" so you can quite completly encapsulate debug-handling into one object. Note that this exec could be special-cased to do nothing if "debug" is None. Another one (ala the Quixote-framework): exec html_stream: "<h1>title</h1>" "<p>%s</p>" % html_quote(para) or even: exec html.tr(): exec html.td(): "column" which comes close to what Walter wants. And maybe you understand now why I choose this strange xml-syntax with my former patch :-) Actually i didn't have the idea of reusing 'exec' which makes a lot more sense to me, now. again with regards and intersted in any remarks, holger _______________________________________________ Python-Dev mailing list Python-Dev(a)python.org http://mail.python.org/mailman/listinfo/python-dev ----- End forwarded message ----- -- "Why are people killing each other when there is so much fun stuff to be had through friendly cooperation?" (Bengt Richter on c.l.py) "Why are people killing each other when there is so much fun stuff to be had through friendly cooperation?" (Bengt Richter on c.l.py)

1 1

Re: [Python-Dev] exec/with thunk-handling proposal
by jo＠jan.csie.ntu.edu.tw Feb. 3, 2003

Feb. 3, 2003

> From: "Gerald S. Williams" <gsw(a)agere.com> > holger krekel wrote: > > I think we can may get away with only a "weak" keyword > > and allow the aforementioned encapsulation of execution > > events into an object like this: > > > > exec expr [with params]: suite > > I was about to propose something similar. This looks like > it can be unified with what Guido proposed. > > > regards and interested in comments, > > An observation: > > Used like this, "with" looks like an alias for "lambda" > (providing default values for all parameters). > Yes. If we have multiline lambda, we could write: def autoclose(f, action): try: action(f) finally: f.close() autoclose(open("blah.txt"), lambda f: for line in f: print line )

1 0

bsddb3 upgrade woes - more gentle transition?
by Skip Montanaro Feb. 3, 2003

Feb. 3, 2003

I've seen a couple messages on c.l.py and the spambayes list about "breakage" with the new bsddb3 module. In both cases the breakage they occurred probably wasn't directly related to the change of the modules. It's almost certain that they have two versions of Berkeley DB installed and were using a 2.x or 1.8[56] version with the old bsddb module and are now linking against the 3.x or 4.x library with the new module. Given that most people aren't aware of how to convert their database files and very possibly don't have the necessary RPMs (or other system-specific packages) installed to do the file conversion, I think we need to work a little harder to smooth the transition. I'm not certain just what form that should take. At the very least I will write some text in Misc/NEWS to better indicate what the underlying problem is and ways to solve it. I suspect that many people won't have immediate access to Sleepycat's conversion tools (for any number of reasons). I think a pair of dump/load scripts in Tools/scripts which use pickle as an intermediate format would also be helpful. Skip

2 3