Mailman 3 February 2007 - Python-Dev

New syntax for 'dynamic' attribute access
by Ben North Feb. 16, 2007

Feb. 16, 2007

Hi, A few days ago I posted to python-ideas with a suggestion for some new Python syntax, which would allow easier access to object attributes where the attribute name is known only at run-time. For example: setattr(self, method_name, getattr(self.metadata, method_name)) from Lib/distutils/dist.py could be rewritten self.(method_name) = self.metadata.(method_name) The new syntax would also be usable in augmented assignments, as in obj.(attr_name) += 1 There was some discussion on python-ideas, which I've linked to in the draft PEP below. In particular, Guido van Rossum wrote: > I've thought of the same syntax. I think you should submit this to the > PEP editor and argue on Python-dev for its inclusion in Python 2.6 -- > there's no benefit that I see of waiting until 3.0. so here I am. Does anybody have any opinions/suggestions, particularly on the "open questions" referred to in the draft PEP? To summarise these open questions: * The draft currently allows a two-argument form, to supply a default value if the object has no attribute of that name. This mimics the behaviour of the three-argument form of getattr, but looks a bit wrong: s = obj.(attr_name, 'default string') I agree that it looks odd, but perhaps the extra expressive power gained might be worth the oddness. * The draft implementation (a sourceforge patch, linked to in the draft PEP) may have a general performance penalty of around 1%, although my initial measurements were quite noisy. Josiah Carlson thought this would not be too substantial, but he did suggest a couple of other implementation routes which could be explored. The general performance hit is offset by a speed gain of around 40% for attribute access using the new syntax over getattr etc. Is 1% "too much" for this feature? Ben. - - - - 8< - - - - PEP: 363 [PROVISIONAL NUMBER] Title: Syntax For Dynamic Attribute Access Version: $Revision$ Last-Modified: $Date$ Author: Ben North <ben at redfrontdoor.org> Status: Draft Type: Standards Track Content-Type: text/plain Created: 29-Jan-2007 Post-History: Abstract Dynamic attribute access is currently possible using the "getattr" and "setattr" builtins. The present PEP suggests a new syntax to make such access easier, allowing the coder for example to write x.('foo_%d' % n) += 1 z = y.('foo_%d' % n).('bar_%s' % s) instead of attr_name = 'foo_%d' % n setattr(x, attr_name, getattr(x, attr_name) + 1) z = getattr(getattr(y, 'foo_%d' % n), 'bar_%s' % s) Note (I wrote this patch mostly to advance my own understanding of and experiment with the python language, but I've written it up in the style of a PEP in case it might be a useful idea.) Rationale Dictionary access and indexing both have a friendly invocation syntax: instead of x.__getitem__(12) the coder can write x[12]. This also allows the use of subscripted elements in an augmented assignment, as in "x[12] += 1". The present proposal brings this ease-of-use to dynamic attribute access too. Attribute access is currently possible in two ways: * When the attribute name is known at code-writing time, the ".NAME" trailer can be used, as in x.foo = 42 y.bar += 100 * When the attribute name is computed dynamically at run-time, the "getattr" and "setattr" builtins must be used: x = getattr(y, 'foo_%d' % n) setattr(z, 'bar_%s' % s, 99) The "getattr" builtin also allows the coder to specify a default value to be returned in the event that the object does not have an attribute of the given name: x = getattr(y, 'foo_%d' % n, 0) This PEP describes a new syntax for dynamic attribute access --- "x.(expr)" --- with examples given in the Abstract above. The new syntax also allows the provision of a default value in the "get" case, as in: x = y.('foo_%d' % n, None) This 2-argument form of dynamic attribute access is not permitted as the target of an (augmented or normal) assignment. Also, this part of the new syntax was not as well received [6] in initial discussions on python-ideas, and I agree that it does not fit so cleanly. I'm happy to prepare a revised PEP/patch without the 2-argument form if the consensus is that this would be preferred. Finally, the new syntax can be used with the "del" statement, as in del x.(attr_name) Impact On Existing Code The proposed new syntax is not currently valid, so no existing well-formed programs have their meaning altered by this proposal. Across all "*.py" files in the 2.5 distribution, there are around 600 uses of "getattr", "setattr" or "delattr". They break down as follows (figures have some room for error because they were arrived at by partially-manual inspection): c.300 uses of plain "getattr(x, attr_name)", which could be replaced with the new syntax; c.150 uses of the 3-argument form, i.e., with the default value; these could be replaced with the 2-argument form of the new syntax (the cases break down into c.125 cases where the attribute name is a literal string, and c.25 where it's only known at run-time); c.5 uses of the 2-argument form with a literal string attribute name, which I think could be replaced with the standard "x.attribute" syntax; c.120 uses of setattr, of which 15 use getattr to find the new value; all could be replaced with the new syntax, the 15 where getattr is also involved would show a particular increase in clarity; c.5 uses which would have to stay as "getattr" because they are calls of a variable named "getattr" whose default value is the builtin "getattr"; c.5 uses of the 2-argument form, inside a try/except block which catches AttributeError and uses a default value instead; these could use 2-argument form of the new syntax; c.10 uses of "delattr", which could use the new syntax. As examples, the line setattr(self, attr, change_root(self.root, getattr(self, attr))) from Lib/distutils/command/install.py could be rewritten self.(attr) = change_root(self.root, self.(attr)) and the line setattr(self, method_name, getattr(self.metadata, method_name)) from Lib/distutils/dist.py could be rewritten self.(method_name) = self.metadata.(method_name) Performance Impact Initial pystone measurements are inconclusive, but suggest there may be a performance penalty of around 1% in the pystones score with the patched version. One suggestion is that this is because the longer main loop in ceval.c hurts the cache behaviour, but this has not been confirmed. (Maybe a tool like valgrind [2] could help here?) On the other hand, measurements suggest a speed-up of around 40--45% for dynamic attribute access. Discussion To Date Initial posting of this PEP in draft form was to python-ideas on 20070209 [4], and the response was generally positive: I've thought of the same syntax. I think you should submit this to the PEP editor and argue on Python-dev for its inclusion in Python 2.6 -- there's no benefit that I see of waiting until 3.0. --- Guido van Rossum [5] Wow! I have to say this is a compelling idea. The syntax is a bit foreign looking, but [...] I feel like I could learn to like it anyway. --- Greg Falcon [6] I look forward to seeing this in Python 2.6. --- Josiah Carlson, further down the thread [8] with Greg Falcon expressing the reservations about the 2-argument form already noted above, and Josiah Carlson raising a query about performance: My only concern with your propsed change is your draft implementation. [...] Specifically, your changes [...] may negatively affect general Python performance. --- Josiah Carlson [7] Some initial measurements (see above) suggest the performance penalty is small, and Josiah Carlson said of such cost that it "isn't really substantial". [8] Questions To Be Resolved * Whether to allow the 2-argument form for default arguments. * Whether the performance impact is real; whether it is acceptable; whether alternative implementations might improve this aspect. Alternative Syntax For The New Feature Other syntaxes could be used, for example braces are currently invalid in a "trailer", so could be used here, giving x{'foo_%d' % n} += 1 My personal preference is for the x.('foo_%d' % n) += 1 syntax though: the presence of the dot shows there is attribute access going on; the parentheses have an analogous meaning to the mathematical "work this out first" meaning. This is also the syntax used in the language Matlab [1] for dynamic "field" access (where "field" is the Matlab term analogous to Python's "attribute"). Discussions on python-ideas (see above) made no comment on the brace alternative, and the .() notation was well-enough received, so the brace alternative should be considered rejected, I think. Error Cases Only strings are permitted as attribute names, so for instance the following error is produced: >>> x.(99) = 8 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: attribute name must be string, not 'int' This is handled by the existing PyObject_GetAttr function. Draft Implementation A draft implementation adds a new alternative to the "trailer" clause in Grammar/Grammar; a new AST type, "DynamicAttribute" in Python.asdl, with accompanying changes to symtable.c, ast.c, and compile.c, and three new opcodes (load/store/del) with accompanying changes to opcode.h and ceval.c. The patch consists of c.180 additional lines in the core code, and c.100 additional lines of tests. It is available as sourceforge patch #1657573 [3]. References [1] Using Dynamic Field Names :: Data Types (MATLAB Programming) http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_prog/f2-41859.… [2] Valgrind: "suite of tools for debugging and profiling Linux programs" http://www.valgrind.org/ [3] Sourceforge patch #1657573 http://sourceforge.net/tracker/index.php?func=detail&aid=1657573&group_id=5… [4] http://mail.python.org/pipermail/python-ideas/2007-February/000210.html [5] http://mail.python.org/pipermail/python-ideas/2007-February/000211.html [6] http://mail.python.org/pipermail/python-ideas/2007-February/000212.html [7] http://mail.python.org/pipermail/python-ideas/2007-February/000213.html [8] http://mail.python.org/pipermail/python-ideas/2007-February/000227.html Copyright This document has been placed in the public domain.

25 40

Re: [Python-Dev] [Python-3000] UserDict revamp
by Raymond Hettinger Feb. 16, 2007

Feb. 16, 2007

> On 2/15/07, Raymond Hettinger <python(a)rcn.com> wrote: >> I would like to be the one to migrate it to Py3.0. [Steve] > No complaints here. Not that you need my permission of course. ;-) Thanks, I had already started working on this one. Of course, everyone is welcome to contribute. Raymond

2 1

Re: [Python-Dev] generic async io (was: microthreading vs. async io)
by Nick Maclaren Feb. 16, 2007

Feb. 16, 2007

Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: > > > An I/O operation passes a buffer, length, file and action and receives a > > token back. > > You seem to be using the word "threading" in a completely > different way than usual here, which may be causing some > confusion. Not really, though I may have been unclear again. Here is why that approach is best regarded as a threading concept: Perhaps the main current approach to using threads to implement asynchronous I/O operates by the main threads doing just that, and the I/O threads transferring the data synchronously. The reason that a token is needed is to avoid a synchronous data copy that blocks both threads. My general point is that all experience is that asynchronous I/O is best done by separating it completely from threads, and defining a proper asynchronous but NOT threaded interface. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

1 0

Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
by glyph＠divmod.com Feb. 16, 2007

Feb. 16, 2007

On 02:20 am, greg.ewing(a)canterbury.ac.nz wrote: >If Twisted is designed so that it absolutely *has* to >use its own special event mechanism, and everything else >needs to be modified to suit its requirements, then it's >part of the problem, not part of the solution. I've often heard this complaint, usually of the form "that's twisted-specific". The thing is, Twisted isn't specific. Its event mechanism isn't "special". In fact it's hard to imagine how it might be made less "special" than it currently is. Whatever your event dispatch mechanism, *some* code has to be responsible for calling the OS API of select(), WaitForMultipleEvents(), g_main_loop_run(), or whatever. Twisted actually imposes very few requirements for code to participate in this, and was designed from day one specifically to be a generalized mainloop mechanism which would not limit you to one underlying multiplexing implementation, event-dispatch mechanism, or operating system if you used its API. There have even been a few hacks to integrate Twisted with the asyncore mainloop, but almost everyone who knows both asyncore and Twisted has rapidly decided to just port all of their code to Twisted rather than maintain that bridge. In fact, Twisted includes fairly robust support for threads, so if you really need it you can mix and match event-driven and blocking styles. Again, most people who try this find that it is just nicer to write straight to the Twisted API, but for those that need really it, such as Zope and other WSGI-contained applications, it is available. Aside from the perennial issue of restartable reactors (which could be resolved as part of a stdlib push), Twisted's event loop imposes very few constraints on your code. It provides a lot of tools, sure, but few of them are required. You don't even *need* to use Deferreds. Now, "Twisted", overall, can be daunting. It has a lot of conventions, a lot of different interfaces to memorize and deal with, but if you are using the main loop you don't have to necessarily care about our SSH, ECMA 48, NNTP, OSCAR or WebDAV implementation. Those are all built at a higher level. It may seem like I'm belaboring the point here, but every time a thread comes up on this list mentioning the possibility of a "standard" event mechanism, people who do not know it very well or haven't used it start in with implied FUD that Twisted is a big, complicated, inseparable hairy mess that has no place in the everyday Python programmer's life. It's tangled-up Deep Magic, only for Wizards Of The Highest Order. Alternatively, more specific to this method, it's highly specific and intricate and very tightly coupled. Nothing could be further from the truth. It is *strictly* layered to prevent pollution of the lower levels by higher level code, and all the dependencies are one-way. Maybe our introductory documentation isn't very good. Maybe event-driven programming is confusing for those expecting something else. Maybe the cute names of some of the modules are offputting. Still, all that aside, if you are looking for an event-driven networking engine, Twisted is about as straightforward as you can get without slavishly coding to one specific platform API. When you boil it down, Twisted's event loop is just a notification for "a connection was made", "some data was received on a connection", "a connection was closed", and a few APIs to listen or initiate different kinds of connections, start timed calls, and communicate with threads. All of the platform details of how data is delivered to the connections are abstracted away. How do you propose we would make a less "specific" event mechanism? I strongly believe that there is room for a portion of the Twisted reactor API to be standardized in the stdlib so that people can write simple event-driven code "out of the box" with Python, but still have the different plug-in implementations live in Twisted itself. The main thing blocking this on my end (why I am not writing PEPs, advocating for it more actively, etc) is that it is an extremely low priority, and other, higher level pieces of Twisted have more pressing issues (such as the current confusion in the "web" universe). Put simply, although it might be nice, nobody really *needs* it in the stdlib, so they're not going to spend the effort to get it there. If someone out there really had a need for an event mechanism in the standard library, though, I encourage them to look long and hard at how the existing interfaces in Twisted could be promoted to the standard library and continue to be maintained compatibly in both places. At the very least, standardizing on something very much like IProtocol would go a long way towards making it possible to write async clients and servers that could run out of the box in the stdlib as well as with Twisted, even if the specific hookup mechanism (listenTCP, listenSSL, et. al.) were incompatible - although a signature compatible callLater would probably be a must. As I said, I don't have time to write the PEPs myself, but I might fix some specific bugs if there were a clear set of issues preventing this from moving forward. Better integration with the standard library would definitely be a big win for both Twisted and Python.

8 11

Wrapping up 'dynamic attribute' discussion
by Ben North Feb. 16, 2007

Feb. 16, 2007

I've sent an updated version of PEP 363 to the editors, which includes the following summary of the discussion. I hope I've captured the important points, but please let me know if there's something important I've left out or misrepresented. - - - - 8< - - - - Mailing Lists Discussion Initial posting of this PEP in draft form was to python-ideas on 20070209 [2], and the response was generally positive. The PEP was then posted to python-dev on 20070212 [3], and an interesting discussion ensued. A brief summary: Initially, there was reasonable (but not unanimous) support for the idea, although the precise choice of syntax had a more mixed reception. Several people thought the "." would be too easily overlooked, with the result that the syntax could be confused with a method/function call. A few alternative syntaxes were suggested: obj.(foo) obj.[foo] obj.{foo} obj{foo} obj.*foo obj->foo obj<-foo obj@foo obj.[[foo]] with "obj.[foo]" emerging as the preferred one. In this initial discussion, the two-argument form was universally disliked, so it was to be taken out of the PEP. Discussion then took a step back to whether this particular feature provided enough benefit to justify new syntax. As well as requiring coders to become familiar with the new syntax, there would also be the problem of backward compatibility --- code using the new syntax would not run on older pythons. Instead of new syntax, a new "wrapper class" was proposed, with the following specification / conceptual implementation suggested by Martin Loewis: class attrs: def __init__(self, obj): self.obj = obj def __getitem__(self, name): return getattr(self.obj, name) def __setitem__(self, name, value): return setattr(self.obj, name, value) def __delitem__(self, name): return delattr(self, name) def __contains__(self, name): return hasattr(self, name) This was considered a cleaner and more elegant solution to the original problem. The decision was made that the present PEP did not meet the burden of proof for the introduction of new syntax, a view which had been put forward by some from the beginning of the discussion. The wrapper class idea was left open as a possibility for a future PEP.

3 2

Summary: rejection of 'dynamic attribute' syntax
by Ben North Feb. 15, 2007

Feb. 15, 2007

Guido van Rossum wrote: > This seems to be the overwhelming feedback at this point, so I'm > withdrawing my support for the proposal. I hope that Ben can write up > a PEP and mark it rejected, to summarize the discussion; it's been a > useful lesson. The feedback is clear, yes. The "new syntax seems like overkill" camp has become dominant, and I certainly think these are very valid arguments. Gentle support remains scattered here and there, but the consensus has emerged. I will summarise all this into the PEP and mark it as rejected. Still, the discussion was useful, I thought. Thanks for the interest. To respond, for the record, to some of the specific points recently raised: glyph(a)divmod.com: > I really, really wish that every feature proposal for Python had to meet > some burden of proof [...]. I suspect this would kill 90% of "hey > wouldn't this syntax be neat" proposals on day zero [...] This is what I understood the initial posting to python-ideas to be about. If the feedback from there had been "that's a stupid idea", then that would have been the end of it. I think it's a good thing that there's the python-ideas mechanism for speculative suggestions. I like the "attrview" or "attrs" wrapper idea, with the example given by Larry Hastings: > setattr(self, method_name, getattr(self.metadata, method_name)) > [...] would be: > attrview(self)[method_name] = attrview(self.metadata)[method_name] As others point out, it's very clean, captures the intended uses well, and has the great advantage of having easily-added backwards compatibility. I might start using it :-) If the people who suggested and refined this were to put it in a PEP, I'd be in favour. Guido van Rossum wrote: > I missed discussion of the source of the 1%. Does it slow down pystone > or other benchmarks by 1%? That would be really odd, since I can't > imagine that the code path changes in any way for code that doesn't > use the feature. Is it that the ceval main loop slows down by having > two more cases? That seems to be it, yes. I tested this by leaving the grammar, compilation, and AST changes in, and conditionally compiling just the three extra cases in the ceval main loop. Measurements were noisy though, as Josiah Carlson has also experienced: > I've found variations of up to 3% in benchark times that seemed to be > based on whether I was drinking juice or eating a scone while working. I'm afraid I can't remember what I was eating or drinking at the time I did my tests. (Thanks also for the kind words regarding my summaries etc. Having caused all the fuss in the first place I felt obliged to try to make myself a bit useful :-) Ben.

6 9

Re: [Python-Dev] New syntax for 'dynamic' attribute access
by Delaney, Timothy (Tim) Feb. 15, 2007

Feb. 15, 2007

Greg Ewing wrote: > Oleg Broytmann wrote: > >>> Given that they say "a camel is a horse designed by a committee" >> >> BTW, camels are very suited for their environments... > > The quote is actually "a camel is a *racehorse* designed by a > committee". Camels are very good at surviving in the desert, but not > so good at winning a horse race (not camel race). Which is the point > of the saying. Speaking of which, have you ever seen a camel race? Those things go *fast* ... I think we're getting way too off-topic now ;) Tim Delaney

1 0

Re: [Python-Dev] Py2.6 ideas
by Delaney, Timothy (Tim) Feb. 15, 2007

Feb. 15, 2007

skip(a)pobox.com wrote: > >> Hm, but why would they still have to be tuples? Why not just > have a >> generic 'record' class? > > Tim> Hmm - possibilities. "record" definitely has greater > connotations Tim> of heterogeneous elements than "tuple", which > would put paid to the Tim> constant arguments that "a tuple is > really just an immutable list". > > (What do you mean by "... put paid ..."? It doesn't parse for me.) > Based on posts the current thread in c.l.py with the improbable > subject "f---ing typechecking", lots of people refuse to believe > tuples are anything other than immutable lists. Sorry - "put paid to" means "to finish" ... http://www.phrases.org.uk/meanings/293200.html That thread is a perfect example why I think a "record" type should be standard in python, and "tuple" should be deprecated (and removed in 3.0). Instead, have mutable and immutable lists, and mutable and immutable records. You could add a mutable list and an immutable list (resulting always in a new mutable list I think). You could *not* add two records together (even if neither had named elements). Cheers, Tim Delaney

1 0

Summary: rejection of 'dynamic attribute' syntax
by Ben North Feb. 15, 2007

Feb. 15, 2007

glyph(a)divmod.com wrote: > > > I really, really wish that every feature proposal for Python had to meet > > > some burden of proof Ben North wrote: > > This is what I understood the initial posting to python-ideas to be > > about. glyph(a)divmod.com wrote: > I'm suggesting that the standards of the community in _evaluating_ to > the proposals should be clearer Perhaps I didn't need to take your initial comments personally then, sorry :-) I do see what you're pointing out: the later part of the "dynamic attribute" discussion was where the question of whether python really needs new syntax for this was addressed, and the outcome made the earlier discussion of "x.[y]" vs "x.(y)" vs "x.{y}" vs "x->y" etc. irrelevant. Ben.

1 0

Re: [Python-Dev] generic async io (was: microthreading vs. async io)
by Nick Maclaren Feb. 15, 2007

Feb. 15, 2007

dustin(a)v.igoro.us wrote: > > I think this discussion would be facilitated by teasing the first > bullet-point from the latter two: the first deals with async IO, while > the latter two deal with cooperative multitasking. > > It's easy to write a single package that does both, but it's much harder > to write *two* fairly generic packages with a clean API between them, > given the varied platform support for async IO and the varied syntax and > structures (continuations vs. microthreads, in my terminology) for > multitasking. Yet I think that division is exactly what's needed. Hmm. Now, please, people, don't take offence, but I don't know how to phrase this tactfully :-( The 'threading' approach to asynchronous I/O was found to be a BAD IDEA back in the 1970s, was abandoned in favour of separating asynchronous I/O from threading, and God alone knows why it was reinvented - except that most of the people with prior experience had died or retired :-( Let's go back to the days when asynchronous I/O was the norm, and I/O performance critical applications drove the devices directly. In those days, yes, that approach did make sense. But it rapidly ceased to do so with the advent of 'semi-intelligent' devices and the virtualisation of I/O by the operating system. That was in the mid-1970s. Nowadays, ALL devices are semi-intelligent and no system since Unix has allowed applications direct access to devices, except for specialised HPC and graphics. We used to get 90% of theoretical peak performance on mainframes using asynchronous I/O from clean, portable applications, but it was NOT done by treating the I/O as threads and controlling their synchronisation by hand. In fact, quite the converse! It was done by realising that asynchronous I/O and explicit threading are best separated ENTIRELY. There were two main models: Streaming, as in most languages (Fortran, C, Python, but NOT in POSIX). The key properties here are that the transfer boundaries have no significance, only heavyweight synchronisation primitives (open, close etc.) provide any constraints on when data are actually transferred and (for very high performance) buffers are unavailable from when a transfer is started to when it is checked. If copying is acceptable, the last constraint can be dropped. In the simple case, this allows the library/system to reblock and perform transfers asynchronously. In the more advanced case, the application has to use multiple buffering (at least double), but can get full performance without any form of threading. IBM MVT applications used to get up to 90% without hassle in parallel with computation and using only a single thread (well, there was only a single CPU, anyway). The other model is transactions. This has the property that there is a global commit primitive, and the order of transfers is undefined between commits. Inter alia, it means that overlapping transfers are undefined behaviour, whether in a single thread or in multiple threads. BSP uses this model. The MPI-2 design team included a lot of ex-mainframe people and specifies both models. While it is designed for parallel applications, the I/O per se is not controlled like threads. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

2 1