Mailman 3 March 2007 - Python-ideas

fixing mutable default argument values
by Chris Rebert 02 May '07

02 May '07

If A.M. Kuchling's list of Python Warts is any indication, Python has removed many of the warts it once had. However, the behavior of mutable default argument values is still a frequent stumbling-block for newbies. It is also present on at least 3 different lists of Python's deficiencies ([0][1][2]). Example of current, unintuitive behavior (snipped from [0]): >>> def popo(x=[]): ... x.append(666) ... print x ... >>> popo() [666] >>> popo() [666, 666] >>> popo() [666, 666, 666] Whereas a newbie with experience with immutable default argument values would, by analogy, expect: >>> popo() [666] >>> popo() [666] >>> popo() [666] In scanning [0], [1], [2], and other similar lists, I have only found one mediocre use-case for this behavior: Using the default argument value to retain state between calls. However, as [2] comments, this purpose is much better served by decorators, classes, or (though less preferred) global variables. Other uses are alluded to be equally esoteric and unpythonic. To work around this behavior, the following idiom is used: def popo(x=None): if x is None: x = [] x.append(666) print x However, why should the programmer have to write this extra boilerplate code when the current, unusual behavior is only relied on by 1% of Python code? Therefore, I propose that default arguments be handled as follows in Py3K: 1. The initial default value is evaluated at definition-time (as in the current behavior). 2. That in a function call where the caller has not specified a value for an optional argument, Python calls copy.deepcopy(initial_default_value), and fills in the optional argument with the resulting value. This is fully backwards-compatible with the aforementioned workaround, and removes the need for the it, allowing one to write the first, simpler definition of popo(). Comments? - Chris Rebert [0] 10 Python pitfalls (http://zephyrfalcon.org/labs/python_pitfalls.html) [1] Python Gotchas (http://www.ferg.org/projects/python_gotchas.html#contents_item_6) [2] When Pythons Attack (http://www.onlamp.com/pub/a/python/2004/02/05/learn_python.html?page=2)

9 25

Re: [Python-ideas] [Python-Dev] Making builtins more efficient
by Greg Ewing 19 Apr '07

19 Apr '07

Steven Elliott wrote: > What I have in mind may be close to what you are suggesting above. My idea is somewhat more uniform and general than that. For the module dict, you use a special mapping type that allows selected items to be accessed by an index as well as a name. The set of such names is determined when the module's code is compiled -- it's simply the names used in that module to refer to globals or builtins. The first time a given builtin is referenced in the module, it will be unbound in the module dict, so it is looked up in the usual way and then written into the module dict, so it can subsequently be retrieved by index. The advantages of this scheme over yours are that it speeds access to module-level names as well as builtins, and it doesn't require the compiler to have knowledge of a predefined set of names. It does entail a slight semantic change, as changes made to a builtin won't be seen by a module that has already used that builtin for the first time. But from what Guido has said before, it seems he is willing to accept a change like that if it will help. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing(a)canterbury.ac.nz +--------------------------------------+

2 1

Python and Concurrency
by Talin 12 Apr '07

12 Apr '07

I would like to direct your attention to a presentation that was given by Mr. Herb Sutter on the subject of concurrency support in programming languages. It's a long video, but I would urge you to watch at least the first five minutes of it: http://video.google.com/videoplay?docid=7625918717318948700&q=herb+sutter Mr. Sutter is the secretary of the ISO/C++ standards committee. I mention this only so that you'll understand he's not just some random yo-yo with an opinion. (There are a number of other papers by Mr. Sutter on the same topic which you can google for, however none of them are as compelling or comprehensive as the video presentation IMHO.) The gist of the introduction is this (my words, not his): Starting from now, and for the rest of your career as a programmer, you better start programming for massively concurrent systems or you are doomed to irrelevance. As he points out in the talk, the good news is that Moore's law is going to continue for several decades to come. The bad news is that most people don't understand Moore's law or what they get from it. Moore's law is really about the number of transistors on a chip. It says nothing about processor speed or performance. He calls it "the end of the free lunch", meaning that in the past you could write a program, wait a little while, and it would run faster - because the hardware got faster while you were waiting. This will no longer be true for single-threaded programs. Scalar processors, with all of their pipelining, prefetching and other tricks, have gotten about as fast as they are ever going to get; We've reached the point of diminishing returns. So the only thing left to do is throw more and more cores on the chip. There's no physical reason why the number of cores won't continue to double every 18 months; the only reason why manufacturers might choose not to make 128-core CPUs by the year 2012 is that there won't be enough software to take advantages of that degree of parallelism. (Even with today's transistor count, you could put 100 Pentium-class processors on a single die.) Moreover, programming languages which have built-in support for scaling to many processors are going to win in the long run, and programming languages that don't have strong, built-in concurrency support will gradually fade away. The reason for this is simple: People don't like using applications which don't get faster when they buy a better computer. Now, on the server side, all of this is a solved problem. We have a very robust model for handling hundreds of requests in parallel, and a very strong model for dealing with shared state. This model is called "transactions" or more formally ACID. On the client side, however, things are much worse. As he puts it, locks are the best we have, and locks suck. Locks are not "logically composable", in the sense that you can't simply take two pieces of software that were written independently and glue them together unless they share a common locking architecture. This ruins a large part of the power of modern software, which is the ability to compose software out of smaller, independently-written components. On the client side, you have heterogeneous components accessing shared state at high bandwidth with myriads of pointer-based references to that shared state. Add concurrency, and what you end up with is a system in which it is impossible to make logical inferences about the behavior of the system. (And before you ask - yes, functional languages - and their drawbacks - are addressed in the talk. The same is true for 'lock-free programming' and other techniques du jour.) Now, in the talk he proposes some strategies for building concurrency into the language so that client-side programming can be done in a way that isn't rocket science. We won't be able to get rid of locks, he claims, but with the right language support at least we'll be able to reason about them, and maybe push them off to the side so that they mostly stay in their corner. In any case, what I would like to open up is a discussion of how these ideas might influence the design of the Python language. One thing that is important to understand is that I'm not talking about "automatic parallelization" where the compiler automatically figures out what parts can be parallelized. That would require so much change to the Python language as to make it no longer Python. Nor am I talking about manually creating "thread" objects, and having to carefully place protections around any shared state. That kind of parallelism is too low-level, too coarse-grained, and too hard to do, even for people who think they know how to write concurrent code. I am not even necessarily talking about changing the Python language (although certainly the internals of the interpreter will need to be changed.) The same language can be used to describe the same kinds of problems and operations, but the implications of those language elements will change. This is analogous to the fact that these massively multicore CPUs 10 years from now will most likely be using the same instruction sets as today - but that does not mean that programming as a discipline will anything like what it is now. As an example of what I mean, suppose the Python 'with' statement could be used to indicate an atomic transaction. Any operations that were done within that block would either be committed or rolled back. This is not a new idea - here's an interesting paper on it: http://www-sal.cs.uiuc.edu/~zilles/tm.html I'm sure that there's a lot more like that out there. However, there is also a lot of stuff out there that is *superficially* similar to what I am talking about, and I want to make the distinction clear. For example, any discussion of concurrency in Python will naturally raise the topic of both IPython and Stackless. However, IPython (from what I understand) is really about distributed computing and not so much about fine-grained concurrency; And Stackless (from what I understand) is really about coroutines or continuations, which is a different kind of concurrency. Unless I am mistaken (and I very well could be) neither of these are immediately applicable to the problem of authoring Python programs for multi-core CPUs, but I think that both of them contain valuable ideas that are worth discussing. Now, I will be the first to admit that I am not an expert in these matters. Don't let my poor, naive ideas be a limit to what is discussed. What I've primarily tried to do in this posting is to get your attention and convince you that this is important and worth talking about. And I hope to be able to learn something along the way. My overall goal here is to be able to continue writing programs in Python 10 years from now, not just as a hobby, but as part of my professional work. If Python is able to leverage the power of the CPUs that are being created at that time, I will be able to make a strong case for doing so. On the other hand, if I have a 128-core CPU on my desk, and Python is only able to utilize 1/128th of that power without resorting to tedious calculations of race conditions and deadlocks, then its likely that my Python programming will be relegated to the role of a hobby. -- Talin

10 33

Python and Concurrency
by Richard Oudkerk 28 Mar '07

28 Mar '07

Aahz wrote: > Maybe that's what it seems to you; to others of us who have been looking > at this problem for a while, the real question is how to get a better > multi-process control and IPC library in Python, preferably one that is > cross-platform. You can investigate that right now, and you don't even > need to discuss it with other people. > (Despite my oft-stated fondness for threading, I do recognize the > problems with threading, and if there were a way to make processes as > simple as threads from a programming standpoint, I'd be much more > willing to push processes.) <plug> The processing package at http://cheeseshop.python.org/pypi/processing is multi-platform and mostly follows the API of threading. It also allows use of 'shared objects' which live in a manager process. For example the following code is almost identical to the equivalent written with threads: from processing import Process, Manager def f(q): for i in range(10): q.put(i*i) q.put('STOP') if __name__ == '__main__': manager = Manager() queue = manager.Queue(maxsize=3) p = Process(target=f, args=[queue]) p.start() result = None while result != 'STOP': result = queue.get() print result p.join() Josiah wrote: > Without work, #2 isn't "fast" using processes in Python. It is trivial > using threads. But here's the thing: with work, #2 can be made fast. > Using unix domain sockets (on linux, 3.4 ghz P4 Xeons, DDR2-PC4200 > memory (you can get 50% faster memory nowadays)), I've been able to push > 400 megs/second between processes. Maybe anonymous or named pipes, or > perhaps a shared mmap with some sort of synchronization would allow for > IPC to be cross platform and just about as fast. The IPC uses sockets or (on Windows) named pipes. Linux and Windows are roughly equal in speed. On a P4 2.5Ghz laptop one can retreive an element from a shared dict about 20,000 times/sec. Not sure if that qualifies as fast enough. </plug> Richard

3 7

Replace time_t by Py_time_t
by Christian Heimes 27 Mar '07

27 Mar '07

In the thread "datetime module enhancements" Guido and others said that it is unpythonic to limit timestamp (seconds since Epoch) to an signed int with 32bit. http://permalink.gmane.org/gmane.comp.python.devel/86750 I've made a patch that introduces a new type Py_time_t as a first step to increase the size of time stamps. For now it's just an alias for time_t. typedef time_t Py_time_t; I'm proposing to change time_t in two steps: Python 2.6: Replace every occurrence of time_t by Py_time_t and give third party authors time to change their software Python 2.7 / 3000: Change Py_time_t to a signed 64bit int on all platforms and provide the necessary workaround for platforms with a 32bit time_t. Patch: http://python.org/sf/1689402 Christian

1 0

Re: [Python-ideas] [Python-checkins] r54539 - in python/trunk: Lib/string.py Misc/NEWS Objects/typeobject.c
by Jim Jewett 23 Mar '07

23 Mar '07

(Please note that most replies should trim at least one list from the Cc) On 3/23/07, guido.van.rossum <python-checkins(a)python.org> wrote: > Note: without the change to string.py, lots of spurious warnings happen. > What's going on there? I assume it was a defensive measure for subclasses of both Template and X, where X has its own metaclass (which might have an __init__ that shouldn't be ignored). This is where cooperative classes get ugly. You could argue that the "correct" code is to not bother making the super call if it would go all the way to object (or even type), but there isn't a good way to spell that. > + # A super call makes no sense since type() doesn't define __init__(). > + # (Or does it? And should type.__init__() accept three args?) > + # super(_TemplateMetaclass, cls).__init__(name, bases, dct) In this particular case, you could define a type.__init__ that did nothing. (If the signature were wrong, type.__new__ would have already caught it. If __new__ and __init__ are seeing something different, then the change was probably intentional.) The problem isn't really limited to type.__init__ though. You'll sometimes see similar patterns for __del__, close, save, etc. The main difference is that they have to either catch an error or check first, since object doesn't have an implementation of those methods. object.__init__ doesn't really do anything either, except check for errors. Getting rid of it should have the same effect as complaining about parameters. The ideal solution (discussion of which probably ought to stay on python-ideas) might be to replace object.__init__ with some sort of PlaceHolder object that raises an error *unless* called through a cooperative super. This PlaceHolder would also be useful for AbstractBaseClasses/Interfaces. PlaceHolder still wouldn't deal with the original concern of verifying that all arguments had already been stripped and used; but the ABCs might be able to. -jJ > Modified: python/trunk/Lib/string.py > ============================================================================== > --- python/trunk/Lib/string.py (original) > +++ python/trunk/Lib/string.py Fri Mar 23 05:58:42 2007 > @@ -108,7 +108,9 @@ > """ > > def __init__(cls, name, bases, dct): > - super(_TemplateMetaclass, cls).__init__(name, bases, dct) > + # A super call makes no sense since type() doesn't define __init__(). > + # (Or does it? And should type.__init__() accept three args?) > + # super(_TemplateMetaclass, cls).__init__(name, bases, dct) > if 'pattern' in dct: > pattern = cls.pattern > else:

1 0

Builtin infinite generator
by Tobias Ivarsson 21 Mar '07

21 Mar '07

Quite often I find myself wanting to write an infinite for-loop, or rather a loop with a counter, that terminates on some inner condition. Recently I even wanted to do this together with the generator comprehension syntax, I don't remember exactly what I wanted to do, but it was something like: zip( some_list_of_unknown_length, ('a' for x in infinit_generator) ) I admit that my example is silly, but it still serves as an example of what I wanted to do. Then I read that Ellipsis will become generally accepted in py3k [1] and thought why not let range accept ... as end-parameter to mean "until forever". More silly examples to demonstrate how it would work: >>> for i in range(...): ... print i ... if i == 4: break 0 1 2 3 4 >>> for i in range(10,...,10): ... print i ... if i == 40: break 10 20 30 40 Any thoughts on this? /Tobias [1] http://mail.python.org/pipermail/python-3000/2006-April/000996.html

4 3

datetime module enhancements
by Steven Bethard 12 Mar '07

12 Mar '07

On 3/12/07, Christian Heimes <lists at cheimes.de> wrote: > datetime before date: < > datetime during date: in > datetime after date: > > datetime <= date and datetime => date should raise a TypeError. The > result is ambiguous. A date with time is never equal to a date. Jim Jewett wrote: > As a practical matter, sorting uses <= Are you sure? >>> class C(object): ... def __init__(self, x): ... self.x = x ... def __lt__(self, other): ... return self.x < other.x ... def __le__(self, other): ... raise TypeError() ... def __repr__(self): ... return 'C(%r)' % self.x ... >>> sorted([C(3), C(2), C(1)]) [C(1), C(2), C(3)] Looks like it's just using < to me. (And thanks to Collin and Christian for jumping on the patch already.) STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy

3 3

Re: [Python-ideas] [Python-Dev] datetime module enhancements
by Collin Winter 12 Mar '07

12 Mar '07

[Moving to python-ideas...] On 3/9/07, Steven Bethard <steven.bethard(a)gmail.com> wrote on python-dev: > On 3/9/07, Collin Winter <collinw(a)gmail.com> wrote on python-dev: > > One solution that just occurred to me -- and that skirts the issue of > > choosing an interpretation -- is that, when comparing date and > > datetime objects, the datetime's .date() method is called and the > > result of that call is compared to the original date. That is, > > > > datetime_obj < date_obj > > > > is implicitly equivalent to > > > > datetime_obj.date() < date_obj > > Using the .date() is fine when the year/month/day doesn't match. So > the following are fine:: > datetime.datetime(2005, 1, 1, 0, 0, 0) < datetime.date(2006, 1, 1) > datetime.datetime(2007, 1, 1, 0, 0, 0) > datetime.date(2006, 1, 1) > It's *not* okay to say that a date() is less than, greater than or > equal to a datetime() if the year/month/day *does* match. The correct > temporal relation is During, but Python doesn't have a During > operator. During is not the same as less-than, greater-than or > equal-to, so all of these should be False:: > datetime.datetime(2006, 1, 1, 0, 0, 0) < datetime.date(2006, 1, 1) > datetime.datetime(2006, 1, 1, 0, 0, 0) > datetime.date(2006, 1, 1) > datetime.datetime(2006, 1, 1, 0, 0, 0) == datetime.date(2006, 1, 1) > That is, the datetime() is not less than, greater than or equal to the > corresponding date(). > > Some discussion of these kinds of issues is here: > http://citeseer.ist.psu.edu/allen94actions.html > The essence is that in order to properly compare intervals, you need > the Meets, Overlaps, Starts, During and Finishes operators in addition > to the Before (<) and Simulaneous (=) operators. > > So, let's not conflate Before, After or Simultaneous with the other > relations -- if it's not strictly Before (<), After (>) or > Simultaneous (=), we can just say so by returning False. It might be neat to add a __contains__ method to date() objects so that "datetime(2007, 1, 1, 0, 0, 0) in date(2007, 1, 1)" would be True. This would seem to fulfill the During operator. Collin Winter

6 5

Re: [Python-ideas] [Python-Dev] datetime module enhancements
by Collin Winter 12 Mar '07

12 Mar '07

On 3/12/07, Steven Bethard <steven.bethard(a)gmail.com> wrote: > [I'm not on this list, so please keep me in the CC if you reply] > > On 3/11/07, Collin Winter <collinw(a)gmail.com> wrote: > > On 3/9/07, Steven Bethard <steven.bethard(a)gmail.com> wrote on python-dev: > > > It's *not* okay to say that a date() is less than, greater than or > > > equal to a datetime() if the year/month/day *does* match. The correct > > > temporal relation is During, but Python doesn't have a During > > > operator. During is not the same as less-than, greater-than or > > > equal-to, so all of these should be False:: > > > datetime.datetime(2006, 1, 1, 0, 0, 0) < datetime.date(2006, 1, 1) > > > datetime.datetime(2006, 1, 1, 0, 0, 0) > datetime.date(2006, 1, 1) > > > datetime.datetime(2006, 1, 1, 0, 0, 0) == datetime.date(2006, 1, 1) > > > That is, the datetime() is not less than, greater than or equal to the > > > corresponding date(). > > > > > > Some discussion of these kinds of issues is here: > > > http://citeseer.ist.psu.edu/allen94actions.html > > > The essence is that in order to properly compare intervals, you need > > > the Meets, Overlaps, Starts, During and Finishes operators in addition > > > to the Before (<) and Simulaneous (=) operators. > > > > > > So, let's not conflate Before, After or Simultaneous with the other > > > relations -- if it's not strictly Before (<), After (>) or > > > Simultaneous (=), we can just say so by returning False. > > > > It might be neat to add a __contains__ method to date() objects so > > that "datetime(2007, 1, 1, 0, 0, 0) in date(2007, 1, 1)" would be > > True. This would seem to fulfill the During operator. > > That's a nice idea. With the simplest implementation, you could then > guarantee that one of the following would always be true:: > > datetime < date > datetime in date > datetime > date > > (That would actually conflate the Starts, Finishes and During > relations in the __contains__ operator, but I think that's a perfectly > reasonable interpretation, and I know it would be useful in my code at > least.) I'll work up a patch. Collin Winter

3 3