Mailman 3 March 2007 - Python-ideas

fixing mutable default argument values
by Chris Rebert May 3, 2007

May 3, 2007

If A.M. Kuchling's list of Python Warts is any indication, Python has removed many of the warts it once had. However, the behavior of mutable default argument values is still a frequent stumbling-block for newbies. It is also present on at least 3 different lists of Python's deficiencies ([0][1][2]). Example of current, unintuitive behavior (snipped from [0]): >>> def popo(x=[]): ... x.append(666) ... print x ... >>> popo() [666] >>> popo() [666, 666] … [View More]

9 25

Re: [Python-ideas] [Python-Dev] Making builtins more efficient
by Greg Ewing April 20, 2007

April 20, 2007

Steven Elliott wrote: > What I have in mind may be close to what you are suggesting above. My idea is somewhat more uniform and general than that. For the module dict, you use a special mapping type that allows selected items to be accessed by an index as well as a name. The set of such names is determined when the module's code is compiled -- it's simply the names used in that module to refer to globals or builtins. The first time a given builtin is referenced in the module, it will be … [View More]

2 1

Python and Concurrency
by Talin April 12, 2007

April 12, 2007

I would like to direct your attention to a presentation that was given by Mr. Herb Sutter on the subject of concurrency support in programming languages. It's a long video, but I would urge you to watch at least the first five minutes of it: http://video.google.com/videoplay?docid=7625918717318948700&q=herb+sutter Mr. Sutter is the secretary of the ISO/C++ standards committee. I mention this only so that you'll understand he's not just some random yo-yo with an opinion. (There are a … [View More]number of other papers by Mr. Sutter on the same topic which you can google for, however none of them are as compelling or comprehensive as the video presentation IMHO.) The gist of the introduction is this (my words, not his): Starting from now, and for the rest of your career as a programmer, you better start programming for massively concurrent systems or you are doomed to irrelevance. As he points out in the talk, the good news is that Moore's law is going to continue for several decades to come. The bad news is that most people don't understand Moore's law or what they get from it. Moore's law is really about the number of transistors on a chip. It says nothing about processor speed or performance. He calls it "the end of the free lunch", meaning that in the past you could write a program, wait a little while, and it would run faster - because the hardware got faster while you were waiting. This will no longer be true for single-threaded programs. Scalar processors, with all of their pipelining, prefetching and other tricks, have gotten about as fast as they are ever going to get; We've reached the point of diminishing returns. So the only thing left to do is throw more and more cores on the chip. There's no physical reason why the number of cores won't continue to double every 18 months; the only reason why manufacturers might choose not to make 128-core CPUs by the year 2012 is that there won't be enough software to take advantages of that degree of parallelism. (Even with today's transistor count, you could put 100 Pentium-class processors on a single die.) Moreover, programming languages which have built-in support for scaling to many processors are going to win in the long run, and programming languages that don't have strong, built-in concurrency support will gradually fade away. The reason for this is simple: People don't like using applications which don't get faster when they buy a better computer. Now, on the server side, all of this is a solved problem. We have a very robust model for handling hundreds of requests in parallel, and a very strong model for dealing with shared state. This model is called "transactions" or more formally ACID. On the client side, however, things are much worse. As he puts it, locks are the best we have, and locks suck. Locks are not "logically composable", in the sense that you can't simply take two pieces of software that were written independently and glue them together unless they share a common locking architecture. This ruins a large part of the power of modern software, which is the ability to compose software out of smaller, independently-written components. On the client side, you have heterogeneous components accessing shared state at high bandwidth with myriads of pointer-based references to that shared state. Add concurrency, and what you end up with is a system in which it is impossible to make logical inferences about the behavior of the system. (And before you ask - yes, functional languages - and their drawbacks - are addressed in the talk. The same is true for 'lock-free programming' and other techniques du jour.) Now, in the talk he proposes some strategies for building concurrency into the language so that client-side programming can be done in a way that isn't rocket science. We won't be able to get rid of locks, he claims, but with the right language support at least we'll be able to reason about them, and maybe push them off to the side so that they mostly stay in their corner. In any case, what I would like to open up is a discussion of how these ideas might influence the design of the Python language. One thing that is important to understand is that I'm not talking about "automatic parallelization" where the compiler automatically figures out what parts can be parallelized. That would require so much change to the Python language as to make it no longer Python. Nor am I talking about manually creating "thread" objects, and having to carefully place protections around any shared state. That kind of parallelism is too low-level, too coarse-grained, and too hard to do, even for people who think they know how to write concurrent code. I am not even necessarily talking about changing the Python language (although certainly the internals of the interpreter will need to be changed.) The same language can be used to describe the same kinds of problems and operations, but the implications of those language elements will change. This is analogous to the fact that these massively multicore CPUs 10 years from now will most likely be using the same instruction sets as today - but that does not mean that programming as a discipline will anything like what it is now. As an example of what I mean, suppose the Python 'with' statement could be used to indicate an atomic transaction. Any operations that were done within that block would either be committed or rolled back. This is not a new idea - here's an interesting paper on it: http://www-sal.cs.uiuc.edu/~zilles/tm.html I'm sure that there's a lot more like that out there. However, there is also a lot of stuff out there that is *superficially* similar to what I am talking about, and I want to make the distinction clear. For example, any discussion of concurrency in Python will naturally raise the topic of both IPython and Stackless. However, IPython (from what I understand) is really about distributed computing and not so much about fine-grained concurrency; And Stackless (from what I understand) is really about coroutines or continuations, which is a different kind of concurrency. Unless I am mistaken (and I very well could be) neither of these are immediately applicable to the problem of authoring Python programs for multi-core CPUs, but I think that both of them contain valuable ideas that are worth discussing. Now, I will be the first to admit that I am not an expert in these matters. Don't let my poor, naive ideas be a limit to what is discussed. What I've primarily tried to do in this posting is to get your attention and convince you that this is important and worth talking about. And I hope to be able to learn something along the way. My overall goal here is to be able to continue writing programs in Python 10 years from now, not just as a hobby, but as part of my professional work. If Python is able to leverage the power of the CPUs that are being created at that time, I will be able to make a strong case for doing so. On the other hand, if I have a 128-core CPU on my desk, and Python is only able to utilize 1/128th of that power without resorting to tedious calculations of race conditions and deadlocks, then its likely that my Python programming will be relegated to the role of a hobby. -- Talin [View Less]

10 33

Python and Concurrency
by Richard Oudkerk March 29, 2007

March 29, 2007

Aahz wrote: > Maybe that's what it seems to you; to others of us who have been looking > at this problem for a while, the real question is how to get a better > multi-process control and IPC library in Python, preferably one that is > cross-platform. You can investigate that right now, and you don't even > need to discuss it with other people. > (Despite my oft-stated fondness for threading, I do recognize the > problems with threading, and if there were a way to make … [View More]

3 7

Replace time_t by Py_time_t
by Christian Heimes March 27, 2007

March 27, 2007

In the thread "datetime module enhancements" Guido and others said that it is unpythonic to limit timestamp (seconds since Epoch) to an signed int with 32bit. http://permalink.gmane.org/gmane.comp.python.devel/86750 I've made a patch that introduces a new type Py_time_t as a first step to increase the size of time stamps. For now it's just an alias for time_t. typedef time_t Py_time_t; I'm proposing to change time_t in two steps: Python 2.6: Replace every occurrence of time_t by … [View More]

1 0

Re: [Python-ideas] [Python-checkins] r54539 - in python/trunk: Lib/string.py Misc/NEWS Objects/typeobject.c
by Jim Jewett March 23, 2007

March 23, 2007

(Please note that most replies should trim at least one list from the Cc) On 3/23/07, guido.van.rossum <python-checkins(a)python.org> wrote: > Note: without the change to string.py, lots of spurious warnings happen. > What's going on there? I assume it was a defensive measure for subclasses of both Template and X, where X has its own metaclass (which might have an __init__ that shouldn't be ignored). This is where cooperative classes get ugly. You could argue that the "correct" … [View More]code is to not bother making the super call if it would go all the way to object (or even type), but there isn't a good way to spell that. > + # A super call makes no sense since type() doesn't define __init__(). > + # (Or does it? And should type.__init__() accept three args?) > + # super(_TemplateMetaclass, cls).__init__(name, bases, dct) In this particular case, you could define a type.__init__ that did nothing. (If the signature were wrong, type.__new__ would have already caught it. If __new__ and __init__ are seeing something different, then the change was probably intentional.) The problem isn't really limited to type.__init__ though. You'll sometimes see similar patterns for __del__, close, save, etc. The main difference is that they have to either catch an error or check first, since object doesn't have an implementation of those methods. object.__init__ doesn't really do anything either, except check for errors. Getting rid of it should have the same effect as complaining about parameters. The ideal solution (discussion of which probably ought to stay on python-ideas) might be to replace object.__init__ with some sort of PlaceHolder object that raises an error *unless* called through a cooperative super. This PlaceHolder would also be useful for AbstractBaseClasses/Interfaces. PlaceHolder still wouldn't deal with the original concern of verifying that all arguments had already been stripped and used; but the ABCs might be able to. -jJ > Modified: python/trunk/Lib/string.py > ============================================================================== > --- python/trunk/Lib/string.py (original) > +++ python/trunk/Lib/string.py Fri Mar 23 05:58:42 2007 > @@ -108,7 +108,9 @@ > """ > > def __init__(cls, name, bases, dct): > - super(_TemplateMetaclass, cls).__init__(name, bases, dct) > + # A super call makes no sense since type() doesn't define __init__(). > + # (Or does it? And should type.__init__() accept three args?) > + # super(_TemplateMetaclass, cls).__init__(name, bases, dct) > if 'pattern' in dct: > pattern = cls.pattern > else: [View Less]

1 0

Builtin infinite generator
by Tobias Ivarsson March 21, 2007

March 21, 2007

Quite often I find myself wanting to write an infinite for-loop, or rather a loop with a counter, that terminates on some inner condition. Recently I even wanted to do this together with the generator comprehension syntax, I don't remember exactly what I wanted to do, but it was something like: zip( some_list_of_unknown_length, ('a' for x in infinit_generator) ) I admit that my example is silly, but it still serves as an example of what I wanted to do. Then I read that Ellipsis will become … [View More]

4 3

datetime module enhancements
by Steven Bethard March 13, 2007

March 13, 2007

On 3/12/07, Christian Heimes <lists at cheimes.de> wrote: > datetime before date: < > datetime during date: in > datetime after date: > > datetime <= date and datetime => date should raise a TypeError. The > result is ambiguous. A date with time is never equal to a date. Jim Jewett wrote: > As a practical matter, sorting uses <= Are you sure? >>> class C(object): ... def __init__(self, x): ... self.x = x ... def … [View More]

3 3

Re: [Python-ideas] [Python-Dev] datetime module enhancements
by Collin Winter March 13, 2007

March 13, 2007

[Moving to python-ideas...] On 3/9/07, Steven Bethard <steven.bethard(a)gmail.com> wrote on python-dev: > On 3/9/07, Collin Winter <collinw(a)gmail.com> wrote on python-dev: > > One solution that just occurred to me -- and that skirts the issue of > > choosing an interpretation -- is that, when comparing date and > > datetime objects, the datetime's .date() method is called and the > > result of that call is compared to the original date. That is, > &… [View More]

6 5

Re: [Python-ideas] [Python-Dev] datetime module enhancements
by Collin Winter March 12, 2007

March 12, 2007

On 3/12/07, Steven Bethard <steven.bethard(a)gmail.com> wrote: > [I'm not on this list, so please keep me in the CC if you reply] > > On 3/11/07, Collin Winter <collinw(a)gmail.com> wrote: > > On 3/9/07, Steven Bethard <steven.bethard(a)gmail.com> wrote on python-dev: > > > It's *not* okay to say that a date() is less than, greater than or > > > equal to a datetime() if the year/month/day *does* match. The correct > > > temporal … [View More]

3 3