map() methods (was: Re: [Patches] Review (was: Please review before applying))
[Moving this to python-dev because it's a musing
The main point is to avoid string.*.
Agreed. Also replacing map by a loop might not even be slower. What remains as open question: Several modules need access to string constants, and they therefore still have to import string. Is there an elegant solution to this?
import string
That's why i asked for some way to access "".__class__ or whatever, to get into some common namespace with the constants.
I dunno. However, I've noticed that in many situations where map() could be used with a string.* function (*if* you care about the speed-up and you don't care about the readability issue), there's no equivalent that uses the new string methods. This stems from the fact that map() wants a function, not a method. Python 3000 solves this partly, assuming types and classes are unified there. Where in 1.5 we wrote map(string.strip, L) in Python 3K we will be able to write map("".__class__.strip, L) However, this is *still* not as powerful as map(lambda s: s.strip(), L) because the former requires that all items in L are in fact strings, while the latter works for anything with a strip() method (in particular Unicode objects and UserString instances). Maybe Python 3000 should recognize map(lambda) and generate more efficient code for it... --Guido van Rossum (home page: http://www.python.org/~guido/)
[Guido]
... However, this is *still* not as powerful as
map(lambda s: s.strip(), L)
because the former requires that all items in L are in fact strings, while the latter works for anything with a strip() method (in particular Unicode objects and UserString instances).
Maybe Python 3000 should recognize map(lambda) and generate more efficient code for it...
[s.strip() for s in L] That is, list comprehensions solved the speed, generality and clarity problems here before they were discovered <wink>.
Maybe Python 3000 should recognize map(lambda) and generate more efficient code for it...
[s.strip() for s in L]
That is, list comprehensions solved the speed, generality and clarity problems here before they were discovered <wink>.
Ah! I knew there had to be a solution without lambda! :-) --Guido van Rossum (home page: http://www.python.org/~guido/)
Tim> [s.strip() for s in L] Tim> That is, list comprehensions solved the speed, generality and Tim> clarity problems here before they were discovered <wink>. What is the status of list comprehensions in Python? I remember some work being done several months ago. They definitely don't appear to be in the 1.6a2. Was there some reason to defer them until later? Skip
[Skip Montanaro]
What is the status of list comprehensions in Python? I remember some work being done several months ago. They definitely don't appear to be in the 1.6a2. Was there some reason to defer them until later?
Greg Ewing posted a patch to c.l.py that implemented a good start on the proposal. But nobody has pushed it. I had hoped to, but ran out of time; not sure Guido even knows about Greg's patch. Perhaps the 1.6 source distribution could contain a new "intriguing experimental patches" directory? Greg's list-comp and Christian's Stackless have enough fans that this would probably be appreciated. Perhaps some other things too, if we all run out of time (thinking mostly of Vladimir's malloc cleanup and NeilS's gc).
Greg Ewing posted a patch to c.l.py that implemented a good start on the proposal. But nobody has pushed it. I had hoped to, but ran out of time; not sure Guido even knows about Greg's patch.
I vaguely remember, but not really. We did use his f(*args, **kwargs) patches as a starting point for a 1.6 feature though -- if the list comprehensions are in a similar state, they'd be great to start but definitely need work.
Perhaps the 1.6 source distribution could contain a new "intriguing experimental patches" directory? Greg's list-comp and Christian's Stackless have enough fans that this would probably be appreciated. Perhaps some other things too, if we all run out of time (thinking mostly of Vladimir's malloc cleanup and NeilS's gc).
Perhaps a webpage woule make more sense? There's no point in loading every download with this. And e.g. stackless evolves at a much faster page than core Python. I definitely want Vladimir's patches in -- I feel very guilty for not having reviewed his latest proposal yet. I expect that it's right on the mark, but I understand if Vladimir wants to wait with preparing yet another set of patches until I'm happy with the design... --Guido van Rossum (home page: http://www.python.org/~guido/)
Hi, I'm back on-line.
[Tim]
Perhaps the 1.6 source distribution could contain a new "intriguing experimental patches" directory? Greg's list-comp and Christian's Stackless have enough fans that this would probably be appreciated. Perhaps some other things too, if we all run out of time (thinking mostly of Vladimir's malloc cleanup and NeilS's gc).
I'd be in favor of including gc as an optional (experimental) feature. I'm quite confident that it will evolve into a standard feature, in its current or in an improved state. The overall strategy looks good, but there are some black spots w.r.t its cost, both in speed and space. Neil reported in private mail something like 5-10% mem increase, but I doubt that the picture is so optimistic. My understanding is that these numbers reflect the behavior of the Linux VMM in terms of effectively used pages. In terms of absolute, peak requested virtual memory, things are probably worse than that. We're still unclear on this... For 1.6, the gc option would be a handy tool for detecting cyclic trash. It will answer some expectations, and I believe we're ready to give some good feedback on its functioning, its purpose, its limitations, etc. By the time 1.6 is finalized, I expect that we'll know roughly its cost in terms of mem overhead. Overall, it would be nice to have it in the distrib as an experimental feature -- it would both bootstrap some useful feedback, and would encourage enthousiasts to look more closely at DSA/GC (DSA - dynamic storage allocation). By 1.7 (with Py3K on the horizon), we would have a good understanding on what to do with gc and how to do it. If I go one step further, what I expect is that the garbage collector would be enabled together with a Python-specific memory allocator which will compensate the cost introduced by the collector. There will some some stable state again (in terms of speed and size) similar to what we have now, but with a bonus pack of additional memory services.
I definitely want Vladimir's patches in -- I feel very guilty for not having reviewed his latest proposal yet. I expect that it's right on the mark, but I understand if Vladimir wants to wait with preparing yet another set of patches until I'm happy with the design...
Yes, I'd prefer to wait and get it right. There's some basis, but it needs careful rethinking again. I'm willing to fit in the 1.6 timeline but I understand very well that it's a matter of time :-). -- Vladimir MARANGOZOV | Vladimir.Marangozov@inrialpes.fr http://sirac.inrialpes.fr/~marangoz | tel:(+33-4)76615277 fax:76615252
[Vladimir Marangozov, on NeilS's gc patch]
... The overall strategy looks good, but there are some black spots w.r.t its cost, both in speed and space. Neil reported in private mail something like 5-10% mem increase, but I doubt that the picture is so optimistic. My understanding is that these numbers reflect the behavior of the Linux VMM in terms of effectively used pages. In terms of absolute, peak requested virtual memory, things are probably worse than that. We're still unclear on this...
Luckily, that's what Open Source is all about: if we have to wait for you (or Neil, or Guido, or anyone else) to do a formal study of the issue, the patch will never go in. Put the code out there and let people try it, and 50 motivated users will run the only 50 tests that really matter: i.e., does their real code suffer or not? If so, a few of them may even figure out why. less-thought-more-eyeballs-ly y'rs - tim
participants (4)
-
Guido van Rossum -
Skip Montanaro -
Tim Peters -
Vladimir.Marangozov＠inrialpes.fr