I'd like list.pop to accept an optional second argument giving a default value

I like the list pop method because it provides a way to use lists as thread safe queues and stacks (since append and pop are protected by the global interpreter lock). With pop, you can essentially test whether the list is empty and get a value if it isn't in one atomic operation: try: foo=queue.pop(0) except IndexError: ... empty queue case else: ... non-empty case, do something with foo Unfortunately, this incurs exception overhead. I'd rather do something like: foo=queue.pop(0,marker) if foo is marker: ... empty queue case else: ... non-empty case, do something with foo I'd be happy to provide a patch. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Jim> I like the list pop method because it provides a way to use lists Jim> as thread safe queues and stacks (since append and pop are Jim> protected by the global interpreter lock). The global interpreter lock is a property of the current implementation of Python, not of the language itself. At one point in the past Greg Stein created a set of patches that eliminated the lock. While it's perhaps convenient to use now, it may not always exist. I'm not so sure that it should be used as a motivator for changes to libraries in the standard distribution. Skip

Skip Montanaro wrote:
If the global interpreter lock goes away, then some other locking mechanism will be used to make built-in object operations atomic. For example, in Greg's changes, each list was protected by a list lock. The key is that pop combines checking for an empty list and removing an element into a single operation. As long as the operations append and pop are atomic, then lists can be used as thread-safe stacks and queues. The benefit of the proposal does not really depend on the global interpreter lock. It only depends on list operations being atomic. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Jim Fulton writes:
I'm assuming you're asking for the equivalent of: def pop(self, default=None): much like dict.get? Then how do I get the old behavior? (I've been known to do odd things - like change behavior based on the number of args - in extension modules, but this ain't an extension). - Gordon

Gordon McMillan wrote:
Then how do I get the old behavior?
Just pass 0 or 1 argument.
It *is* a built-in method. It will be handled just like dictionaries handle the second argument to get. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Jim Fulton wrote:
d.get(nonexistantkey) does not throw an exception, it returns None. If list.pop() does not throw an exception when list is empty, it's new behavior. Which are you asking for: breaking code that expects IndexError Violating Pythonic expectations by, in effect, creating 2 methods list.pop(void) list.pop(default_return) - Gordon

Gordon McMillan wrote:
Oops, I'd forgotten that.
No.
Yes, except that I disagree that this is non-pythonic. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Jim Fulton wrote:
Gordon McMillan wrote:
...
I'll leave the final determination to Mr. Python, but I disagree. Offhand I can't think of a built-in that can't be expressed in normal Python notation, where "optional" args are really defaulted args. Which would lead us to either a new list method, or redefining pop: def pop(usedefault=0, default=None) and making you use 2 args. But maybe I've missed a precedent because I'm so used to it. (Hmm, I guess string.split is a sort-of precedent, because the first default arg behaves differently than anything you could pass in). - Gordon

Gordon McMillan wrote:
Offhand I can't think of a built-in that can't be expressed in normal Python notation, where "optional" args are really defaulted args.
I can define the pop I want in Python as follows: _marker=[] class list: ... def pop(index=-1, default=marker): try: v=self[index] except IndexError: if default is not marker: return default if self: m='pop index out of range' else: m='pop from empty list' raise IndexError, m del self[index] return v Although I'm not sure why the "pythonicity" of an interface should depend on it's implementation. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

BTW, a good precedent for what I want is getattr. getattr(None,'spam') raises an error, but: getattr(None,'spam',1) returns 1 Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

"JF" == Jim Fulton <jim@digicool.com> writes:
JF> BTW, a good precedent for what I want JF> is getattr. JF> getattr(None,'spam') JF> raises an error, but: JF> getattr(None,'spam',1) JF> returns 1 Okay, how did this one sneak in, huh? I didn't even realize this had been added to getattr()! CVS reveals it was added b/w 1.5.1 and 1.5.2a1, so maybe I just missed the checkin message. Fred, the built-in-funcs doc needs updating: http://www.python.org/doc/current/lib/built-in-funcs.html FWIW, the CVS log message says this feature is experimental too. :) -Barry

"Barry A. Warsaw" wrote:
I don't know. Someone told me about it. I find it wildly useful.
Eek! I want it to stay! I also really like list.pop. :) Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Jim Fulton wrote:
Me too... ;-)
http://www.deja.com/getdoc.xp?AN=366635977 -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 162 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

"M" == M <mal@lemburg.com> writes:
M> http://www.deja.com/getdoc.xp?AN=366635977 Ah, thanks! Your rationale was exactly the reason why I added dict.get(). I'm still not 100% sure about list.pop() though, since it's not exactly equivalent -- list.pop() modifies the list as a side-effect :) Makes me think you might want an alternative spelling for list[s], call it list.get() and put the optional default on that method. Then again, maybe list.pop() with an optional default is good enough. -Barry

"Barry A. Warsaw" wrote:
list.get and list.pop are different, since get wouldn't modify the list and pop would. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

"Barry A. Warsaw" wrote:
Sure. Since a sequence is sort of a special kind of mapping, get makes sense. I definately, want pop. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Seconded! Also, things which appeared between some alphas and made it upto the final, are just there. It would be fair to update the CVS tree and say the features made it into the dist, even if it just was a mistake not to remove them in time. It was time enough. ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home

[Barry]
FWIW, the CVS log message says this feature [3-arg getattr] is experimental too. :)
[Jim]
Eek! I want it to stay!
I also really like list.pop. :)
Don't panic: Guido has never removed a feature explicitly called "experimental"; he's only removed non-experimental ones. that's-why-we-call-stackless-python-"an-experiment"<wink>-ly y'rs - tim

JF> getattr(None,'spam') JF> raises an error, but: JF> getattr(None,'spam',1) JF> returns 1 Barry A. Warsaw writes:
Fred, the built-in-funcs doc needs updating:
This is done in the CVS repository; thanks for pointing out the oversight! Do people realize that pop() already has an optional parameter? That *is* in the docs: http://www.python.org/docs/current/lib/typesseq-mutable.html See note 4 below the table. -Fred -- Fred L. Drake, Jr. <fdrake@acm.org> Corporation for National Research Initiatives

Jim Fulton wrote:
Wouldn't a generic builtin for these kinds of things be better, e.g. a function returning a default value in case an exception occurs... something like: tryexcept(list.pop(), IndexError, default) which returns default in case an IndexError occurs. Don't think this would be much faster that the explicit try:...except: though... -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 162 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

"M" == M <mal@lemburg.com> writes:
M> Wouldn't a generic builtin for these kinds of things be M> better, e.g. a function returning a default value in case M> an exception occurs... something like: M> tryexcept(list.pop(), IndexError, default) M> which returns default in case an IndexError occurs. Don't think M> this would be much faster that the explicit try:...except: M> though... Don't know if this would be better (or useful, etc.), but it could possibly be faster than explicit try/except, because with try/except you have to instantiate the exception object. Presumably tryexcept() -- however it was spelled -- would catch the exception in C, thus avoiding the overhead of exception object instantiation. -Barry

[M.-A. Lemburg]
As a function (builtin or not), tryexcept will never get called if list.pop() raises an exception. tryexcept would need to be a new statement type, and the compiler would have to generate code akin to try: whatever = list.pop() except IndexError: whatever = default If you want to do it in a C function instead to avoid the Python-level exception overhead, the compiler would have to wrap list.pop() in a lambda in order to delay evaluation until the C code got control; and then you've got worse overhead <wink>. generalization-is-the-devil's-playground-ly y'rs - tim

Tim Peters wrote:
Dang. You're right...
Oh well, forget the whole idea then. list.pop() is really not needed that often anyways to warrant the default arg thing, IMHO. dict.get() and getattr() have the default arg as performance enhancement and I believe that you wouldn't get all that much better performance on average by adding a second optional argument to list.pop(). BTW, there is a generic get() function in mxTools (you know where...) in case someone should be looking for such a beast. It works with all sequences and mappings. Also, has anybody considered writing list.pop(..,default) this way: if list: obj = list.pop() else: obj = default No exceptions, no changes, fast as hell :-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 162 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

"M.-A. Lemburg" wrote: ...
Yes, that's the best way to go, I think. But wasn't the primary question directed on an atomic function which is thread-safe? I'm not sure, this thread has grown too fast :-) -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home

Christian Tismer wrote:
I think that was what Jim had in mind in the first place. Hmm, so maybe we're not after lists after all: maybe what we need is access to the global interpreter lock in Python, so that we can write: sys.lock.acquire() if list: obj = list.pop() else: obj = default sys.lock.release() Or maybe we need some general lock in the thread module for these purposes... don't know. It's been some time since I used threads. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 162 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

The thread attempting the sys.lock.acquire() necessarily already owns the global lock, so the attempt to acquire it is a guaranteed deadlock -- arguably not helpful <wink>.
Jim could easily allocate a list lock for this purpose if that's what he wanted; and wrap it in a class with a nice interface too. He'd eventually end up with the std Queue.py module, though. But if he doesn't want the overhead of an exception when the queue is empty, he sure doesn't want the comparatively huge overhead of a (any flavor of) lock either (which may drag the OS into the picture). There's nothing wrong with wanting a fast thread-safe queue! I just don't like the idea of adding an otherwise-ugly new gimmick to core lists for it; also have to wonder about Jim's larger picture if he's writing stuff in Python that's *so* time-critical that the overhead of an ordinary exception from time to time is a genuine problem. The verbosity of the alternative can be hidden in a lock-free class or function, if it's the clumsiness instead of the time that's grating.

Tim Peters wrote:
True, sys.lock.acquire() would have to set a flag *not* to release the lock until the next call to sys.lock.release(), which then clears this flag again. Sort of a lock for the unlocking the lock ;-) Could this work, or am I having a mind twister somewhere in there again ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 160 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

M.-A. Lemburg writes:
Sounds like a critical section to me. On Windows, those are lightweight and very handy. You can build one with Python thread primitives, but unfortunately, they come out on the heavy side. Locks come in 4 types, categorized by whether they can be released only by the owning thread, and whether they can be acquired recursively. The interpreter lock is in the opposite quadrant from a critical section, so "sys.lock.freeze()" and "sys.lock.thaw()" have little chance of having an efficient implementation on any platform. A shame. That would be pretty cool. - Gordon

Gordon McMillan wrote:
Actually, I think all that's needed is another global like the interpreter_lock in ceval.c. Since this lock is only accessed via abstract functions, I presume the unlock flag could easily be added. The locking section would only focus on Python, though: other threads could still be running provided they don't execute Python code, e.g. write data to a spooler. So it's not really the equivalent of a critical section as the one you can define in C. PS: I changed the subject line... hope this doesn't kill the thread ;) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 158 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Well, my personal opinion is that this is really quite wrong. The most obvious thing to me is that we are exposing an implementation detail we all would dearly like to see removed one day - the global interpreter lock. But even if we ignore that, it seems to me that you are describing an application abstraction, not a language abstraction. This thread started with Jim wanting a thread-safe, atomic list operation. This is not an unusual requirement (ie, a thread-safe, atomic operation), so languages give you access to primitives that let you build this. To my mind, you are asking for the equivilent of a C function that says "suspend all threads except me, cos Im doing something _really_ important". C does not provide that, and I have never thought it should. As Gordon said, Win32 has critical sections, but these are really just lightweight locks. I really dont see how Python is different - it gives you all the tools you need to build these abstractions. I really dont see what you are after that can not be done with a lock. If the performance is a problem, then to paraphrase the Timbot, it may be questionable if you are using Python appropriately in this case. Mark.

[The previous mail got truncated due to insufficient disk space; here is a summary...] Mark Hammond wrote:
The locked section may not be leading in the right direction, but it surely helps in situations where you cannot otherwise enforce useage of an object specific lock, e.g. for builtin file objects (some APIs insist on getting the real thing, not a thread safe wrapper). Here is a hack that let's you do much the same with an unpatched Python interpreter: sys.setcheckinterval(sys.maxint) # *) # >=10 Python OPs to flush the ticker counter and have the new # check interavl setting take effect: 0==0; 0==0; 0==0; 0==0 try: ...lock section... finally: sys.setcheckinterval(10) *) sys.setcheckinterval should really return the previous value so that we can reset the value to the original one afterwards. Note that the lock section may not call code which uses the Py_*_ALLOW_THREADS macros. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 157 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

[Marc writes]
Really, all this boils down to is that you want a Python-ish critical section - ie, a light-weight lock. This presumably would be desirable if it could be shown Python locks are indeed "heavy" - I know that from the C POV they may be considered as such, but I havent seen many complaints about lock speed from Python. So in an attempt to get _some_ evidence, I wrote a test program that used the Queue module to append 10000 integers then remove them all. I then hacked the queue module to remove all locking, and ran the same test. The results were 2.4 seconds for the non-locking version, vs 3.8 for the standard version. Without time (or really inclination <wink>) to take this further, it _does_ appear a native Python "critical section" could indeed save a few milli-seconds for a few real-world apps. So if we ignore the implementation details Marc started spelling, does the idea of a Python "critical section" appeal? Could simply be a built-in way of saying "no other _Python_ threads should run" (and of-course the "allow them again"). The semantics could be simply to ensure the Python program integrity - it need say nothing about the Python internal "state" as such. Mark.

Christian Tismer wrote:
Right. And the above code doesn't solve this problem. Tim's code *does* solve the problem. It's the code we were using. It is a bit verbose though.
I'm not sure, this thread has grown too fast :-)
Don't they all? Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Tim Peters wrote:
As a function (builtin or not), tryexcept will never get called if list.pop() raises an exception.
M.-A. Lemburg writes:
Oh well, forget the whole idea then. list.pop() is really not
Giving up already? Wouldn't you just love this as an expression operator (which could work)? How about: top = list.pop() excepting IndexError, default Hehehe... ;-) -Fred -- Fred L. Drake, Jr. <fdrake@acm.org> Corporation for National Research Initiatives

Fred> Giving up already? Wouldn't you just love this as an expression Fred> operator (which could work)? Fred> How about: Fred> top = list.pop() excepting IndexError, default Why not go all the way to Perl with top = list.pop() unless IndexError ??? ;-) Skip

Skip Montanaro writes:
Why not go all the way to Perl with
top = list.pop() unless IndexError
Trying to kill me, Skip? ;-) Actually, the semantics are different. If we interpret that using the Perl semantics for "unless", don't we have the same thing as: if not IndexError: top = list.pop() Since IndexError will normally be a non-empty string or a class, this is pretty much: if 0: top = list.pop() which certainly isn't quite as interesting. ;-) -Fred -- Fred L. Drake, Jr. <fdrake@acm.org> Corporation for National Research Initiatives

Fred> Skip Montanaro writes: >> Why not go all the way to Perl with >> >> top = list.pop() unless IndexError Fred> Trying to kill me, Skip? ;-) Nope, just a flesh wound. I'll wait for the resulting infection to really do you in. ;-) Fred> Actually, the semantics are different. If we interpret that using Fred> the Perl semantics for "unless", don't we have the same thing as: Yes, but the flavor is the same. Reading Perl code that uses the unless keyword always seemed counterintuitive to me. Something like x = y unless foo; always reads to me like, "Assign y to x. No, wait a minute. I forgot something. Only do that if foo isn't true." What was so bad about if (!foo) { x = y; } That was my initial reaction to the use of the trailing except. We argue a lot in the Python community about whether or not a proposed language feature increases the expressive power of the language or not (which is a good idea in my opinion). The Perl community has apparently never been afflicted with that disease. smiles all 'round... Skip

[M.-A. Lemburg]
I like their succinctness too; count = dict.get(key, 0) is helpfully "slimmer" than either of try: count = dict[key] except KeyError: count = 0 or count = 0 if dict.has_key(key): count = dict[key]
and I believe that you wouldn't get all that much better performance on average by adding a second optional argument to list.pop().
I think you wouldn't at *all*, except in Jim's novel case. That is, when a list is empty, it's usually the signal to get out of a loop, and you can either test if list: item = list.pop() else: break today or item = list.pop(-1, marker) if item is marker: break tomorrow. The second way doesn't buy anything to my eye, and the first way is very often the pretty while list: item = list.pop() if-it-weren't-for-jim's-use-i'd-see-no-use-at-all-ly y'rs - tim

"Gordo" == Gordon McMillan <gmcm@hypernet.com> writes:
Gordo> Which are you asking for: breaking code that expects Gordo> IndexError Violating Pythonic expectations by, in effect, Gordo> creating 2 methods Gordo> list.pop(void) Gordo> list.pop(default_return) The docs /do/ say that list.pop() is experimental, so that probably gives Guido all the out he'd need to change the semantics :). I myself have yet to use list.pop() so I don't know how disasterous the change in semantics would be to existing code. -Barry

In a moment of insanity, Guido gave me carte blanche to suggest new list methods, and list.pop & list.extend were the result. I considered spec'ing list.pop to take an optional "default on bad index" argument too, but after playing with it didn't like it (always appeared just as easy & clearer to use "if list:" / "while list:" etc). Jim has a novel use I hadn't considered:
It's both clever and pretty. OTOH, the original try/except isn't expensive unless the "except" triggers frequently, in which case (the queue is often empty) a thread is likely better off with a yielding Queue.get() call. So this strikes me as useful only for thread micro-optimization, and a kind of optimization most users should be steered away from anyway. Does anyone have a real use for this outside of threads? If not, I'd rather it not go in. For threads that need an optimized non-blocking probe, I'd write it: gotone = 0 if queue: try: foo = queue.pop(0) gotone = 1 except IndexError: pass if gotone: # use foo else: # twiddle thumbs For the IndexError to trigger there, a thread has to lose its bytecode slice between a successful "if queue" and the queue.pop, and not get another chance to run until other threads have emptied the queue.

Jim> I like the list pop method because it provides a way to use lists Jim> as thread safe queues and stacks (since append and pop are Jim> protected by the global interpreter lock). The global interpreter lock is a property of the current implementation of Python, not of the language itself. At one point in the past Greg Stein created a set of patches that eliminated the lock. While it's perhaps convenient to use now, it may not always exist. I'm not so sure that it should be used as a motivator for changes to libraries in the standard distribution. Skip

Skip Montanaro wrote:
If the global interpreter lock goes away, then some other locking mechanism will be used to make built-in object operations atomic. For example, in Greg's changes, each list was protected by a list lock. The key is that pop combines checking for an empty list and removing an element into a single operation. As long as the operations append and pop are atomic, then lists can be used as thread-safe stacks and queues. The benefit of the proposal does not really depend on the global interpreter lock. It only depends on list operations being atomic. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Jim Fulton writes:
I'm assuming you're asking for the equivalent of: def pop(self, default=None): much like dict.get? Then how do I get the old behavior? (I've been known to do odd things - like change behavior based on the number of args - in extension modules, but this ain't an extension). - Gordon

Gordon McMillan wrote:
Then how do I get the old behavior?
Just pass 0 or 1 argument.
It *is* a built-in method. It will be handled just like dictionaries handle the second argument to get. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Jim Fulton wrote:
d.get(nonexistantkey) does not throw an exception, it returns None. If list.pop() does not throw an exception when list is empty, it's new behavior. Which are you asking for: breaking code that expects IndexError Violating Pythonic expectations by, in effect, creating 2 methods list.pop(void) list.pop(default_return) - Gordon

Gordon McMillan wrote:
Oops, I'd forgotten that.
No.
Yes, except that I disagree that this is non-pythonic. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Jim Fulton wrote:
Gordon McMillan wrote:
...
I'll leave the final determination to Mr. Python, but I disagree. Offhand I can't think of a built-in that can't be expressed in normal Python notation, where "optional" args are really defaulted args. Which would lead us to either a new list method, or redefining pop: def pop(usedefault=0, default=None) and making you use 2 args. But maybe I've missed a precedent because I'm so used to it. (Hmm, I guess string.split is a sort-of precedent, because the first default arg behaves differently than anything you could pass in). - Gordon

Gordon McMillan wrote:
Offhand I can't think of a built-in that can't be expressed in normal Python notation, where "optional" args are really defaulted args.
I can define the pop I want in Python as follows: _marker=[] class list: ... def pop(index=-1, default=marker): try: v=self[index] except IndexError: if default is not marker: return default if self: m='pop index out of range' else: m='pop from empty list' raise IndexError, m del self[index] return v Although I'm not sure why the "pythonicity" of an interface should depend on it's implementation. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

BTW, a good precedent for what I want is getattr. getattr(None,'spam') raises an error, but: getattr(None,'spam',1) returns 1 Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

"JF" == Jim Fulton <jim@digicool.com> writes:
JF> BTW, a good precedent for what I want JF> is getattr. JF> getattr(None,'spam') JF> raises an error, but: JF> getattr(None,'spam',1) JF> returns 1 Okay, how did this one sneak in, huh? I didn't even realize this had been added to getattr()! CVS reveals it was added b/w 1.5.1 and 1.5.2a1, so maybe I just missed the checkin message. Fred, the built-in-funcs doc needs updating: http://www.python.org/doc/current/lib/built-in-funcs.html FWIW, the CVS log message says this feature is experimental too. :) -Barry

"Barry A. Warsaw" wrote:
I don't know. Someone told me about it. I find it wildly useful.
Eek! I want it to stay! I also really like list.pop. :) Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Jim Fulton wrote:
Me too... ;-)
http://www.deja.com/getdoc.xp?AN=366635977 -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 162 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

"M" == M <mal@lemburg.com> writes:
M> http://www.deja.com/getdoc.xp?AN=366635977 Ah, thanks! Your rationale was exactly the reason why I added dict.get(). I'm still not 100% sure about list.pop() though, since it's not exactly equivalent -- list.pop() modifies the list as a side-effect :) Makes me think you might want an alternative spelling for list[s], call it list.get() and put the optional default on that method. Then again, maybe list.pop() with an optional default is good enough. -Barry

"Barry A. Warsaw" wrote:
list.get and list.pop are different, since get wouldn't modify the list and pop would. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

"Barry A. Warsaw" wrote:
Sure. Since a sequence is sort of a special kind of mapping, get makes sense. I definately, want pop. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Seconded! Also, things which appeared between some alphas and made it upto the final, are just there. It would be fair to update the CVS tree and say the features made it into the dist, even if it just was a mistake not to remove them in time. It was time enough. ciao - chris -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home

[Barry]
FWIW, the CVS log message says this feature [3-arg getattr] is experimental too. :)
[Jim]
Eek! I want it to stay!
I also really like list.pop. :)
Don't panic: Guido has never removed a feature explicitly called "experimental"; he's only removed non-experimental ones. that's-why-we-call-stackless-python-"an-experiment"<wink>-ly y'rs - tim

JF> getattr(None,'spam') JF> raises an error, but: JF> getattr(None,'spam',1) JF> returns 1 Barry A. Warsaw writes:
Fred, the built-in-funcs doc needs updating:
This is done in the CVS repository; thanks for pointing out the oversight! Do people realize that pop() already has an optional parameter? That *is* in the docs: http://www.python.org/docs/current/lib/typesseq-mutable.html See note 4 below the table. -Fred -- Fred L. Drake, Jr. <fdrake@acm.org> Corporation for National Research Initiatives

Jim Fulton wrote:
Wouldn't a generic builtin for these kinds of things be better, e.g. a function returning a default value in case an exception occurs... something like: tryexcept(list.pop(), IndexError, default) which returns default in case an IndexError occurs. Don't think this would be much faster that the explicit try:...except: though... -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 162 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

"M" == M <mal@lemburg.com> writes:
M> Wouldn't a generic builtin for these kinds of things be M> better, e.g. a function returning a default value in case M> an exception occurs... something like: M> tryexcept(list.pop(), IndexError, default) M> which returns default in case an IndexError occurs. Don't think M> this would be much faster that the explicit try:...except: M> though... Don't know if this would be better (or useful, etc.), but it could possibly be faster than explicit try/except, because with try/except you have to instantiate the exception object. Presumably tryexcept() -- however it was spelled -- would catch the exception in C, thus avoiding the overhead of exception object instantiation. -Barry

[M.-A. Lemburg]
As a function (builtin or not), tryexcept will never get called if list.pop() raises an exception. tryexcept would need to be a new statement type, and the compiler would have to generate code akin to try: whatever = list.pop() except IndexError: whatever = default If you want to do it in a C function instead to avoid the Python-level exception overhead, the compiler would have to wrap list.pop() in a lambda in order to delay evaluation until the C code got control; and then you've got worse overhead <wink>. generalization-is-the-devil's-playground-ly y'rs - tim

Tim Peters wrote:
Dang. You're right...
Oh well, forget the whole idea then. list.pop() is really not needed that often anyways to warrant the default arg thing, IMHO. dict.get() and getattr() have the default arg as performance enhancement and I believe that you wouldn't get all that much better performance on average by adding a second optional argument to list.pop(). BTW, there is a generic get() function in mxTools (you know where...) in case someone should be looking for such a beast. It works with all sequences and mappings. Also, has anybody considered writing list.pop(..,default) this way: if list: obj = list.pop() else: obj = default No exceptions, no changes, fast as hell :-) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 162 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

"M.-A. Lemburg" wrote: ...
Yes, that's the best way to go, I think. But wasn't the primary question directed on an atomic function which is thread-safe? I'm not sure, this thread has grown too fast :-) -- Christian Tismer :^) <mailto:tismer@appliedbiometrics.com> Applied Biometrics GmbH : Have a break! Take a ride on Python's Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net 10553 Berlin : PGP key -> http://wwwkeys.pgp.net PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF we're tired of banana software - shipped green, ripens at home

Christian Tismer wrote:
I think that was what Jim had in mind in the first place. Hmm, so maybe we're not after lists after all: maybe what we need is access to the global interpreter lock in Python, so that we can write: sys.lock.acquire() if list: obj = list.pop() else: obj = default sys.lock.release() Or maybe we need some general lock in the thread module for these purposes... don't know. It's been some time since I used threads. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 162 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

The thread attempting the sys.lock.acquire() necessarily already owns the global lock, so the attempt to acquire it is a guaranteed deadlock -- arguably not helpful <wink>.
Jim could easily allocate a list lock for this purpose if that's what he wanted; and wrap it in a class with a nice interface too. He'd eventually end up with the std Queue.py module, though. But if he doesn't want the overhead of an exception when the queue is empty, he sure doesn't want the comparatively huge overhead of a (any flavor of) lock either (which may drag the OS into the picture). There's nothing wrong with wanting a fast thread-safe queue! I just don't like the idea of adding an otherwise-ugly new gimmick to core lists for it; also have to wonder about Jim's larger picture if he's writing stuff in Python that's *so* time-critical that the overhead of an ordinary exception from time to time is a genuine problem. The verbosity of the alternative can be hidden in a lock-free class or function, if it's the clumsiness instead of the time that's grating.

Tim Peters wrote:
True, sys.lock.acquire() would have to set a flag *not* to release the lock until the next call to sys.lock.release(), which then clears this flag again. Sort of a lock for the unlocking the lock ;-) Could this work, or am I having a mind twister somewhere in there again ? -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 160 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

M.-A. Lemburg writes:
Sounds like a critical section to me. On Windows, those are lightweight and very handy. You can build one with Python thread primitives, but unfortunately, they come out on the heavy side. Locks come in 4 types, categorized by whether they can be released only by the owning thread, and whether they can be acquired recursively. The interpreter lock is in the opposite quadrant from a critical section, so "sys.lock.freeze()" and "sys.lock.thaw()" have little chance of having an efficient implementation on any platform. A shame. That would be pretty cool. - Gordon

Gordon McMillan wrote:
Actually, I think all that's needed is another global like the interpreter_lock in ceval.c. Since this lock is only accessed via abstract functions, I presume the unlock flag could easily be added. The locking section would only focus on Python, though: other threads could still be running provided they don't execute Python code, e.g. write data to a spooler. So it's not really the equivalent of a critical section as the one you can define in C. PS: I changed the subject line... hope this doesn't kill the thread ;) -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 158 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

Well, my personal opinion is that this is really quite wrong. The most obvious thing to me is that we are exposing an implementation detail we all would dearly like to see removed one day - the global interpreter lock. But even if we ignore that, it seems to me that you are describing an application abstraction, not a language abstraction. This thread started with Jim wanting a thread-safe, atomic list operation. This is not an unusual requirement (ie, a thread-safe, atomic operation), so languages give you access to primitives that let you build this. To my mind, you are asking for the equivilent of a C function that says "suspend all threads except me, cos Im doing something _really_ important". C does not provide that, and I have never thought it should. As Gordon said, Win32 has critical sections, but these are really just lightweight locks. I really dont see how Python is different - it gives you all the tools you need to build these abstractions. I really dont see what you are after that can not be done with a lock. If the performance is a problem, then to paraphrase the Timbot, it may be questionable if you are using Python appropriately in this case. Mark.

[The previous mail got truncated due to insufficient disk space; here is a summary...] Mark Hammond wrote:
The locked section may not be leading in the right direction, but it surely helps in situations where you cannot otherwise enforce useage of an object specific lock, e.g. for builtin file objects (some APIs insist on getting the real thing, not a thread safe wrapper). Here is a hack that let's you do much the same with an unpatched Python interpreter: sys.setcheckinterval(sys.maxint) # *) # >=10 Python OPs to flush the ticker counter and have the new # check interavl setting take effect: 0==0; 0==0; 0==0; 0==0 try: ...lock section... finally: sys.setcheckinterval(10) *) sys.setcheckinterval should really return the previous value so that we can reset the value to the original one afterwards. Note that the lock section may not call code which uses the Py_*_ALLOW_THREADS macros. -- Marc-Andre Lemburg ______________________________________________________________________ Y2000: 157 days left Business: http://www.lemburg.com/ Python Pages: http://www.lemburg.com/python/

[Marc writes]
Really, all this boils down to is that you want a Python-ish critical section - ie, a light-weight lock. This presumably would be desirable if it could be shown Python locks are indeed "heavy" - I know that from the C POV they may be considered as such, but I havent seen many complaints about lock speed from Python. So in an attempt to get _some_ evidence, I wrote a test program that used the Queue module to append 10000 integers then remove them all. I then hacked the queue module to remove all locking, and ran the same test. The results were 2.4 seconds for the non-locking version, vs 3.8 for the standard version. Without time (or really inclination <wink>) to take this further, it _does_ appear a native Python "critical section" could indeed save a few milli-seconds for a few real-world apps. So if we ignore the implementation details Marc started spelling, does the idea of a Python "critical section" appeal? Could simply be a built-in way of saying "no other _Python_ threads should run" (and of-course the "allow them again"). The semantics could be simply to ensure the Python program integrity - it need say nothing about the Python internal "state" as such. Mark.

Christian Tismer wrote:
Right. And the above code doesn't solve this problem. Tim's code *does* solve the problem. It's the code we were using. It is a bit verbose though.
I'm not sure, this thread has grown too fast :-)
Don't they all? Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.

Tim Peters wrote:
As a function (builtin or not), tryexcept will never get called if list.pop() raises an exception.
M.-A. Lemburg writes:
Oh well, forget the whole idea then. list.pop() is really not
Giving up already? Wouldn't you just love this as an expression operator (which could work)? How about: top = list.pop() excepting IndexError, default Hehehe... ;-) -Fred -- Fred L. Drake, Jr. <fdrake@acm.org> Corporation for National Research Initiatives
participants (9)
-
Barry A. Warsaw
-
Christian Tismer
-
Fred L. Drake
-
Gordon McMillan
-
Jim Fulton
-
M.-A. Lemburg
-
Mark Hammond
-
Skip Montanaro
-
Tim Peters