Add an optional timeout to lock operations
Hello, I've submitted a patch (*) to add an optional timeout to locking operations (Lock.acquire() etc.). Since it's a pretty basic functionality, I would like to know if there was any good reason for not doing it. (*) http://bugs.python.org/issue7316 Thank you Antoine.
I think I can answer the "why" question: thread.c is *very* old code,
in fact it predates the posix threads standard. When we (actually
Sjoerd Mullender) wrote it, we had a number of OS-specific locking
APIs to work with and the API was designed to fit all of them. I don't
even recall the initial set, but I think it included SGI Irix and and
old version of SunOS. So then over time new platforms were added, but
adding a new method or parameter to the API was nearly impossible
because someone would have to find out how to implement the new
feature on all supported platforms. I think the number of platforms
has dwindled to two or three (Posix, Windows, and maybe one minority
OS?), so now's the time to do it. (IOW I think the idea of the patch
is fine.)
Will locks be interruptible with ^C? That is an oft-requested feature
which also wasn't supported at that time; what's the situation
nowadays?
--Guido
On Tue, Nov 17, 2009 at 3:46 AM, Antoine Pitrou
Hello,
I've submitted a patch (*) to add an optional timeout to locking operations (Lock.acquire() etc.). Since it's a pretty basic functionality, I would like to know if there was any good reason for not doing it.
(*) http://bugs.python.org/issue7316
Thank you
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
Guido van Rossum
I think the number of platforms has dwindled to two or three (Posix, Windows, and maybe one minority OS?), so now's the time to do it. (IOW I think the idea of the patch is fine.)
Thanks. (the minority OS would be OS/2, I think)
Will locks be interruptible with ^C? That is an oft-requested feature which also wasn't supported at that time; what's the situation nowadays?
They still aren't interruptible. From what I can read it may be possible to make them interruptible in the POSIX semaphore-based implementation, not in the POSIX condition variable-based implementation (which is used as a fallback when POSIX semaphores are not available, but I don't know whether this fallback is still useful). As for Windows I have absolutely no idea. Regards Antoine.
Antoine Pitrou wrote:
Guido van Rossum
writes: Will locks be interruptible with ^C? That is an oft-requested feature which also wasn't supported at that time; what's the situation nowadays?
They still aren't interruptible. From what I can read it may be possible to make them interruptible in the POSIX semaphore-based implementation, not in the POSIX condition variable-based implementation (which is used as a fallback when POSIX semaphores are not available, but I don't know whether this fallback is still useful).
I'm pretty sure at least some variants of *BSD still don't have OS level semaphores - their lack is the reason multiprocessing doesn't necessarily work everywhere that the threading module works (since mp needs semaphores in order to work its magic). Jesse would probably know the gory details. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Nov 18, 2009, at 5:38 AM, Nick Coghlan
Antoine Pitrou wrote:
Guido van Rossum
writes: Will locks be interruptible with ^C? That is an oft-requested feature which also wasn't supported at that time; what's the situation nowadays?
They still aren't interruptible. From what I can read it may be possible to make them interruptible in the POSIX semaphore-based implementation, not in the POSIX condition variable-based implementation (which is used as a fallback when POSIX semaphores are not available, but I don't know whether this fallback is still useful).
I'm pretty sure at least some variants of *BSD still don't have OS level semaphores - their lack is the reason multiprocessing doesn't necessarily work everywhere that the threading module works (since mp needs semaphores in order to work its magic).
Jesse would probably know the gory details.
Cheers, Nick.
Nick is right, many of the BSDs and FreeBSD up until fairly recently did not have named shared semaphore support. Still yet, the behavior is broken on some OSes such as OS X which you have to work around. I wouldn't recommend using them for threading right now, there's an assumption that threading "just works" unlike multiprocessing, which people understand has caveats. Jesse
Jesse Noller
Nick is right, many of the BSDs and FreeBSD up until fairly recently did not have named shared semaphore support. Still yet, the behavior is broken on some OSes such as OS X which you have to work around.
The core locking support only uses anonymous semaphores (they don't need to be shared accross processes obviously). Regards Antoine.
On Wed, Nov 18, 2009 at 8:50 AM, Antoine Pitrou
Jesse Noller
writes: Nick is right, many of the BSDs and FreeBSD up until fairly recently did not have named shared semaphore support. Still yet, the behavior is broken on some OSes such as OS X which you have to work around.
The core locking support only uses anonymous semaphores (they don't need to be shared accross processes obviously).
Regards
Antoine.
That's what I get for replying from an iphone over morning coffee. See also http://semanchuk.com/philip/sysv_ipc/ for Philip Semanchuk semaphore/IPC work as well. Given both multiprocessing and Philip's work are concerned about the shared aspects, I think you're right - FBSD and others support the posix semaphores for non-ipc stuff ok AFAIK (recent benchmark http://www.ioremap.net/node/216). jesse
Antoine Pitrou
I've submitted a patch (*) to add an optional timeout to locking operations (Lock.acquire() etc.). Since it's a pretty basic functionality, I would like to know if there was any good reason for not doing it.
I always assumed it was because as a least-common-denominator set of functionality, some platforms didn't have the necessary support. Providing the discussion on this ends up with the an implementation being accepted, I'd absolutely love to see this then leveraged by threading.Condition.wait() rather rather than the current poll-with-timed-sleep approach. The overhead (CPU, but most notably latency) of that approach (which also directly impacts Queues) has historically been my top reason in various applications on Windows to have to implement my own Queue or Condition structures using native Windows calls. -- David
Hello,
I've submitted a patch (*) to add an optional timeout to locking operations (Lock.acquire() etc.). Since it's a pretty basic functionality, I would like to know if there was any good reason for not doing it.
I always assumed it was because as a least-common-denominator set of functionality, some platforms didn't have the necessary support.
Guido's answer says so indeed. Now py3k only needs to support POSIX and Windows (and, provided Andrew MacIntyre maintains the port, OS/2), which both have standard support for waiting-with-timeout.
Providing the discussion on this ends up with the an implementation being accepted, I'd absolutely love to see this then leveraged by threading.Condition.wait() rather rather than the current poll-with-timed-sleep approach.
Agreed. The current patch (as proposed on http://bugs.python.org/issue7316) includes exactly that. Poll-with-timed-sleep is especially sub-optimal on laptops where short but frequent wakeups can cause a significant decrease in battery life. (the Linux community has been chasing this using powertop: http://www.lesswatts.org/projects/powertop/) Feel free to test or review if you're interested. Regards Antoine.
participants (5)
-
Antoine Pitrou -
David Bolen -
Guido van Rossum -
Jesse Noller -
Nick Coghlan