Python 2.6, multiprocessing module and BSD
philip at semanchuk.com
Wed Oct 22 18:14:16 CEST 2008
On Oct 22, 2008, at 11:37 AM, Jesse Noller wrote:
> On Wed, Oct 22, 2008 at 11:06 AM, Philip Semanchuk <philip at semanchuk.com
> > wrote:
>>> One oversight I noticed the multiprocessing module docs is that a
>> semaphore's acquire() method shouldn't have a timeout on OS X as
>> sem_timedwait() isn't supported on that platform. (You note OS X's
>> lack of
>> support for sem_getvalue() elsewhere.)
> Please file a ticket or update http://bugs.python.org/issue4012 so I
> don't loose it, my memory is increasingly lossy. Good catch.
I updated issue4012.
>> A question - how do you handle the difference in error messages on
>> platforms? For instance, sem_trywait() raises error 35, "Resource
>> temporarily unavailable" under OS X but error 11 under Ubuntu.
>> Right now I'm
>> just passing these up to the (Python) caller as OSErrors. This
>> makes it
>> really hard for the Python programmer to write cross-platform code.
> If you look at the code, we're pretty much raising OSError - it's
> possible we could enhance this in later versions, but given MP is
> supposed to be a cross-platform as possible and protect the user from
> the seedy underbelly of semaphores/pipes/etc - when an OSError does
> occur, it's generally a bug in our code, not the users.
Gotcha. I had a look at the code, and you're testing for errno ==
EAGAIN when sem_trywait() fails. This is correct for OS X and Ubuntu
(the platforms I mentioned above) and probably most other Unices. I
just don't have confidence that it will be true across all platforms,
esp. ones to which I don't have access like AIX and big-iron systems
that support POSIX. Maybe I am just taking defensive programming too
>> The only solution I can think of (which I haven't coded yet) is to
>> compile &
>> run a series of small C programs during setup.py that test things
>> sem_trywait() to see what errors occur, and provide those constants
>> to my
>> main .c module so that it can detect those errors exactly and wrap
>> them into
>> a specific, custom error for the Python caller.
>> Any thoughts on this?
> That's actually (while feeling hacky) a possibly sensible idea, the
> problem is is that you'd need to maintain documentation to tell users
> the exceptions for their platform.
If I pass all errors up as OSError, yes, that will be true. But that's
actually the situation I was trying to avoid.
By generating the error at install time, I can see exactly what
platform X returns in that situation (e.g. 11). Then I can create a
#define something like this:
#define ERRNO_WHEN_CALLING_SEM_TRYWAIT_ON_A_LOCKED_SEMAPHORE 11
My main .c module can then test for errno ==
ERRNO_WHEN_CALLING_SEM_TRYWAIT_ON_A_LOCKED_SEMAPHORE and wrap that in
a custom error, like posix_ipc.SemaphoreBusyError.
Again, maybe I'm just taking defensive programming too far. I've been
bitten by other cross-platform inconsistencies at the C API level and
I'm trying to get the jump on them here.
More information about the Python-list