[Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD

[Philip Semanchuk]

On Dec 29, 2010, at 8:17 AM, Victor Stinner wrote:

> Hi,
> 
> FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of
> a concurrent.futures failure. The problem is that
> test_concurrent_futures uses many (multiprocessing) POSIX semaphores,
> whereas POSIX semaphores support in FreeBSD is recent and limited. We
> have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead
> of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See:
> 
> * http://bugs.python.org/issue10348
> * "Too many open files" errors on "x86 FreeBSD 7.2 3.x" buildbot
>   ^-- thread in python-dev opened last month
> 
> I would like to know if it should be considered as a release blocker.
> Georg Brandl said yes on IRC. Does anyone know SysV API? I tried to
> write a patch, but I never used semaphores (POSIX or SysV).
> 
> There is a third party module which looks complete and stable:
> http://semanchuk.com/philip/sysv_ipc/
> 
> It is released under the BSD license. It supports semaphores, but also
> shared memory and message queues. We don't need all of those, semaphores
> would be enough. I added its author (Philip Semanchuk) to this thread.

Hi all,
What Victor says above is correct, although I wasn't aware that POSIX IPC under FreeBSD 7.2 was still having problems. Prior to 7.2 it was broken but 7.2 worked OK in my limited testing. In any case, the sysv_ipc module is mine and it's mature and you're welcome to pillage it in whole or in part.


> I don't know how we should decide to use POSIX or SysV semaphores. It
> looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD
> based OSes), and POSIX is preferred on Linux.

Hmmm, "preferred" is a tricky word here. I would phrase it slightly differently: POSIX IPC is preferred everywhere (by me, at least) because it's a more modern API. However, SysV IPC is fully supported everywhere while the same can't be said about POSIX IPC. 

Speaking of POSIX IPC, I also have a posix_ipc module that's quite similar to sysv_ipc and the platform notes in the documentation tell one what support to expect for POSIX IPC under various platforms:
http://semanchuk.com/philip/posix_ipc/#platforms

Cheers
Philip