[Python-Dev] "Too many open files" errors on "x86 FreeBSD 7.2 3.x" buildbot

[David Bolen]

Victor Stinner <victor.stinner at haypocalc.com> writes:

> You wrote that the "POSIX" semaphore are very limited. Do it mean that there 
> is another kind of semaphore will an higher limit?

Well, I think the SYSV semaphores are either less limited or at least
more adjustable.  They've certainly been around longer in FreeBSD.
The POSIX semaphore support is not enabled by default in FreeBSD 7, so
I added loader.conf stuff to load them (as part of issue7272).  I
don't think the Python internals are using the SYSV semaphores though.
SYSV functions have no underscore (e.g., semget) whereas POSIX do
(sem_get).  Also, I believe only POSIX has named semaphores.

> I don't think that skipping the test is a good idea: it looks like a real bug 
> in (a limitation of) the ProcessPoolExecutor implementation on FreeBSD. Eg. 
> test_map fails on FreeBSD 7.2 with ProcessPoolExecutorTest which uses 
> self.executor = futures.ProcessPoolExecutor(max_workers=1): only one worker 
> process!
>
> It looks like it is possible to tune semaphore limits on FreeBSD, without 
> recompiling the kernel, by using boot loader option (kern.ipc.sem* options). 
> But ask the FreeBSD user to tune its boot loader options to use the 
> concurrent.futures module is not pratical :-)

Yeah, I guess the key question is if changing the limit is just needed
to get around an artifact of the test process (which I'm willing to do
for the buildbot), or if it would be needed to be able to use the
regular modules in practice.  If the latter, I doubt too many users
are going to jump through such hoops, particularly if it needs a
kernel rebuild, so we may need to make other choices in terms of
support under FreeBSD.

I'm also not entirely sure just what is the limiting factor.  I think
the kern.ipc.sem* options are for the SYSV semaphores, not POSIX, though
some of them do have a similar limit.  Some are adjustable by sysctl,
others by loader.conf.

The references I found were talking about a limit set explicitly
(#define SEM_MAX) in the kernel source (uipc_sem.c) which exports its
value (at least in 7.2) via the sysctl p1003_1b.sem_nsems_max, which
is read-only.  I got the impression they weren't adjustable even in
loader.conf, but haven't actually tried it yet myself.

It may be different in 8.x, but one email thread I found indicated
that the changes proposed to make the POSIX limits adjustable didn't
make the 8.1 cut (current release), though might make it in the next
8.x release.

-- David