[Python-Dev] "Too many open files" errors on "x86 FreeBSD 7.2 3.x" buildbot

[Victor Stinner]

On Saturday 06 November 2010 22:36:44 you wrote:
> I couldn't find the matching failures that you're talking about, but
> then I figured out you mean the FreeBSD7 (7.2) buildbot, not the
> FreeBSD (6.4) buildbot ....

Search "test_concurrent_futures" in:
http://www.python.org/dev/buildbot/builders/x86%20FreeBSD%207.2%203.x/builds/1154/steps/test/logs/stdio

I specified "x86 FreeBSD 7.2 3.x" in the email title.

> (...)
> I noticed that the failures seem to always be on a semaphore call.
> Some quick googling found a few references that seems to imply that
> the number of posix semaphores are very limited (like 30), and can't
> be changed without recompiling the kernel from source.  So that's not
> so big a threshold for the tests to have perhaps started crossing
> since issue7272 was fixed.  Certainly seems more likely than 3000+
> files or 1500+ processes.

Nice catch. The problem is the total number of semaphores: I reproduced the 
bug in my FreeBSD 8 VM. The first test fails at the creation of the 31th 
semaphore.

The first failing test if test_all_completed. And it looks like this test 
doesn't destroy the semaphore at exit: my counter (based on __init__/__del__) 
is still at 15 when exiting the test!

> I wonder if it's possible to deduce if this started recently or not?
> The web buildbot interface doesn't go back that far, and an additional
> complexity is that the FreeBSD builds tend to have various errors
> somewhat consistently over time, but perhaps there are server logs we
> can grep for this particular error?

No idea.

> Not sure if the best approach at this point is to see if the tests can
> use fewer semaphores, skip these tests under FreeBSD 7 like 6, or if
> it's important enough to compile a new kernel with a higher semaphore
> limit.

You wrote that the "POSIX" semaphore are very limited. Do it mean that there 
is another kind of semaphore will an higher limit?

I don't think that skipping the test is a good idea: it looks like a real bug 
in (a limitation of) the ProcessPoolExecutor implementation on FreeBSD. Eg. 
test_map fails on FreeBSD 7.2 with ProcessPoolExecutorTest which uses 
self.executor = futures.ProcessPoolExecutor(max_workers=1): only one worker 
process!

It looks like it is possible to tune semaphore limits on FreeBSD, without 
recompiling the kernel, by using boot loader option (kern.ipc.sem* options). 
But ask the FreeBSD user to tune its boot loader options to use the 
concurrent.futures module is not pratical :-)

Victor