Py2.7/FreeBSD: maximum number of open files

[Chris Torek]

In article <mailman.2711.1321299276.27778.python-list at python.org>
Christian Heimes  <lists at cheimes.de> wrote:
>Am 14.11.2011 19:28, schrieb Tobias Oberstein:
>> Thanks! This is probably the most practical option I can go.
>> 
>> I've just tested: the backported new IO on Python 2.7 will indeed
>> open >32k files on FreeBSD. It also creates the files much faster.
>> The old, non-monkey-patched version was getting slower and
>> slower as more files were opened/created ..
>
>I wonder what's causing the O(n^2) behavior. Is it the old file type or
>BSD's fopen() fault?

It is code in libc.  My old stdio (still in use on FreeBSD) was
never designed to be used in situations with more than roughly 256
file descriptors -- hence the "short" in the file number field.
(The OS used to be full of other places that kept the maximum
allowable file descriptor fairly small, such as the on-stack copies
of fd_set objects in select() calls.)

You will want to redesign the code that finds or creates a free
"FILE" object, and probably some of the things that work with
line-buffered FILEs (specifically calls to _fwalk() when reading
from a line-buffered stream).
-- 
In-Real-Life: Chris Torek, Wind River Systems
Intel require I note that my opinions are not those of WRS or Intel
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W)  +1 801 277 2603
email: gmail (figure it out)      http://web.torek.net/torek/index.html