Py2.7/FreeBSD: maximum number of open files

[Tobias Oberstein]

I am trying to convince Python to open more than 32k files .. this is on FreeBSD.

Now I know I have to set appropriate limits .. I did:

$ sysctl kern.maxfiles
kern.maxfiles: 204800
$ sysctl kern.maxfilesperproc
kern.maxfilesperproc: 200000
$ sysctl kern.maxvnodes
kern.maxvnodes: 200000
$ ulimit
unlimited

Here is what happens with a Python freshly built from sources .. it'll tell me I can open 200k files .. but will bail out at 32k:

$ ./local/bin/python -V
Python 2.7.2
$ ./local/bin/python
Python 2.7.2 (default, Nov 14 2011, 16:41:56)
[GCC 4.2.1 20070719  [FreeBSD]] on freebsd8
Type "help", "copyright", "credits" or "license" for more information.
>>> import resource
>>> resource.getrlimit(resource.RLIMIT_NOFILE)
(200000L, 200000L)
>>> resource.getrlimit(resource.RLIMIT_OFILE)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute 'RLIMIT_OFILE'
>>> import resource
>>>
>>> max = resource.getrlimit(resource.RLIMIT_NOFILE)
>>> cnt = 0
>>> print "maximum FDs", max
maximum FDs (200000L, 200000L)
fds = []

while cnt < max:
   f = open("/tmp/test1/test_%d" % cnt, "w")
   f.write("test")
   fds.append(f)
   cnt += 1
   if cnt % 1000 == 0:
      print "opened %d files" % cnt

print "ok, created %d files" % cnt
>>>
>>> fds = []
>>>
>>> while cnt < max:
...    f = open("/tmp/test1/test_%d" % cnt, "w")
...    f.write("test")
...    fds.append(f)
...    cnt += 1
...    if cnt % 1000 == 0:
...       print "opened %d files" % cnt
...
opened 1000 files
opened 2000 files
opened 3000 files
opened 4000 files
opened 5000 files
opened 6000 files
opened 7000 files
opened 8000 files
opened 9000 files
opened 10000 files
opened 11000 files
opened 12000 files
opened 13000 files
opened 14000 files
opened 15000 files
opened 16000 files
opened 17000 files
opened 18000 files
opened 19000 files
opened 20000 files
opened 21000 files
opened 22000 files
opened 23000 files
opened 24000 files
opened 25000 files
opened 26000 files
opened 27000 files
opened 28000 files
opened 29000 files
opened 30000 files
opened 31000 files
opened 32000 files
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
IOError: [Errno 24] Too many open files: '/tmp/test1/test_32765'
>>> print "ok, created %d files" % cnt
ok, created 32765 files
>>>