[New-bugs-announce] [issue10897] UNIX mmap unnecessarily dup() file descriptor
report at bugs.python.org
Wed Jan 12 16:16:19 CET 2011
New submission from Lorenz Huedepohl <lorenz at mpa-garching.mpg.de>:
The UNIX mmap version in
makes a call to dup() to duplicate the file descriptor that is passed for creating the memory-mapped region.
This way, I frequently hit the limit for the number of open file handles while creating mmap.mmap() instances, despite closing all my opened files after creating a mapping for them.
My application is scientific data (read: "large data" :-) analysis for which mmap() is very well suited together with numpy/scipy - however, the large number of files causes me to hit the resource limit on open file handles)
I propose to remove this dup(), which was apparently introduced in the process of fixing issue #728515, concerning incorrect behavior of the mmap.resize() method on UNIX, as it was feared the user could have closed the file handle already.
I think it should be the responsibility of the user not to close the file in question or either - if it needs to be closed - not to call the resize method later.
I should stress that a call to mmap(2) does not require an open file handle during the time of the mapping but that a duplicate of the file handle was only kept to allow .size() and .resize() to work. See the POSIX Programmer's Manual:
The mmap() function shall add an extra reference to the file
associated with the file descriptor fildes which is not removed
by a subsequent close() on that file descriptor. This reference
shall be removed when there are no more mappings to the file.
It would be great if this little nicety would translate better to Python!
components: Library (Lib)
title: UNIX mmap unnecessarily dup() file descriptor
type: resource usage
versions: Python 2.5, Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file20373/no_dup.patch
Python tracker <report at bugs.python.org>
More information about the New-bugs-announce