[New-bugs-announce] [issue10897] UNIX mmap unnecessarily dup() file descriptor

[Lorenz Huedepohl]

New submission from Lorenz Huedepohl <lorenz at mpa-garching.mpg.de>:

The UNIX mmap version in

  Modules/mmapmodule.c 

makes a call to dup() to duplicate the file descriptor that is passed for creating the memory-mapped region.

This way, I frequently hit the limit for the number of open file handles while creating mmap.mmap() instances, despite closing all my opened files after creating a mapping for them.

My application is scientific data (read: "large data" :-) analysis for which mmap() is very well suited together with numpy/scipy - however, the large number of files causes me to hit the resource limit on open file handles)

I propose to remove this dup(), which was apparently introduced in the process of fixing issue #728515, concerning incorrect behavior of the mmap.resize() method on UNIX, as it was feared the user could have closed the file handle already.

I think it should be the responsibility of the user not to close the file in question or either - if it needs to be closed - not to call the resize method later.

I should stress that a call to mmap(2) does not require an open file handle during the time of the mapping but that a duplicate of the file handle was only kept to allow .size() and .resize() to work. See the POSIX Programmer's Manual:

  The mmap() function shall add an extra reference to the file
  associated with the file descriptor fildes which is not removed
  by a subsequent close() on that file descriptor. This reference
  shall be removed when there are no more mappings to the file.

It would be great if this little nicety would translate better to Python!

Regards,
   Lorenz

----------
components: Library (Lib)
files: no_dup.patch
keywords: patch
messages: 126104
nosy: lorenz
priority: normal
severity: normal
status: open
title: UNIX mmap unnecessarily dup() file descriptor
type: resource usage
versions: Python 2.5, Python 2.6, Python 2.7, Python 3.1, Python 3.2, Python 3.3
Added file: http://bugs.python.org/file20373/no_dup.patch

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue10897>
_______________________________________