[Python-Dev] Avoiding file descriptors leakage in subprocess.Popen()

[Mark Seaborn]

Cameron Simpson <cs at zip.com.au> wrote:

> On 14Jun2009 16:42, Mark Seaborn <mrs at mythic-beasts.com> wrote:
> | I use a convenience function like this, so that GC takes care of the FDs:
> | 
> | def make_pipe():
> |     read_fd, write_fd = os.pipe()
> |     return os.fdopen(read_fd, "r"), os.fdopen(write_fd, "w")
> 
> Not guarrenteed to be timely. The try/except at least closes things as
> control passes out of the relevant scope. I don't think all pythons
> do immediate ref-counted GC.

Yep.  I don't mind closing FDs explicitly when it's easy to do so in a
try..finally, but it's not always simple.

There are two different problems with non-prompt closing of FDs:

 * Whether an FD has been closed is sometimes externally observable.
   e.g. Pipes and sockets notify the other end of the connection.
   Open file and directory FDs prevent filesystems from being
   unmounted.
 * FDs use up space in the process's FD table.

The second problem could be dealt with by running the GC when we get
EMFILE, or before any calls that allocate FDs when the FD table is
almost full, just as the GC runs when we "run out" of memory.

I wonder if this proposal could help:
'GC & The "Expensive Object" Problem'
http://www.eros-os.org/pipermail/e-lang/1999-May/002590.html

Mark