[ python-Bugs-768649 ] popen4 doesn't close filedescriptors when in Threads

[SourceForge.net]

Bugs item #768649, was opened at 2003-07-09 15:36
Message generated for change (Comment added) made by facundobatista
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=768649&group_id=5470

Category: Python Library
Group: Python 2.2.2
Status: Open
Resolution: None
Priority: 5
Submitted By: martin doudoroff (mdoudoroff)
Assigned to: Nobody/Anonymous (nobody)
Summary: popen4 doesn't close filedescriptors when in Threads

Initial Comment:

Running the following code under Linux will result in a
crash on the 508th thread started. The error is

    OSError: [Errno 24] Too many open files

The nature of the bug seems to be that Python isn't
closing filedescriptors cleanly when running a thread.

---------------------------------------
import os
from threading import Thread

class Crash(Thread):
    def run(self):
        a = os.popen4('ls')
        b = a[1].read()

        # uncommenting these lines fixes the problem
        #     but this isn't really documented as far as
        #     we can tell...
        # a[0].close()
        # a[1].close()

for i in range(1000):
    t = Crash()
    t.start()

    while t.isAlive():
         pass

    print i
---------------------------------------

The same code without threads (Crash as a plain class)
doesn't crash, so the descriptor must be getting taken
care of when the run() method is exited.

import os

class Crash:
    def run(self):
        a = os.popen4('ls')
        b = a[1].read()

for i in range(1000):
    t = Crash()
    t.run()

    print i


----------------------------------------------------------------------

>Comment By: Facundo Batista (facundobatista)
Date: 2005-01-15 17:48

Message:
Logged In: YES 
user_id=752496

Works fine to me:

Python 2.3.4 (#1, Oct 26 2004, 16:42:40)
[GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] on linux2

with glibc-2.3.4-2

----------------------------------------------------------------------

Comment By: Facundo Batista (facundobatista)
Date: 2005-01-15 17:48

Message:
Logged In: YES 
user_id=752496

Please, could you verify if this problem persists in Python 2.3.4
or 2.4?

If yes, in which version? Can you provide a test case?

If the problem is solved, from which version?

Note that if you fail to answer in one month, I'll close this bug
as "Won't fix".

Thank you! 

.    Facundo

----------------------------------------------------------------------

Comment By: Andrew Gaul (gaul)
Date: 2003-10-01 15:51

Message:
Logged In: YES 
user_id=139865

Duplicated with Python 2.3 on Red Hat 7.3 using
glibc-2.2.5-43.  Popen3.{poll,wait} are written under the
incorrect assumption that waitpid can monitor any process in
the same process group, when it only works for immediate
children.  _active.remove is never called, so Popen3 objects
are never destroyed and the associated file descriptors are
not returned to the operating system.

A general solution for Popen[34] is not obvious to me.  With
patch #816059, popen2.popen[234] plugs the _active leak,
which in turn returns the file descriptors to the operating
system when the file objects that popen2.popen[234] return
go out of scope.

----------------------------------------------------------------------

Comment By: Neal Norwitz (nnorwitz)
Date: 2003-07-10 00:20

Message:
Logged In: YES 
user_id=33168

I can't duplicate this on Redhat 9.  What OS, what version
of glibc and what kernel are you using?  Does it always
crash on the 508th iteration?

I tested with both 2.2.3 and 2.3b2 from CVS without
problems.  I even used ulimit to set my open files to 10.

Can you try the patch in bug #761888 to see if that helps? 
http://pythong.org/sf/761888 

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=768649&group_id=5470