[ python-Bugs-1199282 ] subprocess _active.remove(self) self not in list _active

Sun Apr 30 21:52:46 CEST 2006

Bugs item #1199282, was opened at 2005-05-10 18:24
Message generated for change (Comment added) made by gbrandl
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1199282&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Python Library
Group: Python 2.4
>Status: Pending
>Resolution: Fixed
Priority: 5
Submitted By: cheops (atila-cheops)
Assigned to: Peter Ã
strand (astrand)
Summary: subprocess _active.remove(self) self not in list _active

Initial Comment:
I start a subprocess in a seperate thread (25 concurrent 
threads)
in some of the threads the following error occurs

Exception in thread Thread-4:
Traceback (most recent call last):
  File "C:\Python24\lib\threading.py", line 442, in 
__bootstrap
    self.run()
  File "upgrade.py", line 45, in run
    returncode = p.wait()
  File "C:\Python24\lib\subprocess.py", line 765, in wait
    _active.remove(self)
ValueError: list.remove(x): x not in list

this is the code that starts the subprocess and where I 
wait for the result

p = subprocess.Popen('command', \
                             stdin=None, 
stdout=subprocess.PIPE, \
                             stderr=subprocess.STDOUT, 
universal_newlines=True)
returncode = p.wait()
errormessage = p.stdout.readlines()

----------------------------------------------------------------------

>Comment By: Georg Brandl (gbrandl)
Date: 2006-04-30 19:52

Message:
Logged In: YES 
user_id=849994

The error causing "_active.remove(self)" call was removed
from wait() with patch 1467770, so I think this is fixed.

----------------------------------------------------------------------

Comment By: cheops (atila-cheops)
Date: 2006-04-11 20:10

Message:
Logged In: YES 
user_id=1276121

there are 2 definitions of the function poll, one for 
windows and one for POSIX systems
in the windows version _deadlock is not used
in the POSIX version _deadlock is used

see also modification made by loewis in committed version 
45234

----------------------------------------------------------------------

Comment By: HVB bei TUP (hvb_tup)
Date: 2006-04-11 07:21

Message:
Logged In: YES 
user_id=1434251

I looked at the patch #1467770 you mentioned and 
downloaded the patch.

Is it correct that _deadlock variable is used in the 
definition of poll but it is not in the argument list?

----------------------------------------------------------------------

Comment By: cheops (atila-cheops)
Date: 2006-04-10 14:57

Message:
Logged In: YES 
user_id=1276121

see patch #1467770

----------------------------------------------------------------------

Comment By: Tristan Faujour (tfaujour)
Date: 2006-03-29 13:50

Message:
Logged In: YES 
user_id=1488657

> Simply list operations such as remove() and append() are
> thread safe,

OK, my apologies... I did not know.

I did some more tests. On Linux, I found lots of:

  File "./subprocess.py", line 428, in call
    return Popen(*args, **kwargs).wait()
  File "./subprocess.py", line 1023, in wait
    pid, sts = os.waitpid(self.pid, 0)
OSError: [Errno 10] No child processes

The try...except solution does not handle this (since we are
in the "posix" section).

In my opinion, the call to _cleanup() in Popen.__init__() is
useless (it just checks if older processes have stopped when
a new one is started. I cannot see why it could be
mandatory) and it is the root of this bug.

I commented it out (line 506) and retried my tests on Linux
& Windows plateforms: I had no exception at all.

----------------------------------------------------------------------

Comment By: Peter Ã
strand (astrand)
Date: 2006-03-29 05:11

Message:
Logged In: YES 
user_id=344921

>I think accesses to _active should be serialized in a
>thread-safe way. _active could be a Queue (from the Queue
>module) for example

Simply list operations such as remove() and append() are
thread safe, so there should be no need for a Queue. Also, a
Queue cannot be used since the subprocess module needs to be
compatible with Python 2.2. 

----------------------------------------------------------------------

Comment By: Tristan Faujour (tfaujour)
Date: 2006-03-28 23:17

Message:
Logged In: YES 
user_id=1488657

I am running into the same problem on a Windows 2k/XP
plateform with a multi-threaded application. It occurs randomly.

It has never appened (yet) on a Linux plateform.

I think accesses to _active should be serialized in a
thread-safe way. _active could be a Queue (from the Queue
module) for example.

----------------------------------------------------------------------

Comment By: HVB bei TUP (hvb_tup)
Date: 2006-01-25 08:10

Message:
Logged In: YES 
user_id=1434251

Wouldn't it be best to completely remove any references 
to "_active" and "_cleanup"?

----------------------------------------------------------------------

Comment By: cheops (atila-cheops)
Date: 2006-01-25 07:08

Message:
Logged In: YES 
user_id=1276121

As suggested by astrand
adding a try ... except clause in the file subprocess.py did
the job
I had to add that try ... except clause in 2 places
if you look in the file there are 2 instances were
list.remove(x) occurs unprotected.

try:
 list.remove(x)
except:
 pass

I have worked with 2.4.0, 2.4.1 and 2.4.2 and all three
needed the patch.
Hope this helps.

----------------------------------------------------------------------

Comment By: HVB bei TUP (hvb_tup)
Date: 2006-01-23 16:34

Message:
Logged In: YES 
user_id=1434251

BTW: In my case, I call python.exe from a Windows service.

----------------------------------------------------------------------

Comment By: HVB bei TUP (hvb_tup)
Date: 2006-01-23 16:30

Message:
Logged In: YES 
user_id=1434251

I have a similar problem.
Python 2.4.1 under MS Windows 2003,
Multi-Threaded application (about concurrent 10 threads).

In my case the same error occurs during _cleanup called 
from __init__ :

File "E:\lisa_ins\ewu\coop\reporting\python\tup_lisa\util\t
hreadutil.py", line 582, in callSubProcess
    creationflags = creationflags
  File "C:\Python24\lib\subprocess.py", line 506, in 
__init__
    _cleanup()
  File "C:\Python24\lib\subprocess.py", line 414, in 
_cleanup
    inst.poll()
  File "C:\Python24\lib\subprocess.py", line 755, in poll
    _active.remove(self)
ValueError: list.remove(x): x not in list

Is there a work-around?

----------------------------------------------------------------------

Comment By: cheops (atila-cheops)
Date: 2005-09-19 09:29

Message:
Logged In: YES 
user_id=1276121

I noticed this bug under windows
to reproduce the bug, I attached the script I use, but this 
needs input, I tried with a simpler example (pinging a number 
of host concurrently) but that did not cause the bug.

----------------------------------------------------------------------

Comment By: Peter Ã
strand (astrand)
Date: 2005-06-23 16:03

Message:
Logged In: YES 
user_id=344921

I believe it should be sufficient to add a try...except
clause around _active.remove(). Can you upload a complete
example that triggers the bug? Have you noticed this bug on
Windows, UNIX or both platforms?

----------------------------------------------------------------------

Comment By: cheops (atila-cheops)
Date: 2005-05-12 10:17

Message:
Logged In: YES 
user_id=1276121

this might be related to bug 1183780

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1199282&group_id=5470