[Idle-dev] Leaving subprocesses hanging

[John Zelle]

Kurt,

Thanks for the reply on this. Very helpful info.

Kurt B. Kaiser wrote:

>John Zelle <john.zelle at wartburg.edu> writes:
>
>  
>
>>I'm new to the idle-dev list, but felt I needed to post here for some
>>assistance. I am in communication with a number of college faculty
>>using Python to teach introductory programming courses. There has been
>>growing frustration that newer versions of IDLE seem to be less
>>stable. In particular, especially under Windows XP, there are times
>>when IDLE becomes unresponsive (hangs). This seems to be related to
>>losing proper communication with its subprocess. The only way to get
>>things cleaned up again is to go into the process manager and kill the
>>stray pythonw processes. Unfortunately, the behavior is sporadic and
>>hard to pin down.
>>    
>>
>
>This is the major difficulty with using subprocess execution on
>Windows.
>
>I spent quite a bit of time on the 'hanging subprocess' bug a year and
>a half ago.  The bug was so persistent I've named it 'Freddy'.
>
>My priority with IDLE is to choose stability over bells/whistles.
>IDLE is intended to be used for introductory programming as well as by
>advanced programmers, but beginners take precedence in my view.
>
>At one time (May '03), I was interrupting the subprocess before
>restarting it but the resulting traceback was slowing down the restart
>considerably.  So I redesigned the subprocess so that a loss of socket
>connection to the GUI would invoke an interrupt of the main thread
>(which is executing the user's code).  The GUI causes the subprocess to
>restart by simply closing the socket.  On Unix, a TERM signal is also
>sent to the subprocess, but that's not available on Windows.
>
>  
>
Stability would also be my first choice in IDLE. It's just so crucial 
for new programmers to have an evironment that responds the way that 
they expect it to. I'm glad to hear that someone has been working on 
'Freddy.'

>>I did a little poking around in the idle-dev archives and did not
>>see anything about this problem, so I did a bit of experimenting on
>>my own. One simple case that causes IDLE to hang is to simply close
>>the shell while the program that it is executing is waiting for
>>input. After IDLE asks if it should kill the program, it usually
>>hangs. If it doesn't do this the first time, a couple of <f5>-close
>>sequences will produce it.
>>    
>>
>
>OK, so I assume you are using something like
>
>.>>> a = raw_input('prompt: ')
>
>and then either clicking the shell window close widget or using F5.
>That's actually two different scenarios.  I'll go off and duplicate
>them.
>
>  
>
This is basically it. I actually put this line in a script, load the 
script and <F5> it. When the shell is waiting for input, closing the 
shell window either by clicking on the window's close box or by 
selecting close from the file menu will produce the error. From what I 
saw in the code, these are equivalent actions. I think as of version 
1.04, rather than a pure hang what happens is that the next time <F5> is 
done, IDLE pops up the message window about the socket being in use.

If the shell window is waiting for user input and you do <F5> in the 
script window, the shell does do a clean restart, although it seems as 
if one needs to hit <return> in the shell window before the prompt will 
display again.

>>I modified PyShell.py in IDLE 1.03 by adding a call to
>>cancel_callback() before closing the shell window. This seems to
>>solve the problem (perhaps in an inelegant way). 
>>    
>>
>
>Where, exactly, did you put this code?  Did it fix the problem
>you mentioned with <F5>-close sequences?
>
>  
>
I modified the PyShell close method:

    def close(self):
        "Extend EditorWindow.close()"
        if self.executing:
            response = tkMessageBox.askokcancel(
                "Kill?",
                "The program is still running!\n Do you want to kill it?",
                default="ok",
                parent=self.text)
            if response == False:
                return "cancel"
        self.cancel_callback() # This line inserted by JMZ
        self.closing = True
        # Wait for poll_subprocess() rescheduling to stop
        self.text.after(2 * self.pollinterval, self.close2)

This fix does solve the problem that I describe. I noticed that sending 
a keyboard interrupt before closing the shell seemed to take care of the 
problem, so I just inserted the cancel_callback() into the close 
sequence. It seemed the safest way to go (without understanding the 
intricacies of the subprocess code). It does have the drawback of 
producing a bit of "chatter" in the shell window just before it goes away.

By the way, this same bug appears under Linux as well, so It's not just 
the Windows subprocess issues in this case.

This seems to work fine on IDLE 1.03-1.04. I've been working with it for 
a couple days and have not noticed any problems. Unfortunately, last 
night I tried the same fix on IDLE 1.1, and I ran into some problems. 
Has the subprocess code undergone significant revision from 1.04 to 1.1?

>>I do not know yet if fixing this "special" case solves the larger
>>instability problems. 
>>    
>>
>
>Shutting down the subprocess on Windows involves having it detect the
>loss of connection and then successfully shut down the main thread,
>the socket thread, and any threads associated with GUI communication.
>It's complicated, and 'fixes' tend to regress older issues.
>
>  
>
Yes, this does seem to be a particularly ticklish issue. That's why I 
used a Band-Aid approach.  Anecdotally, the general instability problem 
seems to have gotten a bit worse from version 1.0 to 1.03. We run 1.0 in 
our labs and have had only occasional problems, most of which I think 
are linked to quitting the shell when the subprocess is expecting input. 
I've heard from other schools that their problems have been so severe 
and frustrating (versions 1.03 and 1.04) that they are looking at 
alternatives.

>>I'm hoping some of the reader of edu-sig are trying it out and will
>>let me know.
>>
>>In the mean time, I'd appreciate hearing if this is a known problem
>>that is being worked on in IDLE. My preliminary test of IDLE 1.1 shows
>>that it still suffers from the specific hang that I've identified.
>>    
>>
>
>It's an old problem which I thought was fixed since it hasn't been
>reported recently. 
>
>I hope that edu-sig and c.l.p will bring stability issues to this list
>if they can't enter them on the Python Bug tracker.
>
>  
>
OK, could you enlighten me a bit. Where should I have been looking to 
find what you've been working on, and where should we be reporting such 
issues. Obviously, I finally decided to post here because of the 
questions that I've been getting from colleagues at other schools. As I 
said, our problems have not been so serious. To be honest, I haven't 
even been able to track down who's even actively developing IDLE at this 
point. The python.org/idle page is not really that informative.

>I'm also interested in discussing IDLE usability issues, especially
>in an educational context.  That's appropriate for this list.
>
>  
>
Great, then perhaps I've found another "home."

Thanks for all the work you and others already done on IDLE. It's been a 
great IDE for us. It's nice to be able to point people to Python and say 
that everything you need, IDE and all, is in the bundle.

--John