[Idle-dev] Leaving subprocesses hanging

[Kurt B. Kaiser]

John Zelle <john.zelle at wartburg.edu> writes:

> By the way, this same bug appears under Linux as well, so It's not
>just the Windows subprocess issues in this case.

Yes.  Actually this bug is caused by the readline() mainloop not being
exited when the keyboard interrupt is serviced.  It's not a subprocess
issue, though the symptoms are slightly different when IDLE is run
without its subprocess  (-n switch).

> This seems to work fine on IDLE 1.03-1.04. I've been working with it
>for a couple days and have not noticed any problems. Unfortunately,
>last night I tried the same fix on IDLE 1.1, and I ran into some
>problems. Has the subprocess code undergone significant revision from
>1.04 to 1.1?

OK, I understand it's now working again for you.  No, no really
significant changes from 1.0.4 to 1.1, but see below.

I've checked in a patch (see below) which I believe fixes the problem.
It's of long standing, dating back to PyShell.py Rev 1.56 17Feb03
(before the release of IDLE 1.0, and before IDLEfork was merged into
Python), and it's my error.

It's amazing that no one complained before.  I'm adding it to my list
of monkey tests.

>>>I do not know yet if fixing this "special" case solves the larger
>>> instability problems.
>>>
>>
>>Shutting down the subprocess on Windows involves having it detect the
>>loss of connection and then successfully shut down the main thread,
>>the socket thread, and any threads associated with GUI communication.
>>It's complicated, and 'fixes' tend to regress older issues.
>>
>>
>>
> Yes, this does seem to be a particularly ticklish issue. That's why I
> used a Band-Aid approach.  Anecdotally, the general instability
> problem seems to have gotten a bit worse from version 1.0 to 1.03. We
> run 1.0 in our labs and have had only occasional problems, most of
> which I think are linked to quitting the shell when the subprocess is
> expecting input. I've heard from other schools that their problems
> have been so severe and frustrating (versions 1.03 and 1.04) that they
> are looking at alternatives.

This is certainly not what we want to hear!  In my view it's gotten 
better, but I did recently add Tk dialogs which pop up when the GUI
and the subprocess are having communication problems.  That would make
formerly silent errors (unless you are running from a command window)
more apparent and might add to an impression of instability.

The major remaining problem is that on Windows it's possible to
(partially) start more than one copy of IDLE by binding multiple
copies to the same port.  This seems to be a deficiency in the Windows
socket implementation.  Unfortunately, it's not easy to fix because
there is an additional problem: Windows doesn't remove abandoned
socket connections from its kernel table, which has a limit of only
five pending connections.

Therefore until I can fix this, it's important not to start more than
one copy of IDLE!

>>>I'm hoping some of the reader of edu-sig are trying it out and will
>>>let me know.
>>>
>>>In the mean time, I'd appreciate hearing if this is a known problem
>>>that is being worked on in IDLE. My preliminary test of IDLE 1.1 shows
>>>that it still suffers from the specific hang that I've identified.
>>>
>>It's an old problem which I thought was fixed since it hasn't been
>> reported recently.
>>
>>I hope that edu-sig and c.l.p will bring stability issues to this list
>>if they can't enter them on the Python Bug tracker.
>>
> OK, could you enlighten me a bit. Where should I have been looking to
> find what you've been working on, and where should we be reporting
> such issues. Obviously, I finally decided to post here because of the
> questions that I've been getting from colleagues at other schools. As
> I said, our problems have not been so serious. To be honest, I haven't
> even been able to track down who's even actively developing IDLE at
> this point. The python.org/idle page is not really that informative.

This is the list for IDLE development, so any concerned users are more
than welcome to discuss any issues regarding features, stability, and
future directions.  Please direct your colleagues over here.

Development is done on Sourceforge:

http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Lib/idlelib/

The bug tracker is at

http://sourceforge.net/tracker/?group_id=5470&atid=105470

Set the "category" field to IDLE, "status" Open, "assignee" and 
"group" to Any.

The patch tracker is at

http://sourceforge.net/tracker/?group_id=5470&atid=305470

same settings.

One of these days something should be done to update the IDLE home page
and the tutorial.

>>I'm also interested in discussing IDLE usability issues, especially
>>in an educational context.  That's appropriate for this list.
>>
>>
> Great, then perhaps I've found another "home."
>
> Thanks for all the work you and others already done on IDLE. It's been
> a great IDE for us. It's nice to be able to point people to Python and
> say that everything you need, IDE and all, is in the bundle.

That's right, and IDLE is a lot more capable than people may realize.
I use Emacs with python-mode about half the time, and IDLE the rest of
the time.  But I'm using Emacs less and less as I add capability to
IDLE.  When I'm not developing IDLE itself I rarely use Emacs for
Python these days, even though I basically live in Emacs for any other
editing or mail/news handling.  IDLE is faster with the F5 and
autosave.

But I agree that it's just /critical/ that IDLE work perfectly out of
the box.  There is nothing worse than having your exposition to a
beginner interrupted because there's a problem with the tool you are
using.  And beginners can certainly find the bugs....

I hope than anyone who has stability issues with IDLE will either
enter a bug on the tracker or post here.

I'll be away for a few days over Xmas, but check out the patch
and let me know.  Thanks for the report.

-- 
KBK

Index: PyShell.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/idlelib/PyShell.py,v
retrieving revision 1.93
diff -u -u -r1.93 PyShell.py
--- PyShell.py	21 Dec 2004 22:10:32 -0000	1.93
+++ PyShell.py	23 Dec 2004 04:19:10 -0000
@@ -910,6 +910,9 @@
                 parent=self.text)
             if response == False:
                 return "cancel"
+        if self.reading:
+            self.top.quit()
+        self.canceled = True
         self.closing = True
         # Wait for poll_subprocess() rescheduling to stop
         self.text.after(2 * self.pollinterval, self.close2)
@@ -974,10 +977,12 @@
         save = self.reading
         try:
             self.reading = 1
-            self.top.mainloop()
+            self.top.mainloop()  # nested mainloop()
         finally:
             self.reading = save
         line = self.text.get("iomark", "end-1c")
+        if len(line) == 0:  # may be EOF if we quit our mainloop with Ctrl-C
+            line = "\n"
         if isinstance(line, unicode):
             import IOBinding
             try:
@@ -987,10 +992,11 @@
         self.resetoutput()
         if self.canceled:
             self.canceled = 0
-            raise KeyboardInterrupt
+            if not use_subprocess:
+                raise KeyboardInterrupt
         if self.endoffile:
             self.endoffile = 0
-            return ""
+            line = ""
         return line
 
     def isatty(self):
@@ -1009,13 +1015,13 @@
             return "break"
         self.endoffile = 0
         self.canceled = 1
-        if self.reading:
-            self.top.quit()
-        elif (self.executing and self.interp.rpcclt):
+        if (self.executing and self.interp.rpcclt):
             if self.interp.getdebugger():
                 self.interp.restart_subprocess()
             else:
                 self.interp.interrupt_subprocess()
+        if self.reading:
+            self.top.quit()  # exit the nested mainloop() in readline()
         return "break"
 
     def eof_callback(self, event):