[Idle-dev] [ idlefork-Bugs-677263 ] On W32 Can't Terminate Subprocess Not Doing IO

Mon, 26 May 2003 02:07:08 -0700

Bugs item #677263, was opened at 2003-01-30 05:02
Message generated for change (Comment added) made by hfoffani
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=109579&aid=677263&group_id=9579

Category: None
Group: None
Status: Open
Resolution: Fixed
Priority: 8
Submitted By: Bruce Sherwood (bsherwood)
Assigned to: Kurt B. Kaiser (kbk)
Summary: On W32 Can't Terminate Subprocess Not Doing IO

Initial Comment:
With 0.9a2 it is still the case that something is very 
wrong with the run environment for VPython programs, 
at least on Windows. I run the simplest program

from visual import *
sphere()

Next I kill the graphics display window (by clicking in the 
close box), then type in the shell "2+2". I get the 
following:

Traceback (most recent call last):
  File "C:\Python23\Lib\site-packages\idleforklib\rpc.py", 
line 187, in asynccall
    self.putmessage((seq, request))
  File "C:\Python23\Lib\site-packages\idleforklib\rpc.py", 
line 289, in putmessage
    n = self.sock.send(s)
  File "<string>", line 1, in send
error: (10054, 'Connection reset by peer')

I then try to kill the shell window and get a dialog box 
asking this:

"The program is still running; do you want to kill it?"

I say I want to kill it, the dialog box goes away, but 
nothing happens to the shell window. The only way I can 
kill Python is with the Windows task manager. Even 
independent of what may be special about a VPython 
program, shouldn't IDLE honor the request to kill?

I emphasize that with the original idlefork when one 
killed a VPython graphics window you were in a good 
state. Another possible hint at the source of the new 
problem is that with the original idlefork after F5 to run I 
could press F5 and the existing graphics window would 
be killed and I'd get a new spawned process. Now I 
seem to get two spawned processes in this situation.

http://vpython.org is a place to get the VPython stuff. I 
have no idea whether the problem is in some sense with 
VPython or with the new idlefork.

----------------------------------------------------------------------

Comment By: Hernan Martinez Foffani (hfoffani)
Date: 2003-05-26 11:07

Message:
Logged In: YES 
user_id=112690

Freddy's gone. :-)
Tested in 2 different PCs (both Win2K), works ok.

(Just for the record, Kaspersky Antivirus seriously affects 
Python's performance.)

Thanks and Regards,
-Hernan

----------------------------------------------------------------------

Comment By: Kurt B. Kaiser (kbk)
Date: 2003-05-24 23:17

Message:
Logged In: YES 
user_id=149084

OK, I checked in an update.  Give it a shot.

----------------------------------------------------------------------

Comment By: Kurt B. Kaiser (kbk)
Date: 2003-05-20 05:22

Message:
Logged In: YES 
user_id=149084

Hernan, no, save your testing time for when I think I've
got it nailed.  That should be soon, I think it's working
here on my XP box.  I'll let you know here.

The problem would have started when I started processing
user code in MainThread instead of SockThread on 
11 May, but that's not too important at this point.

----------------------------------------------------------------------

Comment By: Hernan Martinez Foffani (hfoffani)
Date: 2003-05-19 22:58

Message:
Logged In: YES 
user_id=112690

kurt:
do you want me to test a special cvs version between the
current one and 0.9a2 ?
it seems that the problem fires since some version in between.

-H.

----------------------------------------------------------------------

Comment By: Hernan Martinez Foffani (hfoffani)
Date: 2003-05-19 15:37

Message:
Logged In: YES 
user_id=112690

more data:

file test2plus2.py:
   print 2 + 2

on a different win2K pro where there were no python up to 
date, did the following:

- install python 2.2.2, then idlefork-0.92, load and run 
test2plus2.py
  result: only 2 pythonw.exe, *always*.

- install python 2.3, then idlefork-0.92 (it works, kurt), load 
and run test2plus2.py
  result: only 2 pythonw.exe, *always*.

- idlefork from CVS May,19th. on python 2.3 Lib/site-
packages/idle ; interrupt.pyd as posted; load and run 
test2plus2.py
  result: one pythonw.exe adds to the process list on every F5

i don't know if it's relevant, but this PC has a Visual 
Studio .NET installed. no debug kernel, though.

-H.

----------------------------------------------------------------------

Comment By: Kurt B. Kaiser (kbk)
Date: 2003-05-19 03:35

Message:
Logged In: YES 
user_id=149084

Hernan, there is no need for a bug report because there
is no interrupt module required for 0.9a2, and I am very
well aware that it needs to be compiled for the next
release.  In fact, it is not possible to generate a
bdist_wininst  without that happening!  The source is
in the IDLEfork directory: interruptmodule.c

Ok, I finally got to the end of the row bringing Python and
IDLEfork up on the new 2.4GHz P4 XP box.  Started with
a floppy drive which wouldn't write, and ended with the
Microsoft server down when trying to get VC5 SP3. So 
my fritter factor (defined as ratio of time dorking with
computer to time doing something useful) has been 
infinite most of the day.

I can say that moving the interrupt.pyd from W2K/Py2.2
to XP/Py2.3 doesn't work, Python2.3 can't import it. 
Also, the Windows installer created on Python2.2
won't install on a Py2.3 system because "Python2.2
is not in the registry."  But I finally got the extension
compiled.  Hopefully a windows installer built on Py2.3
will work on 2.2, but maybe I need to supply both.
We'll see.

And I'm happy to report that I see the problem as
described by Bruce.  So now I've got something
to work on.  I've already got it down to something
happening just after rpc.SocketIO.handle_EOF
tries to interrupt the MainThread: invalid command
name "11153040callit" while executing "11153040
callit" (after script)........

It's curious that you guys can see it on W2K and
I cannot.  But anyway, down to it.

And yes, that is the simple test.  Start with a 
shell and exit.  The GUI drops the link and the
subprocess is supposed to see that and exit.  You
expect to see two processes after the shell opens
because the subprocess is spawned when it does.
And you expect to see none when IDLEfork exits!

----------------------------------------------------------------------

Comment By: Kurt B. Kaiser (kbk)
Date: 2003-05-19 03:09

Message:
Logged In: YES 
user_id=149084

Hernan, there is no need for a bug report because there
is no interrupt module required for 0.9a2, and I am very
well aware that it needs to be compiled for the next
release.  In fact, it is not possible to generate a
bdist_wininst  without that happening!  The source is
in the IDLEfork directory: interruptmodule.c

Ok, I finally got to the end of the row bringing Python and
IDLEfork up on the new 2.4GHz P4 XP box.  Started with
a floppy drive which wouldn't write, and ended with the
Microsoft server down when trying to get VC5 SP3. So 
my fritter factor (defined as ratio of time dorking with
computer to time doing something useful) has been 
infinite most of the day.

I can say that moving the interrupt.pyd from W2K/Py2.2
to XP/Py2.3 doesn't work, Python2.3 can't import it. 
Also, the Windows installer created on Python2.2
won't install on a Py2.3 system because "Python2.2
is not in the registry."  But I finally got the extension
compiled.  Hopefully a windows installer built on Py2.3
will work on 2.2, but maybe I need to supply both.
We'll see.

And I'm happy to report that I see the problem as
described by Bruce.  So now I've got something
to work on.  I've already got it down to something
happening just after rpc.SocketIO.handle_EOF
tries to interrupt the MainThread: invalid command
name "11153040callit" while executing "11153040
callit" (after script)........

It's curious that you guys can see it on W2K and
I cannot.  But anyway, down to it.

And yes, that is the simple test.  Start with a 
shell and exit.  The GUI drops the link and the
subprocess is supposed to see that and exit.  You
expect to see two processes after the shell opens
because the subprocess is spawned when it does.
And you expect to see none when IDLEfork exits!

----------------------------------------------------------------------

Comment By: Hernan Martinez Foffani (hfoffani)
Date: 2003-05-19 00:08

Message:
Logged In: YES 
user_id=112690

bruce:
yes, i know the setup.py thing. i don't have a c compiler at
home, so i can't build the dll myself. thanks for uploading
the .pyd.

kurt:
i'm going to report a bug of the missing interrupt.pyd in
the win32 alpha distro as a reminder just in case there is
another package build before the merge.

bruce & kurt:
performance degrades on subsequent hits of F5.
as kurt pointed out, i can see precisely this: "As you create 
more and more pseudo-zombies, they share the cpu
equally.  So you may see e.g. three each getting 33%."
those zombies keep running even after exiting IDLE.

btw, does pythonw.exe work there? i can only start idle in
windows in the command line using the standard interpreter
(python.exe)

w2K pro, PIII 900 mhz
python 2.3, idlefork extracted from CVS on May, 18th

----------------------------------------------------------------------

Comment By: Bruce Sherwood (bsherwood)
Date: 2003-05-18 22:35

Message:
Logged In: YES 
user_id=34881

Oops. Failed to attach interrupt.pyd. I'll try again.

----------------------------------------------------------------------

Comment By: Bruce Sherwood (bsherwood)
Date: 2003-05-18 22:34

Message:
Logged In: YES 
user_id=34881

At the risk of giving unneeded information: You might not 
realize (as I did not realize) that you have to install using the 
distutil feature of Python. Open a command prompt, cd to the 
folder where you have the downloaded the files from CVS, and 
execute this:

c:\python23\python.exe setup.py install

Assuming you have an appropriate C compiler on your PC, 
this will produce the binary and copy it to c:\python23\Lib\site-
packages\interrupt.pyd, and all other needed files to 
c:\python23\Lib\site-packages\idlelib.

If you don't have an appropriate C compiler (I have Microsoft 
Visual Studio 6, which works), then indeed you need a binary 
distribution. In the hopes that this would help, I've uploaded 
my own c:\python23\Lib\site-packages\interrupt.pyd

----------------------------------------------------------------------

Comment By: Hernan Martinez Foffani (hfoffani)
Date: 2003-05-18 20:52

Message:
Logged In: YES 
user_id=112690

i tried to help but failed.
first installed a fresh python 2.3, later download the last
idelfork CVS.
idlefork didn't load.
i don't have the interrupt binary library. and it seems that
it's missing in the alpha binary distrib also
(IDLEfork-0.9a2.win32.exe)

regards,
-hernan

----------------------------------------------------------------------

Comment By: Bruce Sherwood (bsherwood)
Date: 2003-05-18 17:40

Message:
Logged In: YES 
user_id=34881

I tried the latest CVS idlefork (downloaded 11 am May 18) on 
a Win2000 machine with Python 2.3. Same catastrophic 
behavior as on my XP machine. Both machines are 500 MHz. 
Additional detail: After trying various other sequences, I tried 
the very simplest test I could think of. Set preferences to start 
in shell. Quit. Start up (in shell). I see two pythons--is that 
correct at the start, before doing anything? Without doing 
anything else, I choose the File/Exit menu option. The 
window disappears, but I'm left with a zombie python still up. 
Could some people besides Kurt and me please try the latest 
idlefork on various Win systems?

----------------------------------------------------------------------

Comment By: Kurt B. Kaiser (kbk)
Date: 2003-05-18 05:17

Message:
Logged In: YES 
user_id=149084

On W2K 500MHz P3 it normally takes about 2 sec 
to restart the subprocess and print something.

A 233Mhz P2 Linux box is somewhat faster.   The slow
start you are seeing is completely abnormal and
seems to be caused by the newly spawned process 
fighting for the socket.  There's a time-out somewhere.
I see it also if I do something that causes the
subprocess to block when the link is dropped by
the GUI (that's how the GUI signals an exit).

As far as the cpu being 100%, that is expected because
the subprocess is in a pass loop waiting for termination.
Normally that happens in a few millisec but in your case
the subprocess MainThread seems to be blocking and
so the SockThread never gets killed.  As you create 
more and more pseudo-zombies, they share the cpu
equally.  So you may see e.g. three each getting 33%.  
There are more sophisticated ways of doing the wait,
e.g. join the MainThread, but the spin gives a good
indication of what's going on for now.  Normally it only
lasts a few ms, as I said.

All of what you are seeing is abnormal. I don't see
anything like that here.

Clearly a long restart is unusable.  So is leaving a
mess of stuck processes behind.

OK, time to fire up the XP box here.  Thanks very much
for your help on this.

----------------------------------------------------------------------

Comment By: Bruce Sherwood (bsherwood)
Date: 2003-05-18 03:17

Message:
Logged In: YES 
user_id=34881

I tried run.py 1.18 and it's just as bad. 

The quit problem presumably has a solution. But what about 
the extremely serious problems that it takes 30 seconds to 
get output, and that after reaching the end of the program the 
CPU is pinned at 100%? Or are these problems related to the 
quit problem? The 30-second delay in getting output would 
make idlefork unusable as a programming environment.

----------------------------------------------------------------------

Comment By: Kurt B. Kaiser (kbk)
Date: 2003-05-18 00:49

Message:
Logged In: YES 
user_id=149084

BTW,  with the original IDLEfork the subprocess
terminates when the user code runs off the end.
In the current implementation it doesn't; the shell
stays connected to the subprocess so you can examine
the final environment.  That's why you only see the
one pythonw.exe in the original version.

----------------------------------------------------------------------

Comment By: Kurt B. Kaiser (kbk)
Date: 2003-05-17 23:15

Message:
Logged In: YES 
user_id=149084

I've checked in run.py 1.18, give it a try (though I don't
have much hope...)

I imagine you are starting up with just an edit window.
In that case you will see one pythonw.exe.  Once you
run the program the GUI will open a shell/output window
and spawn the subprocess.  Every time you re-run the
code the subprocess is terminated and a new one spawned.
So what you normally see from then on is two pythonw.exe.
Nothing to be alarmed about.  Now, if for some reason the
MainThread can't exit on Windows the GUI process loses
track of it and it just sits there and spins.  Every time the 
subprocess is restarted, another pseudo-zombie is created.

On W2K I can do all sorts of terrible things: put the 
subprocess into a pass loop or a print loop and then kill
the GUI with the task manager.  When I do that, the
subprocess exits OK.  But on your system, it seems
the MainThread is getting blocked somehow.

Try the new run.py and let me know.  If it doesn't work,
I have a nice fast XP box I built a while ago which I 
haven't had time to install Python on.  

Thanks for spending the time to help on this!  Freddy
has been holding up the release for a month.

----------------------------------------------------------------------

Comment By: Bruce Sherwood (bsherwood)
Date: 2003-05-17 19:59

Message:
Logged In: YES 
user_id=34881

One other comment: With the original idlefork, running my 
trivial program (print 2+2) brings up a 2nd pythonw.exe very 
briefly, but it then quickly goes away as soon as the"4" is 
printed, leaving only one copy active. It is somewhat alarming 
that with the new idlefork there are THREE copies of idlefork, 
and neither of the additional two copies goes away after 
the "4" is printed.

----------------------------------------------------------------------

Comment By: Bruce Sherwood (bsherwood)
Date: 2003-05-17 19:54

Message:
Logged In: YES 
user_id=34881

I replaced os._exit(0) with return, and that didn't fix it. I also 
installed idlefork with Python 2.2 and verified that behavior is 
identical on 2.2 and 2.3 on Windows XP. 

More details: It takes only a few seconds for the shell/output 
window to open (corresponding to the 2nd copy of 
pythonw.exe starting to run). It takes another 30 seconds 
before I see the output of my tiny program (print 2+2), and 
this corresponds to seeing in the system monitor the 3rd 
copy of pythonw.exe very slowly starting up, with its memory 
allocation growing very slowly until at about 5.7 MB I see my 
output. Thereafter CPU usage is pinned at 100%. I then quit 
either by clicking the close boxes on the two windows or by 
choosing the Exit menu option in one of the windows. The 
first pythonw.exe quits, but the 2nd and 3rd copies stay up 
and consume 100% of the CPU. I have to kill them in the 
system monitor. Now what? 

Note that it isn't just the quitting that fails catastrophically. 
There are two other very serious problems: It takes a very 
long time to get output (30 seconds), and there is 100% CPU 
usage even before I try to quit the program.

----------------------------------------------------------------------

Comment By: Kurt B. Kaiser (kbk)
Date: 2003-05-17 17:05

Message:
Logged In: YES 
user_id=149084

I assume you can see this if you just restart the shell
without running any program.

CVS works fine on W2K. But if I replace run.py:56 os._exit(0) 
with sys.exit(0), then the subprocess hangs as you are
seeing.  That's why I used os._exit.  It appears 
something changed between Py2.2/2.3 and/or W2K/XP and
os._exit() on your box is acting like sys.exit() on my setup.

The way this works is when SockThread gets an EOF
due to the link being closed by the GUI, it sets the global
exit_requested and goes into a pass loop waiting for the
MainThread to terminate the whole process.  Every 10
bytecodes the MainThread gets control and checks 
exit_requested.  If True, it exits and is supposed to take
the SockThread with it since it's setDaemon.  But if the
MainThread exits without shutting down SockThread the
latter just spins, taking 100%, which is what you are 
seeing.  If I exit the SockThread instead of waiting, there
are other problems due to the Server objects disappearing
when then are still needed by MainThread and its various
socket threads (Thread-1, Thread-2 etc) not having
wrapped up communication.  That also causes hangs.

Try this, it works on my system: 
replace 
run.py:56 os._exit(0) 
with 
run.py:56 return.  

We'll let Python decide how to exit.

----------------------------------------------------------------------

Comment By: Bruce Sherwood (bsherwood)
Date: 2003-05-17 15:26

Message:
Logged In: YES 
user_id=34881

I do have run.py Rev 1.17. I got everything from CVS on 5/16. 
And I've checked by hand that I do have the addition 
of "except KeyboardInterrupt" in my copy of run.py. So that's 
not the explanation. What should I try now?

----------------------------------------------------------------------

Comment By: Kurt B. Kaiser (kbk)
Date: 2003-05-17 00:15

Message:
Logged In: YES 
user_id=149084

It's critical that you have run.py Rev 1.17
which was checked in 2003/05/14 18:15:40to fix this
problem.

The relevant line is run.py:65  except KeyboardInterrupt:
This line and the except clause following must be present.
Without this, the restart doesn't work because the
subprocess SockThreadstops servicing the socket while
the MainThread is trying to print an exception.

I was seeing the same before the fix.  Freddy's baaack?

----------------------------------------------------------------------

Comment By: Bruce Sherwood (bsherwood)
Date: 2003-05-16 23:44

Message:
Logged In: YES 
user_id=34881

On Windows XP I installed Python 2.3 and idlefork from CVS. 
I see massive problems. Minimal program "print 2+2" takes 
30 seconds or more after pressing F5 before I see the output 
(consuming 100% of the CPU all the while), and even after 
seeing the output the system monitor shows me consuming 
100% of the CPU. I then close both windows (source and 
output), and I'm left with two instances of python.exe which 
together or singly consume 100% of the CPU, and I have to 
kill them in the system monitor. Is there something I'm 
missing about installation? I have the idlefork package in 
C:\Python23\idle. I did run C:\Python23\python.exe setup.py 
install inside the idle directory in order to get the interrupt 
module compiled. I assume that I have all the pieces in that I 
did get output.

----------------------------------------------------------------------

Comment By: Kurt B. Kaiser (kbk)
Date: 2003-05-15 05:48

Message:
Logged In: YES 
user_id=149084

Please try current CVS and report any problems.

----------------------------------------------------------------------

Comment By: Nobody/Anonymous (nobody)
Date: 2003-04-01 07:51

Message:
Logged In: NO 