[Patches] [ python-Patches-489173 ] Make os.spawnv not block the interpreter
noreply@sourceforge.net
noreply@sourceforge.net
Fri, 07 Dec 2001 10:32:02 -0800
Patches item #489173, was opened at 2001-12-04 17:22
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=489173&group_id=5470
Category: Core (C code)
Group: None
Status: Open
Resolution: None
Priority: 5
Submitted By: Anthony Roach (anthonyroach)
Assigned to: Tim Peters (tim_one)
Summary: Make os.spawnv not block the interpreter
Initial Comment:
Someone forgot to bracket the calls to _spawnv() and
_spawnve() with Py_BEGIN_ALLOW_THREADS and
Py_END_ALLOW_THREADS. This patch also fixes an error in
the docstrings for os.spawnv and os.spawnve.
----------------------------------------------------------------------
>Comment By: Anthony Roach (anthonyroach)
Date: 2001-12-07 10:32
Message:
Logged In: YES
user_id=257519
Tim:
You are correct that the problem is we need to get the
return value from the process, so P_NOWAIT is not an option.
My reason for seeing os.spawnv() blocking the interpreter as
a bug is that os.system() does not block the interpreter.
Based on the documentation and intended use of os.spawnv(),
it should work just like os.system() wrt to blocking the
interpreter. os.spawnv() just adds the ability to specify
the arguments as a list, and os.spawnve() adds the ability
to specify the environment. If os.spawnve() blocks the
interpreter, then there is no way, on win32, to have a
single process spawn multiple processes in parallel with a
specific environment.
I understand that you are concerned about breaking backward
compatility. I'd like to see this fix get into Python 2.2,
so we can tell our users that they can have -j support in
scons if they use Python 2.2, but I would understand if you
decide to put this off until Python 2.3.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-12-06 18:52
Message:
Logged In: YES
user_id=31435
I'm still not clear on what's wrong with using P_NOWAIT. A
call to spawnv returns quickly then, and that's no real
impediment to parallelization at the granularity of
compiler invocations. Ah -- but on Windows, there's then
no way in core Python to poll the returned process handle
for termination, or to get the process exit status when
it's done. Is *that* the hangup? If so, you coulda said
so <wink>.
I'm going to sleep on this one. I'm leaning toward
applying the patch for 2.2: there wouldn't have been any
controversy had that been done from the start, so it's
where I want to see Python end up. You don't need to file
a bug report, etc -- and thank you for explaining the
situation.
----------------------------------------------------------------------
Comment By: Steven Knight (stevenknight)
Date: 2001-12-06 16:16
Message:
Logged In: YES
user_id=216109
tim et al.--
Thanks for the discussion. Once I mentioned the
word "bug," I probably should have filed an actual bug
report to accompany Anthony's patch. However, at least for
now, I'm less concerned (personally) about this making it
into 2.2 than I am about knowing that this fix is "OK" so I
can tell people who want -j (parallel build) support in
SCons to patch their Python if they want it to work.
That said, the problem this patch solves is: You can't use
os.spawnve to spawn multiple processes from multiple
*threads* simultaneously. In the SCons architecture, we
create N threads when -j is used, and let each thread keep
pulling the next thing to be done from an iterator and
waiting for it to finish. This works fine with fork+exec,
and if you accept that os.spawnve should have isomorphic
behavior, it should work for os.spawnve as well.
It doesn't, though, because the lack of Py_{BEGIN,END}
_ALLOW_THREADS macros arround the _spawnv() call means the
interpreter stops in its tracks, and no other threads run.
A single thread waiting for a spawned process stops *all*
threads from executing.
Note that P_WAIT still works just fine in the non-threaded
case, of course. I can see, though, that there *might* be
someone out there who has multi-threaded code that's
dependent on os.spawnv(P_WAIT) stopping all threads.
Again, although it would be great to have this fixed, I'm
not overly concerned about it being in 2.2, because I
definitely agree about the need for caution once software
is past final beta...
If there's anything else I can do to help (file a bug
report in Tracker, provide more info, etc.), let me know.
----------------------------------------------------------------------
Comment By: Tim Peters (tim_one)
Date: 2001-12-06 15:57
Message:
Logged In: YES
user_id=31435
To stevenknight: I don't see a description of "a bug"
here, so it's hard to say whether this is a fix.
If an app doesn't want to wait for a spawned program to
complete, passing P_NOWAIT as the mode argument is the
usual cure.
There is some risk attached to this patch: it's possible
that an app passing P_WAIT today is relying on that their
program blocks until the spawned program completes. I
expect that's a small risk, but any risk makes me
uncomfortable for 2.2 since 2.2 had its final beta
release. So I'd like to hear the case for why this patch
is important (as above, there isn't a bug description here,
just a purported cure).
Note that if 2.2 were still in alpha test, I'd wouldn't
think more than twice <wink> about applying the patch.
----------------------------------------------------------------------
Comment By: Fred L. Drake, Jr. (fdrake)
Date: 2001-12-06 07:57
Message:
Logged In: YES
user_id=3066
Hmmm... it would take some time to get a Windows build
environment up on this machine. I'll assign this to Tim;
hopefully someone can contribute a test case for this so
there will be some chance he'll get to this for Python 2.2.
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-12-06 07:37
Message:
Logged In: YES
user_id=6380
Since it's Windows specific, can you test it?
----------------------------------------------------------------------
Comment By: Fred L. Drake, Jr. (fdrake)
Date: 2001-12-06 07:25
Message:
Logged In: YES
user_id=3066
Are there any objections to my checking this in for Python 2.2?
----------------------------------------------------------------------
Comment By: Guido van Rossum (gvanrossum)
Date: 2001-12-06 06:01
Message:
Logged In: YES
user_id=6380
I see no problem with the patch.
----------------------------------------------------------------------
Comment By: Steven Knight (stevenknight)
Date: 2001-12-06 04:04
Message:
Logged In: YES
user_id=216109
As the SCons project lead, it would be nice to have a
response from someone on the Python team acknowledging that
this is the right fix for this bug. As it is, it looks like
we're going to have to ship SCons without -j support on
WIN32 systems due to this problem in Python, but it would be
nice to be able to point people to this patch with
confidence, if they want to add the support to their own
copy of Python themselves.
----------------------------------------------------------------------
You can respond by visiting:
http://sourceforge.net/tracker/?func=detail&atid=305470&aid=489173&group_id=5470