On Wed, 10 Nov 2004, John P Speno wrote:
Hi, sorry for the delayed response.
> While using subprocess (aka popen5), I came across one potential gotcha. I've had
> exceptions ending like this:
>
> File "test.py", line 5, in test
> cmd = popen5.Popen(args, stdout=PIPE)
> File "popen5.py", line 577, in __init__
> data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
> OSError: [Errno 4] Interrupted system call
>
> (on Solaris 9)
>
> Would it make sense for subprocess to use a more robust read() function
> which can handle these cases, i.e. when the parent's read on the pipe
> to the child's stderr is interrupted by a system call, and returns EINTR?
> I imagine it could catch EINTR and EAGAIN and retry the failed read().
I assume you are using signals in your application? The os.read above is
not the only system call that can fail with EINTR. subprocess.py is full
of other system calls that can fail, and I suspect that many other Python
modules are as well.
I've made a patch (attached) to subprocess.py (and test_subprocess.py)
that should guard against EINTR, but I haven't committed it yet. It's
quite large.
Are Python modules supposed to handle EINTR? Why not let the C code handle
this? Or, perhaps the signal module should provide a sigaction function,
so that users can use SA_RESTART.
Index: subprocess.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/subprocess.py,v
retrieving revision 1.8
diff -u -r1.8 subprocess.py
--- subprocess.py 7 Nov 2004 14:30:34 -0000 1.8
+++ subprocess.py 17 Nov 2004 19:42:30 -0000
@@ -888,6 +888,50 @@
pass
+ def _read_no_intr(self, fd, buffersize):
+ """Like os.read, but retries on EINTR"""
+ while True:
+ try:
+ return os.read(fd, buffersize)
+ except OSError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
+
+ def _read_all(self, fd, buffersize):
+ """Like os.read, but retries on EINTR, and reads until EOF"""
+ all = ""
+ while True:
+ data = self._read_no_intr(fd, buffersize)
+ all += data
+ if data == "":
+ return all
+
+
+ def _write_no_intr(self, fd, s):
+ """Like os.write, but retries on EINTR"""
+ while True:
+ try:
+ return os.write(fd, s)
+ except OSError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
+ def _waitpid_no_intr(self, pid, options):
+ """Like os.waitpid, but retries on EINTR"""
+ while True:
+ try:
+ return os.waitpid(pid, options)
+ except OSError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
def _execute_child(self, args, executable, preexec_fn, close_fds,
cwd, env, universal_newlines,
startupinfo, creationflags, shell,
@@ -963,7 +1007,7 @@
exc_value,
tb)
exc_value.child_traceback = ''.join(exc_lines)
- os.write(errpipe_write, pickle.dumps(exc_value))
+ self._write_no_intr(errpipe_write, pickle.dumps(exc_value))
# This exitcode won't be reported to applications, so it
# really doesn't matter what we return.
@@ -979,7 +1023,7 @@
os.close(errwrite)
# Wait for exec to fail or succeed; possibly raising exception
- data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
+ data = self._read_all(errpipe_read, 1048576) # Exceptions limited to 1 MB
os.close(errpipe_read)
if data != "":
child_exception = pickle.loads(data)
@@ -1003,7 +1047,7 @@
attribute."""
if self.returncode == None:
try:
- pid, sts = os.waitpid(self.pid, os.WNOHANG)
+ pid, sts = self._waitpid_no_intr(self.pid, os.WNOHANG)
if pid == self.pid:
self._handle_exitstatus(sts)
except os.error:
@@ -1015,7 +1059,7 @@
"""Wait for child process to terminate. Returns returncode
attribute."""
if self.returncode == None:
- pid, sts = os.waitpid(self.pid, 0)
+ pid, sts = self._waitpid_no_intr(self.pid, 0)
self._handle_exitstatus(sts)
return self.returncode
@@ -1049,27 +1093,33 @@
stderr = []
while read_set or write_set:
- rlist, wlist, xlist = select.select(read_set, write_set, [])
+ try:
+ rlist, wlist, xlist = select.select(read_set, write_set, [])
+ except select.error, e:
+ if e[0] == errno.EINTR:
+ continue
+ else:
+ raise
if self.stdin in wlist:
# When select has indicated that the file is writable,
# we can write up to PIPE_BUF bytes without risk
# blocking. POSIX defines PIPE_BUF >= 512
- bytes_written = os.write(self.stdin.fileno(), input[:512])
+ bytes_written = self._write_no_intr(self.stdin.fileno(), input[:512])
input = input[bytes_written:]
if not input:
self.stdin.close()
write_set.remove(self.stdin)
if self.stdout in rlist:
- data = os.read(self.stdout.fileno(), 1024)
+ data = self._read_no_intr(self.stdout.fileno(), 1024)
if data == "":
self.stdout.close()
read_set.remove(self.stdout)
stdout.append(data)
if self.stderr in rlist:
- data = os.read(self.stderr.fileno(), 1024)
+ data = self._read_no_intr(self.stderr.fileno(), 1024)
if data == "":
self.stderr.close()
read_set.remove(self.stderr)
Index: test/test_subprocess.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v
retrieving revision 1.14
diff -u -r1.14 test_subprocess.py
--- test/test_subprocess.py 12 Nov 2004 15:51:48 -0000 1.14
+++ test/test_subprocess.py 17 Nov 2004 19:42:30 -0000
@@ -7,6 +7,7 @@
import tempfile
import time
import re
+import errno
mswindows = (sys.platform == "win32")
@@ -35,6 +36,16 @@
fname = tempfile.mktemp()
return os.open(fname, os.O_RDWR|os.O_CREAT), fname
+ def read_no_intr(self, obj):
+ while True:
+ try:
+ return obj.read()
+ except IOError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
#
# Generic tests
#
@@ -123,7 +134,7 @@
p = subprocess.Popen([sys.executable, "-c",
'import sys; sys.stdout.write("orange")'],
stdout=subprocess.PIPE)
- self.assertEqual(p.stdout.read(), "orange")
+ self.assertEqual(self.read_no_intr(p.stdout), "orange")
def test_stdout_filedes(self):
# stdout is set to open file descriptor
@@ -151,7 +162,7 @@
p = subprocess.Popen([sys.executable, "-c",
'import sys; sys.stderr.write("strawberry")'],
stderr=subprocess.PIPE)
- self.assertEqual(remove_stderr_debug_decorations(p.stderr.read()),
+ self.assertEqual(remove_stderr_debug_decorations(self.read_no_intr(p.stderr)),
"strawberry")
def test_stderr_filedes(self):
@@ -186,7 +197,7 @@
'sys.stderr.write("orange")'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
- output = p.stdout.read()
+ output = self.read_no_intr(p.stdout)
stripped = remove_stderr_debug_decorations(output)
self.assertEqual(stripped, "appleorange")
@@ -220,7 +231,7 @@
stdout=subprocess.PIPE,
cwd=tmpdir)
normcase = os.path.normcase
- self.assertEqual(normcase(p.stdout.read()), normcase(tmpdir))
+ self.assertEqual(normcase(self.read_no_intr(p.stdout)), normcase(tmpdir))
def test_env(self):
newenv = os.environ.copy()
@@ -230,7 +241,7 @@
'sys.stdout.write(os.getenv("FRUIT"))'],
stdout=subprocess.PIPE,
env=newenv)
- self.assertEqual(p.stdout.read(), "orange")
+ self.assertEqual(self.read_no_intr(p.stdout), "orange")
def test_communicate(self):
p = subprocess.Popen([sys.executable, "-c",
@@ -305,7 +316,8 @@
'sys.stdout.write("\\nline6");'],
stdout=subprocess.PIPE,
universal_newlines=1)
- stdout = p.stdout.read()
+
+ stdout = self.read_no_intr(p.stdout)
if hasattr(open, 'newlines'):
# Interpreter with universal newline support
self.assertEqual(stdout,
@@ -343,7 +355,7 @@
def test_no_leaking(self):
# Make sure we leak no resources
- max_handles = 1026 # too much for most UNIX systems
+ max_handles = 10 # too much for most UNIX systems
if mswindows:
max_handles = 65 # a full test is too slow on Windows
for i in range(max_handles):
@@ -424,7 +436,7 @@
'sys.stdout.write(os.getenv("FRUIT"))'],
stdout=subprocess.PIPE,
preexec_fn=lambda: os.putenv("FRUIT", "apple"))
- self.assertEqual(p.stdout.read(), "apple")
+ self.assertEqual(self.read_no_intr(p.stdout), "apple")
def test_args_string(self):
# args is a string
@@ -457,7 +469,7 @@
p = subprocess.Popen(["echo $FRUIT"], shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertEqual(p.stdout.read().strip(), "apple")
+ self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple")
def test_shell_string(self):
# Run command through the shell (string)
@@ -466,7 +478,7 @@
p = subprocess.Popen("echo $FRUIT", shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertEqual(p.stdout.read().strip(), "apple")
+ self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple")
def test_call_string(self):
# call() function with string argument on UNIX
@@ -525,7 +537,7 @@
p = subprocess.Popen(["set"], shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertNotEqual(p.stdout.read().find("physalis"), -1)
+ self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1)
def test_shell_string(self):
# Run command through the shell (string)
@@ -534,7 +546,7 @@
p = subprocess.Popen("set", shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertNotEqual(p.stdout.read().find("physalis"), -1)
+ self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1)
def test_call_string(self):
# call() function with string argument on Windows
/Peter Åstrand <astrand(a)lysator.liu.se>
Problem:
When the code contains list comprehensions (or for that matter any other
looping construct), the only way to get quickly through this code in pdb
is to set a temporary breakpoint on the line after the loop, which is
inconvenient..
There is a SF bug report #1248119 about this behavior.
Solution:
Should pdb's next command accept an optional numeric argument? It would
specify how many actual lines of code (not "line events")
should be skipped in the current frame before stopping,
i.e "next 5" would mean stop when
line>=line_where_next_N_happened+5
is reached.
This would allow to easily get over/out of loops in the debugger
What do you think?
Ilya
Hello,
This patch is about to celebrate its second birthday :-)
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=790710&group...
It seems from the comments that the feature is nice but the
implementation was not OK.
I redid the implem according to the comments.
What should I do to get it reviewed further ? (perhaps just this :
posting to python-dev :-)
Best,
--
Grégoire
Hi,
I'm having 2 problems with the current cvs :
During compilation this warning occurs:
*** WARNING: renaming "dbm" since importing it failed: build/lib.linux-i686-2.5/
dbm.so: undefined symbol: dbm_firstkey
and the 'dbm' module is unavailable.
I'm running MandrakeLinux 2005 (10.2) gcc 3.4.3
(I'm also having this problem when compiling python 2.3.5 or 2.4.1)
furthermore the 'make install' of current cvs fails halfway trough
with the following errors:
.....
.....
Compiling /opt/python25/lib/python2.5/bsddb/test/test_associate.py ...
Sorry: TabError: ('inconsistent use of tabs and spaces in indentation',
('/opt/python25/lib/python2.5/bsddb/test/test_associate.py', 97, 23, '\t
os.mkdir(homeDir)\n'))
Compiling /opt/python25/lib/python2.5/bsddb/test/test_basics.py ...
Sorry: TabError: ('inconsistent use of tabs and spaces in indentation',
('/opt/python25/lib/python2.5/bsddb/test/test_basics.py', 400, 26, '\t if
get_raises_error:\n'))
Compiling /opt/python25/lib/python2.5/bsddb/test/test_compare.py ...
Sorry: TabError: ('inconsistent use of tabs and spaces in indentation',
('/opt/python25/lib/python2.5/bsddb/test/test_compare.py', 167, 5, '\t"""\n'))
.....
.....
Compiling /opt/python25/lib/python2.5/bsddb/test/test_recno.py ...
Sorry: TabError: ('inconsistent use of tabs and spaces in indentation',
('/opt/python25/lib/python2.5/bsddb/test/test_recno.py', 38, 46, '\tget_returns_none =
d.set_get_returns_none(2)\n'))
.....
.....
make: *** [libinstall] Error 1
$
And then it quits.
Fixing the tab indentation errors locally makes the problem go away.
Regards,
Irmen de Jong
[Christoph, please keep the python-dev list in the loop here, at least
until they get annoyed and decide we're off-topic. I think this is
crucial to the way they package and deliver Python]
Christoph Ludwig <cludwig(a)cdc.informatik.tu-darmstadt.de> writes:
> On Thu, Jul 07, 2005 at 06:27:46PM -0400, David Abrahams wrote:
>> "Martin v. Löwis" <martin(a)v.loewis.de> writes:
>>
>> > David Abrahams wrote:
>> >> I'm wondering if there has been a well-known recent change either in Python
>> >> or GCC that would account for these new reports. Any relevant
>> >> information would be appreciated.
> [...]
>> > Python is linked with g++ if configure thinks this is necessary
>>
>> Right. The question is, when should configure "think it's necessary?"
>
> Just to add to the confusion... I encountered the case that configure decided
> to use gcc for linking but it should have used g++. (It is Python
> PR #1189330 <http://tinyurl.com/dlheb>. This was on a x86 Linux system with
> g++ 3.4.2.)
>
> Background: The description of --with-cxx in the README of the
> Python 2.4.1 source distribution made me think that I need to
> configure my Python installation with
> --with-configure=/opt/gcc/gcc-3.4.2/bin/g++ if I plan to use C++
> extensions built with this compiler. (That was possibly a
> misunderstanding on my part,
AFAICT, yes.
> but Python should build with this option anyway.)
>
> configure set `LINKCC=$(PURIFY) $(CC)'. The result was that make failed when
> linking the python executable due to an unresolved reference to
> __gxx_personality_v0. I had to replace CC by CXX in the definition of LINKCC
> to finish the build of Python.
>
> When I looked into this problem I saw that configure in fact builds a test
> executable that included an object file compiled with g++. If the link step
> with gcc succeeds then LINKCC is set as above, otherwise CXX is
> used. Obviously, on my system this test was successful so configure decided
> to link with gcc. However, minimal changes to the source of the test program
> caused the link step to fail. It was not obvious to me at all why the latter
> source code should cause a dependency on the C++ runtime if the original
> code does not. My conclusion was that this test is fragile and should be
> skipped.
Sounds like it. I have never understood what the test was really
checking for since the moment it was first described to me, FWIW.
> If Python is built with --with-cxx then it should be linked with CXX
> as well.
U betcha.
> I gather from posts on the Boost mailing lists that you can import
> Boost.Python extensions even if Python was configured
> --without-cxx.
Yes, all the tests are passing that way.
> (On ELF based Linux/x86, at least.) That leaves me wondering
>
> * when is --with-cxx really necessary?
I think it's plausible that if you set sys.dlopenflags to share
symbols it *might* end up being necessary, but IIRC Ralf does use
sys.dlopenflags with a standard build of Python (no
--with-cxx)... right, Ralf?
> * what happens if I import extensions built with different g++ versions? Will
> there be a conflict between the different versions of libstdc++ those
> extensions depend on?
Not unless you set sys.dlopenflags to share symbols.
It's conceivable that they might conflict through their shared use of
libboost_python.so, but I think you have to accept that an extension
module and the libboost_python.so it is linked with have to be built
with compatible ABIs anyway. So in that case you may need to make
sure each group of extensions built with a given ABI use their own
libboost_python.so (or just link statically to libboost_python.a if
you don't need cross-module conversions).
HTH,
--
Dave Abrahams
Boost Consulting
www.boost-consulting.com
What happened to the CurrentVersion registry entry documented at
http://www.python.org/windows/python/registry.html
AFAICT, even the python15.wse file did not fill a value in this
entry (perhaps I'm misinterpreting the wse file, though).
So was this ever used? Why is it documented, and who documented it
(unfortunately, registry.html is not in cvs/subversion, either)?
Regards,
Martin
There's a problem with genexp's that I think really needs to get
fixed. See http://python.org/sf/1167751 the details are below. This
code:
>>> foo(a = i for i in range(10))
generates "NameError: name 'i' is not defined" when run because:
2 0 LOAD_GLOBAL 0 (foo)
3 LOAD_CONST 1 ('a')
6 LOAD_GLOBAL 1 (i)
9 CALL_FUNCTION 256
12 POP_TOP
13 LOAD_CONST 0 (None)
16 RETURN_VALUE
If you add parens around the code: foo(a = i for i in range(10))
You get something quite different:
2 0 LOAD_GLOBAL 0 (foo)
3 LOAD_CONST 1 ('a')
6 LOAD_CONST 2 (<code object <generator
expression> at 0x2a960baae8, file "<stdin>", line 2>)
9 MAKE_FUNCTION 0
12 LOAD_GLOBAL 1 (range)
15 LOAD_CONST 3 (10)
18 CALL_FUNCTION 1
21 GET_ITER
22 CALL_FUNCTION 1
25 CALL_FUNCTION 256
28 POP_TOP
29 LOAD_CONST 0 (None)
32 RETURN_VALUE
I agree with the bug report that the code should either raise a
SyntaxError or do the right thing.
n
Hi,
some time ago, I proposed a string method "dedent" (which currently is in the
textwrap module). The RFE is at http://python.org/sf/1237680.
Any opinions? If I don't get positive comments, I'll reject it.
Reinhold
--
Mail address is perfectly valid!
At 08:57 AM 10/20/2005 -0700, Guido van Rossum wrote:
>Whoa, folks! Can I ask the gentlemen to curb their enthusiasm?
>
>PEP 343 is still (back) on the drawing table, PEP 342 has barely been
>implemented (did it survive the AST-branch merge?), and already you
>are talking about adding more stuff. Please put on the brakes!
Sorry. I thought that 343 was just getting a minor tune-up. In the months
since the discussion and approval (and implementation; Michael Hudson
actually had a PEP 343 patch out there), I've been doing a lot of thinking
about how they will be used in applications, and thought that it would be a
good idea to promote people using task-specific variables in place of
globals or thread-locals.
The conventional wisdom is that global variables are bad, but the truth is
that they're very attractive because they allow you to have one less thing
to pass around and think about in every line of code. Without globals, you
would sooner or later end up with every function taking twenty arguments to
pass through states down to other code, or else trying to cram all this
data into some kind of "context" object, which then won't work with code
that doesn't know about *your* definition of what a context is.
Globals are thus extremely attractive for practical software
development. If they weren't so useful, it wouldn't be necessary to warn
people not to use them, after all. :)
The problem with globals, however, is that sometimes they need to be
changed in a particular context. PEP 343 makes it safer to use globals
because you can always offer a context manager that changes them
temporarily, without having to hand-write a try-finally block. This will
make it even *more* attractive to use globals, which is not a problem as
long as the code has no multitasking of any sort.
Of course, the multithreading scenario is usually fixed by using
thread-locals. All I'm proposing is that we replace thread locals with
task locals, and promote the use of task-local variables for managed
contexts (such as the decimal context) *that would otherwise be a global or
a thread-local variable*. This doesn't seem to me like a very big deal;
just an encouragement for people to make their stuff easy to use with PEP
342 and 343.
By the way, I don't know if you do much with Java these days, but a big
part of the whole J2EE fiasco and the rise of the so-called "lightweight
containers" in Java has all been about how to manage implicit context so
that you don't get stuck with either the inflexibility of globals or the
deadweight of passing tons of parameters around. One of the big selling
points of AspectJ is that it lets you implicitly funnel parameters from
point A to point B without having to modify all the call signatures in
between. In other words, its use is promoted for precisely the sort of
thing that 'with' plus a task variable would be ideal for. As far as I can
tell, 'with' plus a task variable is *much* easier to explain, use, and
understand than an aspect-oriented programming tool is! (Especially from
the "if the implementation is easy to explain, it may be a good idea"
perspective.)
>I know that somewhere in the proto-PEP Phillip argues that the context
>API needs to be made a part of the standard library so that his
>trampoline can efficiently swap implicit contexts required by
>arbitrary standard and third-party library code. My response to that
>is that library code (whether standard or third-party) should not
>depend on implicit context unless it assumes it can assume complete
>control over the application.
I think maybe there's some confusion here, at least on my part. :) I see
two ways to read your statement, one of which seems to be saying that we
should get rid of the decimal context (because it doesn't have complete
control over the application), and the other way of reading it doesn't seem
connected to what I proposed.
Anything that's a global variable is an "implicit context". Because of
that, I spent considerable time and effort in PEAK trying to utterly stamp
out global variables. *Everything* in PEAK has an explicit context. But
that then becomes more of a pain to *use*, because you are now stuck with
managing it, even if you cram it into a Zope-style acquisition tree so
there's only one "context" to deal with. Plus, it assumes that everything
the developer wants to do can be supplied by *one* framework, be it PEAK,
Zope, or whatever, which is rarely the case but still forces framework
developers to duplicate everybody else's stuff.
In other words, I've come to realize that the path the major Python
application frameworks is not really Pythonic. A Pythonic framework
shouldn't load you down with new management burdens and keep you from using
other frameworks. It should make life easier, and make your code *more*
interoperable, not less. Indeed, I've pretty much come to agreement with
the part of the Python developer community that has says Frameworks Are
Evil. A primary source of this evil in the big three frameworks (PEAK,
Twisted, and Zope) stem from their various approaches to dealing with this
issue of context, which lack the simplicity of global (or task-local)
variables.
So, the lesson I've taken from my attempt to make everything explicit is
that what developers *really* want is to have global variables, just
without the downsides of uncontrolled modifications, and inter-thread or
inter-task pollution. Explicit isn't always better than implicit, because
oftentimes the practicality of having implicit things is much more
important than the purity of making them all explicit. Simple is better
than complex, and task-local variables are *much* simpler than trying to
make everything explicit.
>Also, Nick wants the name 'context' for PEP-343 style context
>managers. I think it's overloading too much to use the same word for
>per-thread or per-coroutine context.
Actually, I was the one who originally proposed the term "context manager",
and it doesn't seem like a conflict to me. Indeed, I suggested in the
pre-PEP that "@context.manager" might be where we could put the
decorator. The overload was intentional, to suggest that when creating a
new context manager, it's worth considering whether the state should be
kept in a context variable, rather than a global variable. The naming
choice was for propaganda purposes, in other words. :)
Anyway, I'll withdraw the proposal for now. We can always leave it out of
2.5, I can release an independent implementation, and then submit it for
consideration again in the 2.6 timeframe. I just thought it would be a
no-brainer to use task locals where thread locals are currently being used,
and that's really all I was proposing we do as far as stdlib changes
anyway. I was also hoping to get good input from Python-dev regarding some
of the open issues, to try and build a consensus on them from the beginning.
