On Wed, 10 Nov 2004, John P Speno wrote:
Hi, sorry for the delayed response.
> While using subprocess (aka popen5), I came across one potential gotcha. I've had
> exceptions ending like this:
>
> File "test.py", line 5, in test
> cmd = popen5.Popen(args, stdout=PIPE)
> File "popen5.py", line 577, in __init__
> data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
> OSError: [Errno 4] Interrupted system call
>
> (on Solaris 9)
>
> Would it make sense for subprocess to use a more robust read() function
> which can handle these cases, i.e. when the parent's read on the pipe
> to the child's stderr is interrupted by a system call, and returns EINTR?
> I imagine it could catch EINTR and EAGAIN and retry the failed read().
I assume you are using signals in your application? The os.read above is
not the only system call that can fail with EINTR. subprocess.py is full
of other system calls that can fail, and I suspect that many other Python
modules are as well.
I've made a patch (attached) to subprocess.py (and test_subprocess.py)
that should guard against EINTR, but I haven't committed it yet. It's
quite large.
Are Python modules supposed to handle EINTR? Why not let the C code handle
this? Or, perhaps the signal module should provide a sigaction function,
so that users can use SA_RESTART.
Index: subprocess.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/subprocess.py,v
retrieving revision 1.8
diff -u -r1.8 subprocess.py
--- subprocess.py 7 Nov 2004 14:30:34 -0000 1.8
+++ subprocess.py 17 Nov 2004 19:42:30 -0000
@@ -888,6 +888,50 @@
pass
+ def _read_no_intr(self, fd, buffersize):
+ """Like os.read, but retries on EINTR"""
+ while True:
+ try:
+ return os.read(fd, buffersize)
+ except OSError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
+
+ def _read_all(self, fd, buffersize):
+ """Like os.read, but retries on EINTR, and reads until EOF"""
+ all = ""
+ while True:
+ data = self._read_no_intr(fd, buffersize)
+ all += data
+ if data == "":
+ return all
+
+
+ def _write_no_intr(self, fd, s):
+ """Like os.write, but retries on EINTR"""
+ while True:
+ try:
+ return os.write(fd, s)
+ except OSError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
+ def _waitpid_no_intr(self, pid, options):
+ """Like os.waitpid, but retries on EINTR"""
+ while True:
+ try:
+ return os.waitpid(pid, options)
+ except OSError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
def _execute_child(self, args, executable, preexec_fn, close_fds,
cwd, env, universal_newlines,
startupinfo, creationflags, shell,
@@ -963,7 +1007,7 @@
exc_value,
tb)
exc_value.child_traceback = ''.join(exc_lines)
- os.write(errpipe_write, pickle.dumps(exc_value))
+ self._write_no_intr(errpipe_write, pickle.dumps(exc_value))
# This exitcode won't be reported to applications, so it
# really doesn't matter what we return.
@@ -979,7 +1023,7 @@
os.close(errwrite)
# Wait for exec to fail or succeed; possibly raising exception
- data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
+ data = self._read_all(errpipe_read, 1048576) # Exceptions limited to 1 MB
os.close(errpipe_read)
if data != "":
child_exception = pickle.loads(data)
@@ -1003,7 +1047,7 @@
attribute."""
if self.returncode == None:
try:
- pid, sts = os.waitpid(self.pid, os.WNOHANG)
+ pid, sts = self._waitpid_no_intr(self.pid, os.WNOHANG)
if pid == self.pid:
self._handle_exitstatus(sts)
except os.error:
@@ -1015,7 +1059,7 @@
"""Wait for child process to terminate. Returns returncode
attribute."""
if self.returncode == None:
- pid, sts = os.waitpid(self.pid, 0)
+ pid, sts = self._waitpid_no_intr(self.pid, 0)
self._handle_exitstatus(sts)
return self.returncode
@@ -1049,27 +1093,33 @@
stderr = []
while read_set or write_set:
- rlist, wlist, xlist = select.select(read_set, write_set, [])
+ try:
+ rlist, wlist, xlist = select.select(read_set, write_set, [])
+ except select.error, e:
+ if e[0] == errno.EINTR:
+ continue
+ else:
+ raise
if self.stdin in wlist:
# When select has indicated that the file is writable,
# we can write up to PIPE_BUF bytes without risk
# blocking. POSIX defines PIPE_BUF >= 512
- bytes_written = os.write(self.stdin.fileno(), input[:512])
+ bytes_written = self._write_no_intr(self.stdin.fileno(), input[:512])
input = input[bytes_written:]
if not input:
self.stdin.close()
write_set.remove(self.stdin)
if self.stdout in rlist:
- data = os.read(self.stdout.fileno(), 1024)
+ data = self._read_no_intr(self.stdout.fileno(), 1024)
if data == "":
self.stdout.close()
read_set.remove(self.stdout)
stdout.append(data)
if self.stderr in rlist:
- data = os.read(self.stderr.fileno(), 1024)
+ data = self._read_no_intr(self.stderr.fileno(), 1024)
if data == "":
self.stderr.close()
read_set.remove(self.stderr)
Index: test/test_subprocess.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v
retrieving revision 1.14
diff -u -r1.14 test_subprocess.py
--- test/test_subprocess.py 12 Nov 2004 15:51:48 -0000 1.14
+++ test/test_subprocess.py 17 Nov 2004 19:42:30 -0000
@@ -7,6 +7,7 @@
import tempfile
import time
import re
+import errno
mswindows = (sys.platform == "win32")
@@ -35,6 +36,16 @@
fname = tempfile.mktemp()
return os.open(fname, os.O_RDWR|os.O_CREAT), fname
+ def read_no_intr(self, obj):
+ while True:
+ try:
+ return obj.read()
+ except IOError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
#
# Generic tests
#
@@ -123,7 +134,7 @@
p = subprocess.Popen([sys.executable, "-c",
'import sys; sys.stdout.write("orange")'],
stdout=subprocess.PIPE)
- self.assertEqual(p.stdout.read(), "orange")
+ self.assertEqual(self.read_no_intr(p.stdout), "orange")
def test_stdout_filedes(self):
# stdout is set to open file descriptor
@@ -151,7 +162,7 @@
p = subprocess.Popen([sys.executable, "-c",
'import sys; sys.stderr.write("strawberry")'],
stderr=subprocess.PIPE)
- self.assertEqual(remove_stderr_debug_decorations(p.stderr.read()),
+ self.assertEqual(remove_stderr_debug_decorations(self.read_no_intr(p.stderr)),
"strawberry")
def test_stderr_filedes(self):
@@ -186,7 +197,7 @@
'sys.stderr.write("orange")'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
- output = p.stdout.read()
+ output = self.read_no_intr(p.stdout)
stripped = remove_stderr_debug_decorations(output)
self.assertEqual(stripped, "appleorange")
@@ -220,7 +231,7 @@
stdout=subprocess.PIPE,
cwd=tmpdir)
normcase = os.path.normcase
- self.assertEqual(normcase(p.stdout.read()), normcase(tmpdir))
+ self.assertEqual(normcase(self.read_no_intr(p.stdout)), normcase(tmpdir))
def test_env(self):
newenv = os.environ.copy()
@@ -230,7 +241,7 @@
'sys.stdout.write(os.getenv("FRUIT"))'],
stdout=subprocess.PIPE,
env=newenv)
- self.assertEqual(p.stdout.read(), "orange")
+ self.assertEqual(self.read_no_intr(p.stdout), "orange")
def test_communicate(self):
p = subprocess.Popen([sys.executable, "-c",
@@ -305,7 +316,8 @@
'sys.stdout.write("\\nline6");'],
stdout=subprocess.PIPE,
universal_newlines=1)
- stdout = p.stdout.read()
+
+ stdout = self.read_no_intr(p.stdout)
if hasattr(open, 'newlines'):
# Interpreter with universal newline support
self.assertEqual(stdout,
@@ -343,7 +355,7 @@
def test_no_leaking(self):
# Make sure we leak no resources
- max_handles = 1026 # too much for most UNIX systems
+ max_handles = 10 # too much for most UNIX systems
if mswindows:
max_handles = 65 # a full test is too slow on Windows
for i in range(max_handles):
@@ -424,7 +436,7 @@
'sys.stdout.write(os.getenv("FRUIT"))'],
stdout=subprocess.PIPE,
preexec_fn=lambda: os.putenv("FRUIT", "apple"))
- self.assertEqual(p.stdout.read(), "apple")
+ self.assertEqual(self.read_no_intr(p.stdout), "apple")
def test_args_string(self):
# args is a string
@@ -457,7 +469,7 @@
p = subprocess.Popen(["echo $FRUIT"], shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertEqual(p.stdout.read().strip(), "apple")
+ self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple")
def test_shell_string(self):
# Run command through the shell (string)
@@ -466,7 +478,7 @@
p = subprocess.Popen("echo $FRUIT", shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertEqual(p.stdout.read().strip(), "apple")
+ self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple")
def test_call_string(self):
# call() function with string argument on UNIX
@@ -525,7 +537,7 @@
p = subprocess.Popen(["set"], shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertNotEqual(p.stdout.read().find("physalis"), -1)
+ self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1)
def test_shell_string(self):
# Run command through the shell (string)
@@ -534,7 +546,7 @@
p = subprocess.Popen("set", shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertNotEqual(p.stdout.read().find("physalis"), -1)
+ self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1)
def test_call_string(self):
# call() function with string argument on Windows
/Peter Åstrand <astrand(a)lysator.liu.se>
Problem:
When the code contains list comprehensions (or for that matter any other
looping construct), the only way to get quickly through this code in pdb
is to set a temporary breakpoint on the line after the loop, which is
inconvenient..
There is a SF bug report #1248119 about this behavior.
Solution:
Should pdb's next command accept an optional numeric argument? It would
specify how many actual lines of code (not "line events")
should be skipped in the current frame before stopping,
i.e "next 5" would mean stop when
line>=line_where_next_N_happened+5
is reached.
This would allow to easily get over/out of loops in the debugger
What do you think?
Ilya
Hello,
This patch is about to celebrate its second birthday :-)
https://sourceforge.net/tracker/?func=detail&atid=305470&aid=790710&group...
It seems from the comments that the feature is nice but the
implementation was not OK.
I redid the implem according to the comments.
What should I do to get it reviewed further ? (perhaps just this :
posting to python-dev :-)
Best,
--
Grégoire
Hi,
I'm having 2 problems with the current cvs :
During compilation this warning occurs:
*** WARNING: renaming "dbm" since importing it failed: build/lib.linux-i686-2.5/
dbm.so: undefined symbol: dbm_firstkey
and the 'dbm' module is unavailable.
I'm running MandrakeLinux 2005 (10.2) gcc 3.4.3
(I'm also having this problem when compiling python 2.3.5 or 2.4.1)
furthermore the 'make install' of current cvs fails halfway trough
with the following errors:
.....
.....
Compiling /opt/python25/lib/python2.5/bsddb/test/test_associate.py ...
Sorry: TabError: ('inconsistent use of tabs and spaces in indentation',
('/opt/python25/lib/python2.5/bsddb/test/test_associate.py', 97, 23, '\t
os.mkdir(homeDir)\n'))
Compiling /opt/python25/lib/python2.5/bsddb/test/test_basics.py ...
Sorry: TabError: ('inconsistent use of tabs and spaces in indentation',
('/opt/python25/lib/python2.5/bsddb/test/test_basics.py', 400, 26, '\t if
get_raises_error:\n'))
Compiling /opt/python25/lib/python2.5/bsddb/test/test_compare.py ...
Sorry: TabError: ('inconsistent use of tabs and spaces in indentation',
('/opt/python25/lib/python2.5/bsddb/test/test_compare.py', 167, 5, '\t"""\n'))
.....
.....
Compiling /opt/python25/lib/python2.5/bsddb/test/test_recno.py ...
Sorry: TabError: ('inconsistent use of tabs and spaces in indentation',
('/opt/python25/lib/python2.5/bsddb/test/test_recno.py', 38, 46, '\tget_returns_none =
d.set_get_returns_none(2)\n'))
.....
.....
make: *** [libinstall] Error 1
$
And then it quits.
Fixing the tab indentation errors locally makes the problem go away.
Regards,
Irmen de Jong
[Christoph, please keep the python-dev list in the loop here, at least
until they get annoyed and decide we're off-topic. I think this is
crucial to the way they package and deliver Python]
Christoph Ludwig <cludwig(a)cdc.informatik.tu-darmstadt.de> writes:
> On Thu, Jul 07, 2005 at 06:27:46PM -0400, David Abrahams wrote:
>> "Martin v. Löwis" <martin(a)v.loewis.de> writes:
>>
>> > David Abrahams wrote:
>> >> I'm wondering if there has been a well-known recent change either in Python
>> >> or GCC that would account for these new reports. Any relevant
>> >> information would be appreciated.
> [...]
>> > Python is linked with g++ if configure thinks this is necessary
>>
>> Right. The question is, when should configure "think it's necessary?"
>
> Just to add to the confusion... I encountered the case that configure decided
> to use gcc for linking but it should have used g++. (It is Python
> PR #1189330 <http://tinyurl.com/dlheb>. This was on a x86 Linux system with
> g++ 3.4.2.)
>
> Background: The description of --with-cxx in the README of the
> Python 2.4.1 source distribution made me think that I need to
> configure my Python installation with
> --with-configure=/opt/gcc/gcc-3.4.2/bin/g++ if I plan to use C++
> extensions built with this compiler. (That was possibly a
> misunderstanding on my part,
AFAICT, yes.
> but Python should build with this option anyway.)
>
> configure set `LINKCC=$(PURIFY) $(CC)'. The result was that make failed when
> linking the python executable due to an unresolved reference to
> __gxx_personality_v0. I had to replace CC by CXX in the definition of LINKCC
> to finish the build of Python.
>
> When I looked into this problem I saw that configure in fact builds a test
> executable that included an object file compiled with g++. If the link step
> with gcc succeeds then LINKCC is set as above, otherwise CXX is
> used. Obviously, on my system this test was successful so configure decided
> to link with gcc. However, minimal changes to the source of the test program
> caused the link step to fail. It was not obvious to me at all why the latter
> source code should cause a dependency on the C++ runtime if the original
> code does not. My conclusion was that this test is fragile and should be
> skipped.
Sounds like it. I have never understood what the test was really
checking for since the moment it was first described to me, FWIW.
> If Python is built with --with-cxx then it should be linked with CXX
> as well.
U betcha.
> I gather from posts on the Boost mailing lists that you can import
> Boost.Python extensions even if Python was configured
> --without-cxx.
Yes, all the tests are passing that way.
> (On ELF based Linux/x86, at least.) That leaves me wondering
>
> * when is --with-cxx really necessary?
I think it's plausible that if you set sys.dlopenflags to share
symbols it *might* end up being necessary, but IIRC Ralf does use
sys.dlopenflags with a standard build of Python (no
--with-cxx)... right, Ralf?
> * what happens if I import extensions built with different g++ versions? Will
> there be a conflict between the different versions of libstdc++ those
> extensions depend on?
Not unless you set sys.dlopenflags to share symbols.
It's conceivable that they might conflict through their shared use of
libboost_python.so, but I think you have to accept that an extension
module and the libboost_python.so it is linked with have to be built
with compatible ABIs anyway. So in that case you may need to make
sure each group of extensions built with a given ABI use their own
libboost_python.so (or just link statically to libboost_python.a if
you don't need cross-module conversions).
HTH,
--
Dave Abrahams
Boost Consulting
www.boost-consulting.com
Hi,
some time ago, I proposed a string method "dedent" (which currently is in the
textwrap module). The RFE is at http://python.org/sf/1237680.
Any opinions? If I don't get positive comments, I'll reject it.
Reinhold
--
Mail address is perfectly valid!
I just finished debugging some code that broke after upgrading to
Python 2.4 (from 2.3). Turns out the code was testing list iterators
for their boolean value (to distinguish them from None). In 2.3, a
list iterator (like any iterator) is always true. In 2.4, an exhausted
list iterator is false; probably by virtue of having a __len__()
method that returns the number of remaining items.
I realize that this was a deliberate feature, and that it exists in
2.4 as well as in 2.4.1 and will in 2.4.2; yet, I'm not sure I *like*
it. Was this breakage (which is not theoretical!) considered at all?
--
--Guido van Rossum (home page: http://www.python.org/~guido/)
Hi, has anybody considered adding something like this:
a = [1, 2]
[ 'x', *a, 'y']
as syntactic sugar for
a = [1, 2]
[ 'x' ] + a + [ 'y' ].
Notes:
- This is a common operation
- To me, the splicing form looks much better than the
concatenation form
- It can be implemented more efficiently than a bunch of list
concatenations
- This would be similar to the "apply" feature [ foo(*a) ]. The
'*' operator is reminiscent of dereferencing operators in
C-derived languages. Alternatively it could be '@', like perl's
implicit array splicing, and Lisp's ',@' in backquoted lists.
- (I know "splicing" traditionally refers to something else in
Python, but other languages refer to this as splicing, and the
English definition of "splice" applies better to this.)
--
Karl 2005-09-18 18:26
According to this list's welcome message, I should introduce myself.
I'm Bruce Eckel, I write, consult and give seminars about computer
programming, I use Python whenever I can get away with it, and I've
spoken at Pycon a number of times. There are further URLs in my
signature at the bottom.
I'm joining this list because I was having a conversation with Guido
about concurrency support in Python (in particular, using an
actor/active-object approach) and he suggested I bring it over
here. Here are the highlights of my most recent reply to him (Guido's
remarks are marked by the '>' signs):
> Agreed. IMO the pthreads-style solution doesn't work well no matter
> *what* the programming model.
Exactly. I think the problem may be that low-level threads people are
one step behind (much like embedded systems programmers, which is
where I began). They may be just now catching up to objects, but as
far as concurrency goes their brains still think in terms of threads,
so it seems natural to apply thread concepts to objects.
But from an OO standpoint, pthread-style thinking is two steps
backwards. You effectively throw open the innards of the object that
you just spent time decoupling from the rest of your system, and the
coupling is now unpredictable. It's the worst of all possible worlds.
> I worked on Mobile Agents at CNRI from 1995 till 2000, but they were
> very heavy-weight. Yet, I think that any solution that moves objects
> between CPUs on a regular basis is going to suffer the same problem
> that process switching in general gets you -- the move is so expensive
> that it's hard to decide when it's justified.
The examples I've seen haven't relied on mobility, so that's a
question: how important is mobility really to an agent system? And
could it be done in a trivial fashion in Python, by sending source
code. And if it was really necessary sometimes, could it be something
where the cost of mobility only occurred when moving an object?
It's possible that either an argument can be made against mobility
and/or some kind of trivial mobility system could be developed that
would work when necessary. Possibly a Linda-like put-take system with
pickles (off the top of my head).
I don't know that much about mobility, but it does seem like mobile
agents could be a powerful solution to the problem solved by
enterprise messaging systems.
> I believe you pointed me to Active Objects before, right? (Years ago,
> maybe when you visited us at Zope.)
I may have mentioned them in the past.
> If multiple active objects can co-exist in the same *process*, but
> nevertheless the language implementation prevents them from sharing
> data except via channels, and in addition allows dynamic reallocation
> of active objects across multiple CPUs, they just might be the ticket.
> But it would require a complete implementation to prove it.
Yes, defining an class as "active" would:
1) Install a worker thread and concurrent queue in each object of that
class.
2) Automatically turn method calls into tasks and enqueue them
3) Prevent any other interaction other than enqueued messages
At that point, the only time the GIL might be needed is to lock the
queue at the point of putting ant taking objects. But there has been
work done on lock-free data structures, which has been incorporated
into Java 5 libraries, so it might even be possible to use a lock-free
queue to do this and thus eliminate the need for the GIL at all.
Since the enqueuing process serializes all requests to active objects,
and since each message-task runs to completion before the next one
starts, the problem of threads interfering with each other is
eliminated. Also, the enqueuing process will always happen, so even if
the queue blocks it will be for a very short, deterministic time. So
the theory is that Active Object systems cannot deadlock (although I
believe they can livelock).
So yes, the best way for this to work might be some kind of enforced
implementation -- but forcing you to do something is not particularly
Pythonic, so I would think that it would be possible to build an
active object framework along with some checking tools to warn you if
you are breaking the rules. But what's not clear is whether this would
require knowledge of the innards of the Python interpreter (which I
don't have) or if it could be built using libraries.
BTW: I think that Hoare's Communicating Sequential Processes (CSP)
basically describes the idea of active objects but without objects.
That is, I think active objects is CSP applied to OO.
Bruce Eckel http://www.BruceEckel.com
Contains electronic books: "Thinking in Java 3e" & "Thinking in C++ 2e"
Web log: http://www.artima.com/weblogs/index.jsp?blogger=beckel
