On Wed, 10 Nov 2004, John P Speno wrote:
Hi, sorry for the delayed response.
> While using subprocess (aka popen5), I came across one potential gotcha. I've had
> exceptions ending like this:
>
> File "test.py", line 5, in test
> cmd = popen5.Popen(args, stdout=PIPE)
> File "popen5.py", line 577, in __init__
> data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
> OSError: [Errno 4] Interrupted system call
>
> (on Solaris 9)
>
> Would it make sense for subprocess to use a more robust read() function
> which can handle these cases, i.e. when the parent's read on the pipe
> to the child's stderr is interrupted by a system call, and returns EINTR?
> I imagine it could catch EINTR and EAGAIN and retry the failed read().
I assume you are using signals in your application? The os.read above is
not the only system call that can fail with EINTR. subprocess.py is full
of other system calls that can fail, and I suspect that many other Python
modules are as well.
I've made a patch (attached) to subprocess.py (and test_subprocess.py)
that should guard against EINTR, but I haven't committed it yet. It's
quite large.
Are Python modules supposed to handle EINTR? Why not let the C code handle
this? Or, perhaps the signal module should provide a sigaction function,
so that users can use SA_RESTART.
Index: subprocess.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/subprocess.py,v
retrieving revision 1.8
diff -u -r1.8 subprocess.py
--- subprocess.py 7 Nov 2004 14:30:34 -0000 1.8
+++ subprocess.py 17 Nov 2004 19:42:30 -0000
@@ -888,6 +888,50 @@
pass
+ def _read_no_intr(self, fd, buffersize):
+ """Like os.read, but retries on EINTR"""
+ while True:
+ try:
+ return os.read(fd, buffersize)
+ except OSError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
+
+ def _read_all(self, fd, buffersize):
+ """Like os.read, but retries on EINTR, and reads until EOF"""
+ all = ""
+ while True:
+ data = self._read_no_intr(fd, buffersize)
+ all += data
+ if data == "":
+ return all
+
+
+ def _write_no_intr(self, fd, s):
+ """Like os.write, but retries on EINTR"""
+ while True:
+ try:
+ return os.write(fd, s)
+ except OSError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
+ def _waitpid_no_intr(self, pid, options):
+ """Like os.waitpid, but retries on EINTR"""
+ while True:
+ try:
+ return os.waitpid(pid, options)
+ except OSError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
def _execute_child(self, args, executable, preexec_fn, close_fds,
cwd, env, universal_newlines,
startupinfo, creationflags, shell,
@@ -963,7 +1007,7 @@
exc_value,
tb)
exc_value.child_traceback = ''.join(exc_lines)
- os.write(errpipe_write, pickle.dumps(exc_value))
+ self._write_no_intr(errpipe_write, pickle.dumps(exc_value))
# This exitcode won't be reported to applications, so it
# really doesn't matter what we return.
@@ -979,7 +1023,7 @@
os.close(errwrite)
# Wait for exec to fail or succeed; possibly raising exception
- data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
+ data = self._read_all(errpipe_read, 1048576) # Exceptions limited to 1 MB
os.close(errpipe_read)
if data != "":
child_exception = pickle.loads(data)
@@ -1003,7 +1047,7 @@
attribute."""
if self.returncode == None:
try:
- pid, sts = os.waitpid(self.pid, os.WNOHANG)
+ pid, sts = self._waitpid_no_intr(self.pid, os.WNOHANG)
if pid == self.pid:
self._handle_exitstatus(sts)
except os.error:
@@ -1015,7 +1059,7 @@
"""Wait for child process to terminate. Returns returncode
attribute."""
if self.returncode == None:
- pid, sts = os.waitpid(self.pid, 0)
+ pid, sts = self._waitpid_no_intr(self.pid, 0)
self._handle_exitstatus(sts)
return self.returncode
@@ -1049,27 +1093,33 @@
stderr = []
while read_set or write_set:
- rlist, wlist, xlist = select.select(read_set, write_set, [])
+ try:
+ rlist, wlist, xlist = select.select(read_set, write_set, [])
+ except select.error, e:
+ if e[0] == errno.EINTR:
+ continue
+ else:
+ raise
if self.stdin in wlist:
# When select has indicated that the file is writable,
# we can write up to PIPE_BUF bytes without risk
# blocking. POSIX defines PIPE_BUF >= 512
- bytes_written = os.write(self.stdin.fileno(), input[:512])
+ bytes_written = self._write_no_intr(self.stdin.fileno(), input[:512])
input = input[bytes_written:]
if not input:
self.stdin.close()
write_set.remove(self.stdin)
if self.stdout in rlist:
- data = os.read(self.stdout.fileno(), 1024)
+ data = self._read_no_intr(self.stdout.fileno(), 1024)
if data == "":
self.stdout.close()
read_set.remove(self.stdout)
stdout.append(data)
if self.stderr in rlist:
- data = os.read(self.stderr.fileno(), 1024)
+ data = self._read_no_intr(self.stderr.fileno(), 1024)
if data == "":
self.stderr.close()
read_set.remove(self.stderr)
Index: test/test_subprocess.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v
retrieving revision 1.14
diff -u -r1.14 test_subprocess.py
--- test/test_subprocess.py 12 Nov 2004 15:51:48 -0000 1.14
+++ test/test_subprocess.py 17 Nov 2004 19:42:30 -0000
@@ -7,6 +7,7 @@
import tempfile
import time
import re
+import errno
mswindows = (sys.platform == "win32")
@@ -35,6 +36,16 @@
fname = tempfile.mktemp()
return os.open(fname, os.O_RDWR|os.O_CREAT), fname
+ def read_no_intr(self, obj):
+ while True:
+ try:
+ return obj.read()
+ except IOError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
#
# Generic tests
#
@@ -123,7 +134,7 @@
p = subprocess.Popen([sys.executable, "-c",
'import sys; sys.stdout.write("orange")'],
stdout=subprocess.PIPE)
- self.assertEqual(p.stdout.read(), "orange")
+ self.assertEqual(self.read_no_intr(p.stdout), "orange")
def test_stdout_filedes(self):
# stdout is set to open file descriptor
@@ -151,7 +162,7 @@
p = subprocess.Popen([sys.executable, "-c",
'import sys; sys.stderr.write("strawberry")'],
stderr=subprocess.PIPE)
- self.assertEqual(remove_stderr_debug_decorations(p.stderr.read()),
+ self.assertEqual(remove_stderr_debug_decorations(self.read_no_intr(p.stderr)),
"strawberry")
def test_stderr_filedes(self):
@@ -186,7 +197,7 @@
'sys.stderr.write("orange")'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
- output = p.stdout.read()
+ output = self.read_no_intr(p.stdout)
stripped = remove_stderr_debug_decorations(output)
self.assertEqual(stripped, "appleorange")
@@ -220,7 +231,7 @@
stdout=subprocess.PIPE,
cwd=tmpdir)
normcase = os.path.normcase
- self.assertEqual(normcase(p.stdout.read()), normcase(tmpdir))
+ self.assertEqual(normcase(self.read_no_intr(p.stdout)), normcase(tmpdir))
def test_env(self):
newenv = os.environ.copy()
@@ -230,7 +241,7 @@
'sys.stdout.write(os.getenv("FRUIT"))'],
stdout=subprocess.PIPE,
env=newenv)
- self.assertEqual(p.stdout.read(), "orange")
+ self.assertEqual(self.read_no_intr(p.stdout), "orange")
def test_communicate(self):
p = subprocess.Popen([sys.executable, "-c",
@@ -305,7 +316,8 @@
'sys.stdout.write("\\nline6");'],
stdout=subprocess.PIPE,
universal_newlines=1)
- stdout = p.stdout.read()
+
+ stdout = self.read_no_intr(p.stdout)
if hasattr(open, 'newlines'):
# Interpreter with universal newline support
self.assertEqual(stdout,
@@ -343,7 +355,7 @@
def test_no_leaking(self):
# Make sure we leak no resources
- max_handles = 1026 # too much for most UNIX systems
+ max_handles = 10 # too much for most UNIX systems
if mswindows:
max_handles = 65 # a full test is too slow on Windows
for i in range(max_handles):
@@ -424,7 +436,7 @@
'sys.stdout.write(os.getenv("FRUIT"))'],
stdout=subprocess.PIPE,
preexec_fn=lambda: os.putenv("FRUIT", "apple"))
- self.assertEqual(p.stdout.read(), "apple")
+ self.assertEqual(self.read_no_intr(p.stdout), "apple")
def test_args_string(self):
# args is a string
@@ -457,7 +469,7 @@
p = subprocess.Popen(["echo $FRUIT"], shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertEqual(p.stdout.read().strip(), "apple")
+ self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple")
def test_shell_string(self):
# Run command through the shell (string)
@@ -466,7 +478,7 @@
p = subprocess.Popen("echo $FRUIT", shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertEqual(p.stdout.read().strip(), "apple")
+ self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple")
def test_call_string(self):
# call() function with string argument on UNIX
@@ -525,7 +537,7 @@
p = subprocess.Popen(["set"], shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertNotEqual(p.stdout.read().find("physalis"), -1)
+ self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1)
def test_shell_string(self):
# Run command through the shell (string)
@@ -534,7 +546,7 @@
p = subprocess.Popen("set", shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertNotEqual(p.stdout.read().find("physalis"), -1)
+ self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1)
def test_call_string(self):
# call() function with string argument on Windows
/Peter Åstrand <astrand(a)lysator.liu.se>
Hi,
I'm having 2 problems with the current cvs :
During compilation this warning occurs:
*** WARNING: renaming "dbm" since importing it failed: build/lib.linux-i686-2.5/
dbm.so: undefined symbol: dbm_firstkey
and the 'dbm' module is unavailable.
I'm running MandrakeLinux 2005 (10.2) gcc 3.4.3
(I'm also having this problem when compiling python 2.3.5 or 2.4.1)
furthermore the 'make install' of current cvs fails halfway trough
with the following errors:
.....
.....
Compiling /opt/python25/lib/python2.5/bsddb/test/test_associate.py ...
Sorry: TabError: ('inconsistent use of tabs and spaces in indentation',
('/opt/python25/lib/python2.5/bsddb/test/test_associate.py', 97, 23, '\t
os.mkdir(homeDir)\n'))
Compiling /opt/python25/lib/python2.5/bsddb/test/test_basics.py ...
Sorry: TabError: ('inconsistent use of tabs and spaces in indentation',
('/opt/python25/lib/python2.5/bsddb/test/test_basics.py', 400, 26, '\t if
get_raises_error:\n'))
Compiling /opt/python25/lib/python2.5/bsddb/test/test_compare.py ...
Sorry: TabError: ('inconsistent use of tabs and spaces in indentation',
('/opt/python25/lib/python2.5/bsddb/test/test_compare.py', 167, 5, '\t"""\n'))
.....
.....
Compiling /opt/python25/lib/python2.5/bsddb/test/test_recno.py ...
Sorry: TabError: ('inconsistent use of tabs and spaces in indentation',
('/opt/python25/lib/python2.5/bsddb/test/test_recno.py', 38, 46, '\tget_returns_none =
d.set_get_returns_none(2)\n'))
.....
.....
make: *** [libinstall] Error 1
$
And then it quits.
Fixing the tab indentation errors locally makes the problem go away.
Regards,
Irmen de Jong
With 343 accepted, we can now add __enter__() and __exit__() methods to
objects.
What term should describe those objects in the documentation?
For instance, if the new magic methods are added to decimal.Context(),
do we then describe Context objects as "withable" ;-)
The PEP itself doesn't provide much of a clue. The terms "template",
"wrapper", and "block" are over-broad and do not provide a metaphor for
setup/teardown, r entry/exit, or initialization/finalization.
Whatever term is selected, it should be well thought-out and added to
the glossary. The choice of words will likely have a great effect on
learnability and on how people think about the new tool.
Raymond
Hi,
while bugs and patches are sometimes tricky to close, RFE can be very easy
to decide whether to implement in the first place. So what about working a
bit on this front? Here are several RFE reviewed, perhaps some can be
closed ("should" is always from submitter's point of view):
1193128:
str.translate(None, delchars) should be allowed and only delete delchars
from the string.
Review: may be a good feature, although string._idmap can be passed as the
first parameter too.
1226256:
The "path" module by Jason Orendorff should be in the standard library.
http://www.jorendorff.com/articles/python/path/
Review: the module is great and seems to have a large user base. On c.l.py
there are frequent praises about it.
1216944:
urllib(2) should gain a dict mapping HTTP status codes to the correspondig
status/error text.
Review: I can't see anything speaking against it.
1214675:
warnings should get a removefilter() method. An alternative would be to
fully document the "filters" attribute to allow direct tinkering with it.
Review: As mwh said in a comment, this isn't Java, so the latter may be
the way to go.
1205239:
Shift operands should be allowed to be negative integers, so e.g.
a << -2 is the same as a >> 2.
Review: Allowing this would open a source of bugs previously well identifiable.
1152248:
In order to read "records" separated by something other than newline, file objects
should either support an additional parameter (the separator) to (x)readlines(),
or gain an additional method which does this.
Review: The former is a no-go, I think, because what is read won't be lines.
The latter is further complicating the file interface, so I would follow the
principle that not every 3-line function should be builtin.
1110010:
A function "attrmap" should be introduced which is used as follows:
attrmap(x)['att'] == getattr(x, 'att')
The submitter mentions the use case of new-style classes without a __dict__ used
at the right of %-style string interpolation.
Review: I don't know whether this is worth it.
1052098:
An environment variable should be supported to set the default encoding.
Review: If one wants this for a single application, he can still implement it himself.
985094:
getattr should support a callable as the second argument, used as follows:
getattr(obj, func) == func(obj)
Review: Highly unintuitive to me.
That's all for today; sorry if it was too much ;)
Reinhold
--
Mail address is perfectly valid!
IIRC, there was a decision to not implement phase C and to keep the
trailing L in representations of long integers.
If so, I believe the PEP can be marked as final. We've done all we're
going to do.
Raymond
I've been looking at the API for sets.Set and built-in set objects in
Python 2.4.1 and I think I have found some minor inconsistencies.
Background: we have an object that is very similar to "sets" and we
originally modeled the API after sets.Set since we started with Python
2.3. Now I'm trying to update the API so that it's consistent with
Python 2.4's built-in set object and I've noticed the following
discrepancies.
set.Set has both an .update() method and a .union_update() method. They
appear to be completely identical, accepting either a Set object or an
arbitrary sequence. This is the case despite the docstring difference
for these two methods and the fact that Set.update() isn't documented on
the texinfo page.
Built-in set has only .update() though but it acts just like the set.Set
methods above. Note that of all these methods, only Set.update() is
documented to take an iterable argument.
These inconsistencies could prove a bit problematic when porting Py2.3
applications using sets.Set to Py2.4 using built-in set. I'd like to
fix this for Python 2.4.2, and I think the changes are pretty minimal.
If there are no objections, I propose to do the following (only in
Python 2.4 and 2.5):
* Add set.union_update() as an alias for set.update().
* Add to docstrings for all methods that 't' can be any iterable.
* Update texinfo documentation to add Set.update() and
set.union_update() and explain that all can take iterables
I consider this a bug in 2.4, not a new feature, because without it, it
makes more work in porting applications.
Thoughts?
I'm willing to Just Fix It, but if someone wants to see a patch first,
I'll be happy to generate it and post it to SF.
-Barry
Hello,
I was asking about a problem I was having over on the C++-python list,
and they suggested I report it here as a possible Python problem.
I was getting bus errors with a C module I was linking to, so factored
it down too a very small example that reproduced the problem. Here it
is:
#include <Python.h>
static double gfSumChiSquare = 123.0;
static PyObject *
getSumChiSquare(PyObject *self, PyObject *args){
return Py_BuildValue("d", gfSumChiSquare);
}
static PyMethodDef SimMethods[] = {
{"getSumChiSquare", getSumChiSquare, METH_NOARGS, "Return
fSumChiSquare"},
{NULL, NULL, 0, NULL} /* Sentinel */
};
PyMODINIT_FUNC
inittestfloat(void)
{
(void) Py_InitModule("testfloat", SimMethods);
}
That caused a bus error 100% of the time when I simply imported the
module into Python and called getSumChiSquare(), i.e.:
>>> import testfloat
>>> testfloat.getSumChiSquare()
However, the problem seems to go away if I use METH_VARARGS, and parse
the non-existent args with
PyArg_ParseTuple:
#include <Python.h>
static double gfSumChiSquare = 123.0;
static PyObject *
getSumChiSquare(PyObject *self, PyObject *args){
if (!PyArg_ParseTuple(args, "", NULL))
return NULL;
return Py_BuildValue("d", gfSumChiSquare);
}
static PyMethodDef SimMethods[] = {
{"getSumChiSquare", getSumChiSquare, METH_VARARGS, "Return
fSumChiSquare"},
{NULL, NULL, 0, NULL} /* Sentinel */
};
PyMODINIT_FUNC
inittestfloat(void)
{
(void) Py_InitModule("testfloat", SimMethods);
}
This approach seems to work reliably -- at least variations I've tried
haven't caused a bus error. But I haven't been able to discern an
explanation from the docs as to why this would be better. The docs say
that both METH_VARARGS and METH_NOARGS expect a PyCFunction. So if I am
calling the function with no arguments, why can't I use METH_NOARGS and
skip the call to PyArg_ParseTuple?
Could it be that this is a python bug? Or am I doing something wrong?
Note: this is using Python 2.3 on OS X:
Python 2.3 (#1, Sep 13 2003, 00:49:11)
Thanks in advance for any help or insight you can give,
Gary
--
Gary Robinson
CTO
Emergent Music, LLC
grobinson(a)goombah.com
207-942-3463
Company: http://www.goombah.com
Blog: http://www.garyrobinson.net
I was trying to compile a python plugin (for gimp) using the MSYS shell and
the MINGW compiler.
I now have put this in 'sitecustomize'
$ cat /c/Python24/lib/site-packages/sitecustomize.py
import sys
import os
import re
if os.environ.get("MSYSTEM") == "MINGW32":
os.sep='/'
os.pathsep='/'
sys.prefix = re.sub('\\\\','/',sys.prefix)
sys.exec_prefix = re.sub('\\\\','/',sys.exec_prefix)
It would probably be better to have python detect that it's running from
inside the msys shell,
and output '/'es instead of '\'es.
maybe someone could extend os.path to do this in the standard distribution:
implement an msyspath.py, which calls ntpath for each function, and does a
replace at the end of the evaluation. Unfortunately, I'm just starting to
look into python, so I do not know what the cleanest implementation of this
would be...
Hi,
I'm sorry, I don't seem to have done a very good job at explaining
the situation. I'll try again:
'getch()' is a low-level function provided on Windows to capture a
single character of input from a user, /without echoing it to the
screen/. As far as I can tell there's no other way of doing this with
Python on Windows. The Python interface to this function is in the C
code in msvcrtmodule.c, which has a (very thin) wrapper around the raw
OS system call. Microsoft provide a way of accepting both normal ASCII
codes, and extended characters via this system call. Currently, the
Python wrapper in msvcrtmodule.c only supports the semantics of getting
the bare ASCII codes, and not the extended characters. I would strongly
suggest that it should support both.
So, I guess in answer to the questions raised below;
1) I can't do it in Python code without getch() (hence the original
email)
2) I would argue that in fact getch() is 'broken' in its current Python
implementation, as it doesn't support what the OS implies /should/ be
supported (and, indeed, if I input an extended character in response to
a 'getch()' call, all I get back currently is an empty string).
Hope this helps,
D
Fredrik wrote:
>Darryl Dixon wrote:
>
>> Microsoft support capturing extended characters via _getch, but it requires making a
>> second call to getch() if one of the 'magic' returns is encountered in the first call (0x00
>> or 0xE0).
>
>so why not do that in your python code?
>
>> The relevant chunk of code in Python that would probably need to be
>> changed to support this appears to be in msvcrtmodule.c:
>
>if you change msvcrt, you'll break all the programs that uses getch() in
>the prescribed way...
>
></F>
--
Darryl Dixon <esrever_otua(a)pythonhacker.is-a-geek.net>
