On Wed, 10 Nov 2004, John P Speno wrote:
Hi, sorry for the delayed response.
> While using subprocess (aka popen5), I came across one potential gotcha. I've had
> exceptions ending like this:
>
> File "test.py", line 5, in test
> cmd = popen5.Popen(args, stdout=PIPE)
> File "popen5.py", line 577, in __init__
> data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
> OSError: [Errno 4] Interrupted system call
>
> (on Solaris 9)
>
> Would it make sense for subprocess to use a more robust read() function
> which can handle these cases, i.e. when the parent's read on the pipe
> to the child's stderr is interrupted by a system call, and returns EINTR?
> I imagine it could catch EINTR and EAGAIN and retry the failed read().
I assume you are using signals in your application? The os.read above is
not the only system call that can fail with EINTR. subprocess.py is full
of other system calls that can fail, and I suspect that many other Python
modules are as well.
I've made a patch (attached) to subprocess.py (and test_subprocess.py)
that should guard against EINTR, but I haven't committed it yet. It's
quite large.
Are Python modules supposed to handle EINTR? Why not let the C code handle
this? Or, perhaps the signal module should provide a sigaction function,
so that users can use SA_RESTART.
Index: subprocess.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/subprocess.py,v
retrieving revision 1.8
diff -u -r1.8 subprocess.py
--- subprocess.py 7 Nov 2004 14:30:34 -0000 1.8
+++ subprocess.py 17 Nov 2004 19:42:30 -0000
@@ -888,6 +888,50 @@
pass
+ def _read_no_intr(self, fd, buffersize):
+ """Like os.read, but retries on EINTR"""
+ while True:
+ try:
+ return os.read(fd, buffersize)
+ except OSError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
+
+ def _read_all(self, fd, buffersize):
+ """Like os.read, but retries on EINTR, and reads until EOF"""
+ all = ""
+ while True:
+ data = self._read_no_intr(fd, buffersize)
+ all += data
+ if data == "":
+ return all
+
+
+ def _write_no_intr(self, fd, s):
+ """Like os.write, but retries on EINTR"""
+ while True:
+ try:
+ return os.write(fd, s)
+ except OSError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
+ def _waitpid_no_intr(self, pid, options):
+ """Like os.waitpid, but retries on EINTR"""
+ while True:
+ try:
+ return os.waitpid(pid, options)
+ except OSError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
def _execute_child(self, args, executable, preexec_fn, close_fds,
cwd, env, universal_newlines,
startupinfo, creationflags, shell,
@@ -963,7 +1007,7 @@
exc_value,
tb)
exc_value.child_traceback = ''.join(exc_lines)
- os.write(errpipe_write, pickle.dumps(exc_value))
+ self._write_no_intr(errpipe_write, pickle.dumps(exc_value))
# This exitcode won't be reported to applications, so it
# really doesn't matter what we return.
@@ -979,7 +1023,7 @@
os.close(errwrite)
# Wait for exec to fail or succeed; possibly raising exception
- data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB
+ data = self._read_all(errpipe_read, 1048576) # Exceptions limited to 1 MB
os.close(errpipe_read)
if data != "":
child_exception = pickle.loads(data)
@@ -1003,7 +1047,7 @@
attribute."""
if self.returncode == None:
try:
- pid, sts = os.waitpid(self.pid, os.WNOHANG)
+ pid, sts = self._waitpid_no_intr(self.pid, os.WNOHANG)
if pid == self.pid:
self._handle_exitstatus(sts)
except os.error:
@@ -1015,7 +1059,7 @@
"""Wait for child process to terminate. Returns returncode
attribute."""
if self.returncode == None:
- pid, sts = os.waitpid(self.pid, 0)
+ pid, sts = self._waitpid_no_intr(self.pid, 0)
self._handle_exitstatus(sts)
return self.returncode
@@ -1049,27 +1093,33 @@
stderr = []
while read_set or write_set:
- rlist, wlist, xlist = select.select(read_set, write_set, [])
+ try:
+ rlist, wlist, xlist = select.select(read_set, write_set, [])
+ except select.error, e:
+ if e[0] == errno.EINTR:
+ continue
+ else:
+ raise
if self.stdin in wlist:
# When select has indicated that the file is writable,
# we can write up to PIPE_BUF bytes without risk
# blocking. POSIX defines PIPE_BUF >= 512
- bytes_written = os.write(self.stdin.fileno(), input[:512])
+ bytes_written = self._write_no_intr(self.stdin.fileno(), input[:512])
input = input[bytes_written:]
if not input:
self.stdin.close()
write_set.remove(self.stdin)
if self.stdout in rlist:
- data = os.read(self.stdout.fileno(), 1024)
+ data = self._read_no_intr(self.stdout.fileno(), 1024)
if data == "":
self.stdout.close()
read_set.remove(self.stdout)
stdout.append(data)
if self.stderr in rlist:
- data = os.read(self.stderr.fileno(), 1024)
+ data = self._read_no_intr(self.stderr.fileno(), 1024)
if data == "":
self.stderr.close()
read_set.remove(self.stderr)
Index: test/test_subprocess.py
===================================================================
RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v
retrieving revision 1.14
diff -u -r1.14 test_subprocess.py
--- test/test_subprocess.py 12 Nov 2004 15:51:48 -0000 1.14
+++ test/test_subprocess.py 17 Nov 2004 19:42:30 -0000
@@ -7,6 +7,7 @@
import tempfile
import time
import re
+import errno
mswindows = (sys.platform == "win32")
@@ -35,6 +36,16 @@
fname = tempfile.mktemp()
return os.open(fname, os.O_RDWR|os.O_CREAT), fname
+ def read_no_intr(self, obj):
+ while True:
+ try:
+ return obj.read()
+ except IOError, e:
+ if e.errno == errno.EINTR:
+ continue
+ else:
+ raise
+
#
# Generic tests
#
@@ -123,7 +134,7 @@
p = subprocess.Popen([sys.executable, "-c",
'import sys; sys.stdout.write("orange")'],
stdout=subprocess.PIPE)
- self.assertEqual(p.stdout.read(), "orange")
+ self.assertEqual(self.read_no_intr(p.stdout), "orange")
def test_stdout_filedes(self):
# stdout is set to open file descriptor
@@ -151,7 +162,7 @@
p = subprocess.Popen([sys.executable, "-c",
'import sys; sys.stderr.write("strawberry")'],
stderr=subprocess.PIPE)
- self.assertEqual(remove_stderr_debug_decorations(p.stderr.read()),
+ self.assertEqual(remove_stderr_debug_decorations(self.read_no_intr(p.stderr)),
"strawberry")
def test_stderr_filedes(self):
@@ -186,7 +197,7 @@
'sys.stderr.write("orange")'],
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
- output = p.stdout.read()
+ output = self.read_no_intr(p.stdout)
stripped = remove_stderr_debug_decorations(output)
self.assertEqual(stripped, "appleorange")
@@ -220,7 +231,7 @@
stdout=subprocess.PIPE,
cwd=tmpdir)
normcase = os.path.normcase
- self.assertEqual(normcase(p.stdout.read()), normcase(tmpdir))
+ self.assertEqual(normcase(self.read_no_intr(p.stdout)), normcase(tmpdir))
def test_env(self):
newenv = os.environ.copy()
@@ -230,7 +241,7 @@
'sys.stdout.write(os.getenv("FRUIT"))'],
stdout=subprocess.PIPE,
env=newenv)
- self.assertEqual(p.stdout.read(), "orange")
+ self.assertEqual(self.read_no_intr(p.stdout), "orange")
def test_communicate(self):
p = subprocess.Popen([sys.executable, "-c",
@@ -305,7 +316,8 @@
'sys.stdout.write("\\nline6");'],
stdout=subprocess.PIPE,
universal_newlines=1)
- stdout = p.stdout.read()
+
+ stdout = self.read_no_intr(p.stdout)
if hasattr(open, 'newlines'):
# Interpreter with universal newline support
self.assertEqual(stdout,
@@ -343,7 +355,7 @@
def test_no_leaking(self):
# Make sure we leak no resources
- max_handles = 1026 # too much for most UNIX systems
+ max_handles = 10 # too much for most UNIX systems
if mswindows:
max_handles = 65 # a full test is too slow on Windows
for i in range(max_handles):
@@ -424,7 +436,7 @@
'sys.stdout.write(os.getenv("FRUIT"))'],
stdout=subprocess.PIPE,
preexec_fn=lambda: os.putenv("FRUIT", "apple"))
- self.assertEqual(p.stdout.read(), "apple")
+ self.assertEqual(self.read_no_intr(p.stdout), "apple")
def test_args_string(self):
# args is a string
@@ -457,7 +469,7 @@
p = subprocess.Popen(["echo $FRUIT"], shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertEqual(p.stdout.read().strip(), "apple")
+ self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple")
def test_shell_string(self):
# Run command through the shell (string)
@@ -466,7 +478,7 @@
p = subprocess.Popen("echo $FRUIT", shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertEqual(p.stdout.read().strip(), "apple")
+ self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple")
def test_call_string(self):
# call() function with string argument on UNIX
@@ -525,7 +537,7 @@
p = subprocess.Popen(["set"], shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertNotEqual(p.stdout.read().find("physalis"), -1)
+ self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1)
def test_shell_string(self):
# Run command through the shell (string)
@@ -534,7 +546,7 @@
p = subprocess.Popen("set", shell=1,
stdout=subprocess.PIPE,
env=newenv)
- self.assertNotEqual(p.stdout.read().find("physalis"), -1)
+ self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1)
def test_call_string(self):
# call() function with string argument on Windows
/Peter Åstrand <astrand(a)lysator.liu.se>
After I finally understood what thread-safety guarantees the Python
memory allocator needs to provide, I went and did some hard thinking
about the code this afternoon. I believe that my modifications provide
the same guarantees that the original version did. I do need to declare
the arenas array to be volatile, and leak the array when resizing it.
Please correct me if I am wrong, but the situation that needs to be
supported is this:
While one thread holds the GIL, any other thread can call PyObject_Free
with a pointer that was returned by the system malloc.
The following situation is *not* supported:
While one thread holds the GIL, another thread calls PyObject_Free with
a pointer that was returned by PyObject_Malloc.
I'm hoping that I got things a little better this time around. I've
submitted my updated patch to the patch tracker. For reference, I've
included links to SourceForge and the previous thread.
Thank you,
Evan Jones
Previous thread:
http://mail.python.org/pipermail/python-dev/2005-January/051255.html
Patch location:
http://sourceforge.net/tracker/index.php?
func=detail&aid=1123430&group_id=5470&atid=305470
Perhaps this is more approprate for python-list but I looks like a
bug to me. Example code:
class A:
def __str__(self):
return u'\u1234'
'%s' % u'\u1234' # this works
'%s' % A() # this doesn't work
It will work if 'A' subclasses from 'unicode' but should not be
necessary, IMHO. Any reason why this shouldn't be fixed?
Neil
This is something I've typed way too many times:
Py> class C():
File "<stdin>", line 1
class C():
^
SyntaxError: invalid syntax
It's the asymmetry with functions that gets to me - defining a function with no
arguments still requires parentheses in the definition statement, but defining a
class with no bases requires the parentheses to be omitted.
Which leads in to the real question: Does this *really* need to be a syntax
error? Or could it be used as an easier way to spell "class C(object):"?
Then, in Python 3K, simply drop support for omitting the parentheses from class
definitions - require inheriting from ClassicClass instead. This would also have
the benefit that the elimination of defaulting to classic classes would cause a
syntax error rather than subtle changes in behaviour.
Cheers,
Nick.
--
Nick Coghlan | ncoghlan(a)email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Hello
I've looked at one bug and a bunch of patches and
added a comment to them:
(bug) [ 1102649 ] pickle files should be opened in binary mode
Added a comment about a possible different solution
[ 946207 ] Non-blocking Socket Server
Useless, what are the mixins for? Recommend close
[ 756021 ] Allow socket.inet_aton('255.255.255.255') on Windows
Looks good but added suggestion about when to test for special case
[ 740827 ] add urldecode() method to urllib
I think it's better to group these things into urlparse
[ 579435 ] Shadow Password Support Module
Would be nice to have, I recently just couldn't do the user
authentication that I wanted: based on the users' unix passwords
[ 1093468 ] socket leak in SocketServer
Trivial and looks harmless, but don't the sockets
get garbage collected once the request is done?
[ 1049151 ] adding bool support to xdrlib.py
Simple patch and 2.4 is out now, so...
It would be nice if somebody could have a look at my
own patches or help me a bit with them:
[ 1102879 ] Fix for 926423: socket timeouts + Ctrl-C don't play nice
[ 1103213 ] Adding the missing socket.recvall() method
[ 1103350 ] send/recv SEGMENT_SIZE should be used more in socketmodule
[ 1062014 ] fix for 764437 AF_UNIX socket special linux socket names
[ 1062060 ] fix for 1016880 urllib.urlretrieve silently truncates dwnld
Some of them come from the last Python Bug Day, see
http://www.python.org/moin/PythonBugDayStatus
Thank you !
Regards,
--Irmen de Jong
Hello,
just felt a little bored and tried to review a few (no-brainer) patches.
Here are the results:
* Patch #1051395
Minor fix in Lib/locale.py: Docs say that function _parse_localename
returns a tuple; but for one codepath it returns a list.
Patch fixes this by adding tuple(), recommending apply.
* Patch #1046831
Minor fix in Lib/distutils/sysconfig.py: it defines a function to
retrieve the Python version but does not use it everywhere; Patch
fixes this, recommending apply.
* Patch #751031
Adds recognizing JPEG-EXIF files (produced by digicams) to imghdr.py.
Recommending apply.
* Patch #712317
Fixes URL parsing in urlparse for URLs such as http://foo?bar. Splits
at '?', so assigns 'foo' to netloc and 'bar' to query instead of
assigning 'foo?bar' to netloc. Recommending apply.
regards,
Reinhold
>>Overall, I have no major objections to the PEP or the patch. Before it
>>goes in on auto-pilot, it would be darned nice if the proponents said
>>that they've found it helpful in real code and that they are satisfied
>>with the timings.
>>
>>
>
>I guess "darned nice" is the best you can hope for. Not sure if Peter
>Harris is still around.
>
>Regards,
>Martin
>
>
Yes, I'm still lurking, slightly aghast that my little PEP is getting
such ferocious scrutiny. I would
have like some of that in time for it to go into 2.4, but I suppose you
should be careful what you
wish for.
I'll answer a few points from this thread, in no particular order:
My original desire was a built-in, but it was suggested that the first
step would be a Python
implementation in the standard library to try it out. That was the
basis for the PEP, and in fact
a C implementation would have been beyond my expertise.
However, I sympathise with anyone who feels unhappy about a new module
just for what amounts
to one function. I'd be happy to go back to the built-in, now someone
cleverer than I am has
written one. Sorry I can't rememeber your name, whoever you are. I'm
having trouble with my
e-mails.
I was never too bothered about efficiency, and still am not. For me it
was always primarily a
way to save typing or build call-back functions on the fly. The
discussion about using it to
make instancemethods and classmethods -- way over my head! I would count
that as something
weird enough to be worth spelling out in "plain Python", in my code anyway.
The PEP was scattered over a few patches because I wasn't too sure how
to go about it, so there
was my Python module, the C implementation, the unit tests and the docs
all in separate patches.
3/4 of that was my fault - sorry!
Once the PEP had been accepted, I didn't like to mess with it, which is
why I went quiet for a while.
It's gone past the point where I can personally contribute much, so I'd
just like to say thanks and
I look forward to the day when I can just use it.
Peter Harris
[After a long delay, the thread continues....]
Hi All,
I'm pushing ahead on the tasks necessary to add the 'fcponst' module
described in PEP 754: IEEE 754 Floating Point Special Values.
Per http://www.python.org/psf/contrib, I've
- Changed the license to the Apache License, Version 2.0
- Just faxed the "Contributor Agreement" to the PSF
I've also
- created a patch on sourceforge.net for fpconst code and documentation
(see
https://sourceforge.net/tracker/index.php?func=detail&aid=1151323&group_id=5
470&atid=305470)
I will need help connecting the included test functions into the python test
suite.
What else needs to be done to allow fpconst to go into the Python library?
-Greg
> -----Original Message-----
> From: Tim Peters [mailto:tim.one@comcast.net]
> Sent: Friday, December 10, 2004 1:43 PM
> To: Warnes, Gregory R; goodger(a)python.org
> Cc: peps(a)python.org
> Subject: RE: [Python-Dev] Re: PEP 754
>
>
> [Warnes, Gregory R]
> > Hi David & Tim,
> >
> > First, I really like to see this go forward. The fpconst module is
> > getting alot of use across the net, and it would be very useful to
> > finally get it into the standard python library. What
> needs to be done
> > to move forward?
>
> Looks to me like exactly the same stuff as was needed before.
> Guido needs
> to pronounce on it. It needs a patch on SourceForge, adding
> the new module,
> new docs, and a test suite run by Python's standard regrtest.py. The
> non-PSF copyright and license remain problematic. That last should be
> easier to deal with after the PSF approves a Contributor
> Agreement (probably
> within a month), but will be a show-stopper if Pfizer won't
> contribute the
> module under the terms of the Contributor Agreement (which
> can't be answered
> now by anyone, since the Contributor Agreement doesn't exist yet).
>
>
>
LEGAL NOTICE
Unless expressly stated otherwise, this message is confidential and may be privileged. It is intended for the addressee(s) only. Access to this E-mail by anyone else is unauthorized. If you are not an addressee, any disclosure or copying of the contents of this E-mail or any action taken (or not taken) in reliance on it is unauthorized and may be unlawful. If you are not an addressee, please inform the sender immediately.
Moving a discussion from the PEP309 SF tracker (Patch #941881) to here, since
it's gone beyond the initial PEP 309 concept (and the SF tracker is a lousy
place to have a design discussion, anyway).
The discussion started when Steven Bethard pointed out that partial objects
can't be used as instance methods (using new.instancemethod means that the
automatically supplied 'self' argument ends up in the wrong place, after the
originally supplied arguments).
This problem is mentioned only indirectly in the PEP (pointing out that a
version which prepended later arguments, rather than appending them might be
useful). Such a class wouldn't solve the problem anyway, as it is only the
*first* argument we want to give special treatment.
Keyword arguments won't help us, since the 'self' argument is always positional.
The initial suggestion was to provide a __get__ method on partial objects, which
forces the insertion of the reference to self at the beginning of the argument
list instead of at the end:
def __get__(self, obj, type=None):
if obj is None:
return self
return partial(self.fn, obj, *self.args, **self.kw)
However, this breaks down for nested partial functions - the call to the nested
partial again moves the 'self' argument to after the originally supplied
argument list. This can be addressed by automatically 'unfolding' nested
partials (which should also give a speed benefit when supplying arguments
piecemeal, since building incrementally or all at once will get you to the same
place):
def __init__(*args, **kw):
self = args[0]
try:
func = args[1]
except IndexError:
raise TypeError("Expected at least 2 arguments, got %s" % len(args))
if isinstance(func, partial):
self.fn = func.fn
self.args = func.args + args[2:]
d = func.kw.copy()
d.update(kw)
self.kw = d
else:
self.fn, self.args, self.kw = (func, args[2:], kw)
At this point, the one thing you can't do is use a partial function as a *class*
method, as the classmethod implementation doesn't give descriptors any special
treatment.
So, instead of the above, I propose the inclusion of a callable 'partialmethod'
descriptor in the functional module that takes the first positional argument
supplied at call time and prepends it in the actual function call (this still
requires automatic 'unfolding'in order to work correctly with nested partial
functions):
class partialmethod(partial):
def __call__(self, *args, **kw):
if kw and self.kw:
d = self.kw.copy()
d.update(kw)
else:
d = kw or self.kw
if args:
first = args[:1]
rest = args[1:]
else:
first = rest = ()
return self.fn(*(first + self.args + rest), **d)
def __get__(self, obj, type=None):
if obj is None:
return self
return partial(self.fn, obj, *self.args, **self.kw)
Using a function that simply prints its arguments:
Py> class C:
... a = functional.partialmethod(f, 'a')
... b = classmethod(functional.partialmethod(f, 'b'))
... c = staticmethod(functional.partial(f, 'c'))
... d = functional.partial(functional.partialmethod(f, 'd', 1), 2)
...
Py> C.e = new.instancemethod(functional.partialmethod(f, 'e'), None, C)
Py> C().a(0)
((<__main__.C instance at 0x00A95710>, 'a'), {}, 0)
Py> C().b(0)
(<class __main__.C at 0x00A93FC0>, 'b', 0)
Py> C().c(0)
('c', 0)
Py> C().d(0)
('d', 1, 2, 0)
Py> C().e(0)
(<__main__.C instance at 0x00A95710>, 'e', 0)
Notice that you *don't* want to use partialmethod when creating a static method.
Cheers,
Nick.
--
Nick Coghlan | ncoghlan(a)email.com | Brisbane, Australia
---------------------------------------------------------------
http://boredomandlaziness.skystorm.net
Paul Moore wrote:
> Personally, I'd rather see partial as it stands, with its current
> limitations, included. The alternative seems to be a potentially long
> discussion, petering out without conclusion, and the whole thing
> missing Python 2.5. (I know that's a long way off, but this already
> happened with 2.4...)
-0 My preference is that it not go in as-is.
It is better to teach how to write a closure than to introduce a new
construct that has its own problems and doesn't provide a real
improvement over what we have now. Despite my enthusiasm for functional
programming and the ideas in PEP, I find the PFA implementation
annoying.
I had tried out the implementation that was being pushed for Py2.4 and
found it wanting. Hopefully, it has improved since then. Here are a
few thoughts based on trying to apply it to my existing code.
* The PFA implementation proposed for Py2.4 ran slower than an
equivalent closure. If the latest implementation offers better
performance, then that may be a reason for having it around.
* Having PFA only handle the first argument was a PITA with Python. In
Haskell and ML, function signatures seems to have been designed with
argument ordering better suited for left curries. In contrast, Python
functions seem to have adverse argument ordering where the variables you
want to freeze appear toward the right. This is subjective and may just
reflect the way I was aspiring to use "partial" to freeze options and
flags rather than the first argument of a binary operator. Still, I
found closures to be more flexible in that they could handle any
argument pattern and could freeze more than one variable or keyword at a
time.
* The instance method limitation never came up for me. However, it
bites to have a tool working in a way that doesn't match your mental
model. We have to document the limitations, keep them in mind while
programming, and hope to remember them as possible causes if bugs ever
arise. It would be great if these limitations could get ironed out.
* Using the word "partial" instead of "lambda" traded one bit of
unreadability for another. The "lambda" form was often better because
it didn't abstract away its mechanism and because it supported more
general expressions. I found that desk-checking code was harder because
I had to mentally undo the abstraction to check that the argument
signature was correct.
* It is not clear that the proposed implementation achieves one of the
principal benefits laid out in the PEP: "I agree that lambda is usually
good enough, just not always. And I want the possibility of useful
introspection and subclassing."
If we get a better implementation, it would be nice if the PEP were
updated with better examples. The TkInter example is weak because we
often want to set multiple defaults at the same time (foreground,
background, textsize, etc) and often those values are config options
rather than hardwired constants.
Raymond
