Python-Dev April 2006

python-dev@python.org

136 participants
278 discussions

by Peter Astrand

On Wed, 10 Nov 2004, John P Speno wrote: Hi, sorry for the delayed response. > While using subprocess (aka popen5), I came across one potential gotcha. I've had > exceptions ending like this: > > File "test.py", line 5, in test > cmd = popen5.Popen(args, stdout=PIPE) > File "popen5.py", line 577, in __init__ > data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB > OSError: [Errno 4] Interrupted system call > > (on Solaris 9) > > Would it make sense for subprocess to use a more robust read() function > which can handle these cases, i.e. when the parent's read on the pipe > to the child's stderr is interrupted by a system call, and returns EINTR? > I imagine it could catch EINTR and EAGAIN and retry the failed read(). I assume you are using signals in your application? The os.read above is not the only system call that can fail with EINTR. subprocess.py is full of other system calls that can fail, and I suspect that many other Python modules are as well. I've made a patch (attached) to subprocess.py (and test_subprocess.py) that should guard against EINTR, but I haven't committed it yet. It's quite large. Are Python modules supposed to handle EINTR? Why not let the C code handle this? Or, perhaps the signal module should provide a sigaction function, so that users can use SA_RESTART. Index: subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/subprocess.py,v retrieving revision 1.8 diff -u -r1.8 subprocess.py --- subprocess.py 7 Nov 2004 14:30:34 -0000 1.8 +++ subprocess.py 17 Nov 2004 19:42:30 -0000 @@ -888,6 +888,50 @@ pass + def _read_no_intr(self, fd, buffersize): + """Like os.read, but retries on EINTR""" + while True: + try: + return os.read(fd, buffersize) + except OSError, e: + if e.errno == errno.EINTR: + continue + else: + raise + + + def _read_all(self, fd, buffersize): + """Like os.read, but retries on EINTR, and reads until EOF""" + all = "" + while True: + data = self._read_no_intr(fd, buffersize) + all += data + if data == "": + return all + + + def _write_no_intr(self, fd, s): + """Like os.write, but retries on EINTR""" + while True: + try: + return os.write(fd, s) + except OSError, e: + if e.errno == errno.EINTR: + continue + else: + raise + + def _waitpid_no_intr(self, pid, options): + """Like os.waitpid, but retries on EINTR""" + while True: + try: + return os.waitpid(pid, options) + except OSError, e: + if e.errno == errno.EINTR: + continue + else: + raise + def _execute_child(self, args, executable, preexec_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell, @@ -963,7 +1007,7 @@ exc_value, tb) exc_value.child_traceback = ''.join(exc_lines) - os.write(errpipe_write, pickle.dumps(exc_value)) + self._write_no_intr(errpipe_write, pickle.dumps(exc_value)) # This exitcode won't be reported to applications, so it # really doesn't matter what we return. @@ -979,7 +1023,7 @@ os.close(errwrite) # Wait for exec to fail or succeed; possibly raising exception - data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB + data = self._read_all(errpipe_read, 1048576) # Exceptions limited to 1 MB os.close(errpipe_read) if data != "": child_exception = pickle.loads(data) @@ -1003,7 +1047,7 @@ attribute.""" if self.returncode == None: try: - pid, sts = os.waitpid(self.pid, os.WNOHANG) + pid, sts = self._waitpid_no_intr(self.pid, os.WNOHANG) if pid == self.pid: self._handle_exitstatus(sts) except os.error: @@ -1015,7 +1059,7 @@ """Wait for child process to terminate. Returns returncode attribute.""" if self.returncode == None: - pid, sts = os.waitpid(self.pid, 0) + pid, sts = self._waitpid_no_intr(self.pid, 0) self._handle_exitstatus(sts) return self.returncode @@ -1049,27 +1093,33 @@ stderr = [] while read_set or write_set: - rlist, wlist, xlist = select.select(read_set, write_set, []) + try: + rlist, wlist, xlist = select.select(read_set, write_set, []) + except select.error, e: + if e[0] == errno.EINTR: + continue + else: + raise if self.stdin in wlist: # When select has indicated that the file is writable, # we can write up to PIPE_BUF bytes without risk # blocking. POSIX defines PIPE_BUF >= 512 - bytes_written = os.write(self.stdin.fileno(), input[:512]) + bytes_written = self._write_no_intr(self.stdin.fileno(), input[:512]) input = input[bytes_written:] if not input: self.stdin.close() write_set.remove(self.stdin) if self.stdout in rlist: - data = os.read(self.stdout.fileno(), 1024) + data = self._read_no_intr(self.stdout.fileno(), 1024) if data == "": self.stdout.close() read_set.remove(self.stdout) stdout.append(data) if self.stderr in rlist: - data = os.read(self.stderr.fileno(), 1024) + data = self._read_no_intr(self.stderr.fileno(), 1024) if data == "": self.stderr.close() read_set.remove(self.stderr) Index: test/test_subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v retrieving revision 1.14 diff -u -r1.14 test_subprocess.py --- test/test_subprocess.py 12 Nov 2004 15:51:48 -0000 1.14 +++ test/test_subprocess.py 17 Nov 2004 19:42:30 -0000 @@ -7,6 +7,7 @@ import tempfile import time import re +import errno mswindows = (sys.platform == "win32") @@ -35,6 +36,16 @@ fname = tempfile.mktemp() return os.open(fname, os.O_RDWR|os.O_CREAT), fname + def read_no_intr(self, obj): + while True: + try: + return obj.read() + except IOError, e: + if e.errno == errno.EINTR: + continue + else: + raise + # # Generic tests # @@ -123,7 +134,7 @@ p = subprocess.Popen([sys.executable, "-c", 'import sys; sys.stdout.write("orange")'], stdout=subprocess.PIPE) - self.assertEqual(p.stdout.read(), "orange") + self.assertEqual(self.read_no_intr(p.stdout), "orange") def test_stdout_filedes(self): # stdout is set to open file descriptor @@ -151,7 +162,7 @@ p = subprocess.Popen([sys.executable, "-c", 'import sys; sys.stderr.write("strawberry")'], stderr=subprocess.PIPE) - self.assertEqual(remove_stderr_debug_decorations(p.stderr.read()), + self.assertEqual(remove_stderr_debug_decorations(self.read_no_intr(p.stderr)), "strawberry") def test_stderr_filedes(self): @@ -186,7 +197,7 @@ 'sys.stderr.write("orange")'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - output = p.stdout.read() + output = self.read_no_intr(p.stdout) stripped = remove_stderr_debug_decorations(output) self.assertEqual(stripped, "appleorange") @@ -220,7 +231,7 @@ stdout=subprocess.PIPE, cwd=tmpdir) normcase = os.path.normcase - self.assertEqual(normcase(p.stdout.read()), normcase(tmpdir)) + self.assertEqual(normcase(self.read_no_intr(p.stdout)), normcase(tmpdir)) def test_env(self): newenv = os.environ.copy() @@ -230,7 +241,7 @@ 'sys.stdout.write(os.getenv("FRUIT"))'], stdout=subprocess.PIPE, env=newenv) - self.assertEqual(p.stdout.read(), "orange") + self.assertEqual(self.read_no_intr(p.stdout), "orange") def test_communicate(self): p = subprocess.Popen([sys.executable, "-c", @@ -305,7 +316,8 @@ 'sys.stdout.write("\\nline6");'], stdout=subprocess.PIPE, universal_newlines=1) - stdout = p.stdout.read() + + stdout = self.read_no_intr(p.stdout) if hasattr(open, 'newlines'): # Interpreter with universal newline support self.assertEqual(stdout, @@ -343,7 +355,7 @@ def test_no_leaking(self): # Make sure we leak no resources - max_handles = 1026 # too much for most UNIX systems + max_handles = 10 # too much for most UNIX systems if mswindows: max_handles = 65 # a full test is too slow on Windows for i in range(max_handles): @@ -424,7 +436,7 @@ 'sys.stdout.write(os.getenv("FRUIT"))'], stdout=subprocess.PIPE, preexec_fn=lambda: os.putenv("FRUIT", "apple")) - self.assertEqual(p.stdout.read(), "apple") + self.assertEqual(self.read_no_intr(p.stdout), "apple") def test_args_string(self): # args is a string @@ -457,7 +469,7 @@ p = subprocess.Popen(["echo $FRUIT"], shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertEqual(p.stdout.read().strip(), "apple") + self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple") def test_shell_string(self): # Run command through the shell (string) @@ -466,7 +478,7 @@ p = subprocess.Popen("echo $FRUIT", shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertEqual(p.stdout.read().strip(), "apple") + self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple") def test_call_string(self): # call() function with string argument on UNIX @@ -525,7 +537,7 @@ p = subprocess.Popen(["set"], shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertNotEqual(p.stdout.read().find("physalis"), -1) + self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1) def test_shell_string(self): # Run command through the shell (string) @@ -534,7 +546,7 @@ p = subprocess.Popen("set", shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertNotEqual(p.stdout.read().find("physalis"), -1) + self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1) def test_call_string(self): # call() function with string argument on Windows /Peter Åstrand <astrand(a)lysator.liu.se>

10 years, 12 months

Re: [Python-Dev] [Python-checkins] r45510 - python/trunk/Lib/pkgutil.py python/trunk/Lib/pydoc.py

by M.-A. Lemburg

Phillip.eby wrote: > Author: phillip.eby > Date: Tue Apr 18 02:59:55 2006 > New Revision: 45510 > > Modified: > python/trunk/Lib/pkgutil.py > python/trunk/Lib/pydoc.py > Log: > Second phase of refactoring for runpy, pkgutil, pydoc, and setuptools > to share common PEP 302 support code, as described here: > > http://mail.python.org/pipermail/python-dev/2006-April/063724.html Shouldn't this new module be named "pkglib" to be in line with the naming scheme used for all the other utility modules, e.g. httplib, imaplib, poplib, etc. ? > pydoc now supports PEP 302 importers, by way of utility functions in > pkgutil, such as 'walk_packages()'. It will properly document > modules that are in zip files, and is backward compatible to Python > 2.3 (setuptools installs for Python <2.5 will bundle it so pydoc > doesn't break when used with eggs.) Are you saying that the installation of setuptools in Python 2.3 and 2.4 will then overwrite the standard pydoc included with those versions ? I think that's the wrong way to go if not made an explicit option in the installation process or a separate installation altogether. I bothered by the fact that installing setuptools actually changes the standard Python installation by either overriding stdlib modules or monkey-patching them at setuptools import time. > What has not changed is that pydoc command line options do not support > zip paths or other importer paths, and the webserver index does not > support sys.meta_path. Those are probably okay as limitations. > > Tasks remaining: write docs and Misc/NEWS for pkgutil/pydoc changes, > and update setuptools to use pkgutil wherever possible, then add it > to the stdlib. Add setuptools to the stdlib ? I'm still missing the PEP for this along with the needed discussion touching among other things, the change of the distutils standard "python setup.py install" to install an egg instead of a site package. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 18 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::

12 years, 3 months

Making builtins more efficient

by Steven Elliott

I'm interested in how builtins could be more efficient. I've read over some of the PEPs having to do with making global variables more efficient (search for "global"): http://www.python.org/doc/essays/pepparade.html But I think the problem can be simplified by focusing strictly on builtins. One of my assumptions is that only a small fractions of modules override the default builtins with something like: import mybuiltins __builtins__ = mybuiltins As you probably know each access of a builtin requires two hash table lookups. First, the builtin is not found in the list of globals. It is then found in the list of builtins. Why not have a means of referencing the default builtins with some sort of index the way the LOAD_FAST op code currently works? In other words, by default each module gets the default set of builtins indexed (where the index indexes into an array) in a certain order. The version stored in the pyc file would be bumped each time the set of default builtins is changed. I don't have very strong feelings whether things like True = (1 == 1) would be a syntax error, but assigning to a builtin could just do the equivalent of STORE_FAST. I also don't have very strong feelings about whether the array of default builtins would be shared between modules. To simulate the current behavior where attempting to assign to builtin actually alters that module's global hashtable a separate array of builtins could be used for each module. As to assigning to __builtins__ (like I mentioned at the beginning of this post) perhaps it could assign to the builtin array for those items that have a name that matches a default builtin (such as "True" or "len"). Those items that don't match a default builtin would just create global variables. Perhaps what I'm suggesting isn't feasible for reasons that have already been discussed. But it seems like it should be possible to make "while True" as efficient as "while 1". -- ----------------------------------------------------------------------- | Steven Elliott | selliott4(a)austin.rr.com | -----------------------------------------------------------------------

13 years, 2 months

GeneratorExit inheriting from Exception

by Nick Coghlan

Should GeneratorExit inherit from Exception or BaseException? Currently, a generator that catches Exception and continues on to yield another value can't be closed properly (you get a runtime error pointing out that the generator ignored GeneratorExit). The only decent reference I could find to it in the old PEP 348/352 discussions is Guido writing [1]: > when GeneratorExit or StopIteration > reach the outer level of an app, it's a bug like all the others that > bare 'except:' WANTS to catch. (at that point in the conversation, I believe bare except was considered the equivalent of "except Exception:") While I agree with what Guido says about GeneratorExit being a bug if it reaches the outer level of an app, it seems like a bit of a trap that a correctly written generator can't write "except Exception:" without preceding it with an "except GeneratorExit:" that reraises the exception. Isn't that exactly the idiom we're trying to get rid of for SystemExit and KeyboardInterrupt? Regards, Nick. [1] http://mail.python.org/pipermail/python-dev/2005-August/055173.html -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org

13 years, 4 months

Plea to distribute debugging lib

by David Abrahams

For years, Boost.Python has been doing some hacks to work around the fact that a Windows Python distro doesn't include the debug build of the library. http://www.boost.org/libs/python/doc/building.html#variants explains. We wanted to make it reasonably convenient for Windows developers (and our distributed testers) to work with a debug build of the Boost.Python library and of their own code. Having to download the Python source and build the debug DLL was deemed unacceptable. Well, those hacks have run out of road. VC++8 now detects that some of its headers have been #included with _DEBUG and some without, and it will refuse to build anything when it does. We have several new hacks to work around that detection, and I think we _might_ be able to get away with them for one more release. But it's really time to do it right. MS is recommending that we (Boost) start distributing a debug build of the Python DLL with Boost, but Boost really seems like the wrong place to host such a thing. Is there any way Python.org can make a debug build more accessible? Thanks, Dave -- Dave Abrahams Boost Consulting www.boost-consulting.com

13 years, 8 months

Dropping __init__.py requirement for subpackages

by Guido van Rossum

(Context: There's a large crowd with pitchforks and other sharp pointy farm implements just outside the door of my office at Google. They are making an unbelievable racket. It appears they are Google engineers who have been bitten by a misfeature of Python, and they won't let me go home before I have posted this message.) The requirement that a directlry must contain an __init__.py file before it is considered a valid package has always been controversial. It's designed to prevent the existence of a directory with a common name like "time" or "string" from preventing fundamental imports to work. But the feature often trips newbie Python programmers (of which there are a lot at Google, at our current growth rate we're probably training more new Python programmers than any other organization worldwide :-). One particular egregious problem is that *subpackage* are subject to the same rule. It so happens that there is essentially only one top-level package in the Google code base, and it already has an __init__.py file. But developers create new subpackages at a frightening rate, and forgetting to do "touch __init__.py" has caused many hours of lost work, not to mention injuries due to heads banging against walls. So I have a very simple proposal: keep the __init__.py requirement for top-level pacakages, but drop it for subpackages. This should be a small change. I'm hesitant to propose *anything* new for Python 2.5, so I'm proposing it for 2.6; if Neal and Anthony think this would be okay to add to 2.5, they can do so. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

14 years

FishEye on Python CVS Repository

by Peter Moore

Greetings Python Developers, I'm responsible for setting up free FishEye hosting for community projects. As a long time python user I of course added Python up front. You can see it here: http://fisheye.cenqua.com/viewrep/python/ If you aren't familiar with FishEye, it is a repository browsing, searching, analysis, monitoring..., tool for CVS (SVN coming soon). Cool features include: RSS feeds http://fisheye.cenqua.com/changelog/%7Erss/python/rss.xml Synthetic changesets http://fisheye.cenqua.com/changelog/python/ and Pretty ediffs http://fisheye.cenqua.com/viewrep/python/python/nondist/peps/pep-0343.txt?r… SQL like search http://fisheye.cenqua.com/search/python/?ql= Note that normally FishEye is pretty close to real time (i.e. when run on a local repository), but we can only get SF updates once a day, and they are potentially a bit old when we grab them. Feel free to mail me direct pete _at_ cenqua _dot_ com if you want to tweak the python instance or have any other questions/comments not appropriate for this list. Cheers, Pete. --- +61-414-587637 www.cenqua.com/fisheye

14 years

how about adding ping's uuid module to the standard lib ?

by Fredrik Lundh

see subject and http://python.org/sf/1368955 comments ? </F> (note sure how my previous attempt to post this ended up on comp.lang.python instead, but I'll blame it on gmane... ;-)

14 years

Adding wsgiref to stdlib

by Guido van Rossum

PEP 333 specifies WSGI, the Python Web Server Gateway Interface v1.0; it's written by Phillip Eby who put a lot of effort in it to make it acceptable to very diverse web frameworks. The PEP has been well received by web framework makers and users. As a supplement to the PEP, Phillip has written a reference implementation, "wsgiref". I don't know how many people have used wsgiref; I'm using it myself for an intranet webserver and am very happy with it. (I'm asking Phillip to post the URL for the current source; searching for it produces multiple repositories.) I believe that it would be a good idea to add wsgiref to the stdlib, after some minor cleanups such as removing the extra blank lines that Phillip puts in his code. Having standard library support will remove the last reason web framework developers might have to resist adopting WSGI, and the resulting standardization will help web framework users. Last time this was brought up there were feature requests and discussion on how "industrial strength" the webserver in wsgiref ought to be but nothing like the flamefest that setuptools caused (no comments please). I'm inviting people to discuss the addition of wsgiref to the standard library. I'd like the discussion to be finished before a3 goes out; technically it can go in up till the b1 code freeze, but I don't really want to push it that close. I'd like the focus of the discussion to be "what are the risks of adding wsgiref to the stdlib"; not "what could we think of that's even better". Achieving a perfect decision is not the goal; having general consensus that adding it would be better than not adding is would be good. Pointing out specific bugs in wsgiref and suggesting how they ought to be fixed is also welcome. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

14 years, 1 month

base64 module

by Sanghyeon Seo

Hello, base64 module documentation for b64decode function says, "TypeError is raised if s were incorrectly padded or if there are non-alphabet characters present in the string." But this doesn't seem to be the case. Testcase: import base64 base64.b64decode('%') Since % is a non-alphabet character, this should raise TypeError (btw, shouldn't this be ValueError instead?), but Python 2.4.3 silently ignores. I found this while experimenting with IronPython. IronPython 1.0 Beta 5 gives: Traceback (most recent call last): File base64, line unknown, in b64decode SystemError: cannot decode byte: '%' It's not TypeError, but it doesn't silently ignore either. Seo Sanghyeon

14 years, 1 month

Jump to page:

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev April 2006