Python-Dev June 2005

python-dev@python.org

111 participants
115 discussions

by Peter Astrand

On Wed, 10 Nov 2004, John P Speno wrote: Hi, sorry for the delayed response. > While using subprocess (aka popen5), I came across one potential gotcha. I've had > exceptions ending like this: > > File "test.py", line 5, in test > cmd = popen5.Popen(args, stdout=PIPE) > File "popen5.py", line 577, in __init__ > data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB > OSError: [Errno 4] Interrupted system call > > (on Solaris 9) > > Would it make sense for subprocess to use a more robust read() function > which can handle these cases, i.e. when the parent's read on the pipe > to the child's stderr is interrupted by a system call, and returns EINTR? > I imagine it could catch EINTR and EAGAIN and retry the failed read(). I assume you are using signals in your application? The os.read above is not the only system call that can fail with EINTR. subprocess.py is full of other system calls that can fail, and I suspect that many other Python modules are as well. I've made a patch (attached) to subprocess.py (and test_subprocess.py) that should guard against EINTR, but I haven't committed it yet. It's quite large. Are Python modules supposed to handle EINTR? Why not let the C code handle this? Or, perhaps the signal module should provide a sigaction function, so that users can use SA_RESTART. Index: subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/subprocess.py,v retrieving revision 1.8 diff -u -r1.8 subprocess.py --- subprocess.py 7 Nov 2004 14:30:34 -0000 1.8 +++ subprocess.py 17 Nov 2004 19:42:30 -0000 @@ -888,6 +888,50 @@ pass + def _read_no_intr(self, fd, buffersize): + """Like os.read, but retries on EINTR""" + while True: + try: + return os.read(fd, buffersize) + except OSError, e: + if e.errno == errno.EINTR: + continue + else: + raise + + + def _read_all(self, fd, buffersize): + """Like os.read, but retries on EINTR, and reads until EOF""" + all = "" + while True: + data = self._read_no_intr(fd, buffersize) + all += data + if data == "": + return all + + + def _write_no_intr(self, fd, s): + """Like os.write, but retries on EINTR""" + while True: + try: + return os.write(fd, s) + except OSError, e: + if e.errno == errno.EINTR: + continue + else: + raise + + def _waitpid_no_intr(self, pid, options): + """Like os.waitpid, but retries on EINTR""" + while True: + try: + return os.waitpid(pid, options) + except OSError, e: + if e.errno == errno.EINTR: + continue + else: + raise + def _execute_child(self, args, executable, preexec_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell, @@ -963,7 +1007,7 @@ exc_value, tb) exc_value.child_traceback = ''.join(exc_lines) - os.write(errpipe_write, pickle.dumps(exc_value)) + self._write_no_intr(errpipe_write, pickle.dumps(exc_value)) # This exitcode won't be reported to applications, so it # really doesn't matter what we return. @@ -979,7 +1023,7 @@ os.close(errwrite) # Wait for exec to fail or succeed; possibly raising exception - data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB + data = self._read_all(errpipe_read, 1048576) # Exceptions limited to 1 MB os.close(errpipe_read) if data != "": child_exception = pickle.loads(data) @@ -1003,7 +1047,7 @@ attribute.""" if self.returncode == None: try: - pid, sts = os.waitpid(self.pid, os.WNOHANG) + pid, sts = self._waitpid_no_intr(self.pid, os.WNOHANG) if pid == self.pid: self._handle_exitstatus(sts) except os.error: @@ -1015,7 +1059,7 @@ """Wait for child process to terminate. Returns returncode attribute.""" if self.returncode == None: - pid, sts = os.waitpid(self.pid, 0) + pid, sts = self._waitpid_no_intr(self.pid, 0) self._handle_exitstatus(sts) return self.returncode @@ -1049,27 +1093,33 @@ stderr = [] while read_set or write_set: - rlist, wlist, xlist = select.select(read_set, write_set, []) + try: + rlist, wlist, xlist = select.select(read_set, write_set, []) + except select.error, e: + if e[0] == errno.EINTR: + continue + else: + raise if self.stdin in wlist: # When select has indicated that the file is writable, # we can write up to PIPE_BUF bytes without risk # blocking. POSIX defines PIPE_BUF >= 512 - bytes_written = os.write(self.stdin.fileno(), input[:512]) + bytes_written = self._write_no_intr(self.stdin.fileno(), input[:512]) input = input[bytes_written:] if not input: self.stdin.close() write_set.remove(self.stdin) if self.stdout in rlist: - data = os.read(self.stdout.fileno(), 1024) + data = self._read_no_intr(self.stdout.fileno(), 1024) if data == "": self.stdout.close() read_set.remove(self.stdout) stdout.append(data) if self.stderr in rlist: - data = os.read(self.stderr.fileno(), 1024) + data = self._read_no_intr(self.stderr.fileno(), 1024) if data == "": self.stderr.close() read_set.remove(self.stderr) Index: test/test_subprocess.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_subprocess.py,v retrieving revision 1.14 diff -u -r1.14 test_subprocess.py --- test/test_subprocess.py 12 Nov 2004 15:51:48 -0000 1.14 +++ test/test_subprocess.py 17 Nov 2004 19:42:30 -0000 @@ -7,6 +7,7 @@ import tempfile import time import re +import errno mswindows = (sys.platform == "win32") @@ -35,6 +36,16 @@ fname = tempfile.mktemp() return os.open(fname, os.O_RDWR|os.O_CREAT), fname + def read_no_intr(self, obj): + while True: + try: + return obj.read() + except IOError, e: + if e.errno == errno.EINTR: + continue + else: + raise + # # Generic tests # @@ -123,7 +134,7 @@ p = subprocess.Popen([sys.executable, "-c", 'import sys; sys.stdout.write("orange")'], stdout=subprocess.PIPE) - self.assertEqual(p.stdout.read(), "orange") + self.assertEqual(self.read_no_intr(p.stdout), "orange") def test_stdout_filedes(self): # stdout is set to open file descriptor @@ -151,7 +162,7 @@ p = subprocess.Popen([sys.executable, "-c", 'import sys; sys.stderr.write("strawberry")'], stderr=subprocess.PIPE) - self.assertEqual(remove_stderr_debug_decorations(p.stderr.read()), + self.assertEqual(remove_stderr_debug_decorations(self.read_no_intr(p.stderr)), "strawberry") def test_stderr_filedes(self): @@ -186,7 +197,7 @@ 'sys.stderr.write("orange")'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT) - output = p.stdout.read() + output = self.read_no_intr(p.stdout) stripped = remove_stderr_debug_decorations(output) self.assertEqual(stripped, "appleorange") @@ -220,7 +231,7 @@ stdout=subprocess.PIPE, cwd=tmpdir) normcase = os.path.normcase - self.assertEqual(normcase(p.stdout.read()), normcase(tmpdir)) + self.assertEqual(normcase(self.read_no_intr(p.stdout)), normcase(tmpdir)) def test_env(self): newenv = os.environ.copy() @@ -230,7 +241,7 @@ 'sys.stdout.write(os.getenv("FRUIT"))'], stdout=subprocess.PIPE, env=newenv) - self.assertEqual(p.stdout.read(), "orange") + self.assertEqual(self.read_no_intr(p.stdout), "orange") def test_communicate(self): p = subprocess.Popen([sys.executable, "-c", @@ -305,7 +316,8 @@ 'sys.stdout.write("\\nline6");'], stdout=subprocess.PIPE, universal_newlines=1) - stdout = p.stdout.read() + + stdout = self.read_no_intr(p.stdout) if hasattr(open, 'newlines'): # Interpreter with universal newline support self.assertEqual(stdout, @@ -343,7 +355,7 @@ def test_no_leaking(self): # Make sure we leak no resources - max_handles = 1026 # too much for most UNIX systems + max_handles = 10 # too much for most UNIX systems if mswindows: max_handles = 65 # a full test is too slow on Windows for i in range(max_handles): @@ -424,7 +436,7 @@ 'sys.stdout.write(os.getenv("FRUIT"))'], stdout=subprocess.PIPE, preexec_fn=lambda: os.putenv("FRUIT", "apple")) - self.assertEqual(p.stdout.read(), "apple") + self.assertEqual(self.read_no_intr(p.stdout), "apple") def test_args_string(self): # args is a string @@ -457,7 +469,7 @@ p = subprocess.Popen(["echo $FRUIT"], shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertEqual(p.stdout.read().strip(), "apple") + self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple") def test_shell_string(self): # Run command through the shell (string) @@ -466,7 +478,7 @@ p = subprocess.Popen("echo $FRUIT", shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertEqual(p.stdout.read().strip(), "apple") + self.assertEqual(self.read_no_intr(p.stdout).strip(), "apple") def test_call_string(self): # call() function with string argument on UNIX @@ -525,7 +537,7 @@ p = subprocess.Popen(["set"], shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertNotEqual(p.stdout.read().find("physalis"), -1) + self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1) def test_shell_string(self): # Run command through the shell (string) @@ -534,7 +546,7 @@ p = subprocess.Popen("set", shell=1, stdout=subprocess.PIPE, env=newenv) - self.assertNotEqual(p.stdout.read().find("physalis"), -1) + self.assertNotEqual(self.read_no_intr(p.stdout).find("physalis"), -1) def test_call_string(self): # call() function with string argument on Windows /Peter Åstrand <astrand(a)lysator.liu.se>

10 years, 9 months

FishEye on Python CVS Repository

by Peter Moore

Greetings Python Developers, I'm responsible for setting up free FishEye hosting for community projects. As a long time python user I of course added Python up front. You can see it here: http://fisheye.cenqua.com/viewrep/python/ If you aren't familiar with FishEye, it is a repository browsing, searching, analysis, monitoring..., tool for CVS (SVN coming soon). Cool features include: RSS feeds http://fisheye.cenqua.com/changelog/%7Erss/python/rss.xml Synthetic changesets http://fisheye.cenqua.com/changelog/python/ and Pretty ediffs http://fisheye.cenqua.com/viewrep/python/python/nondist/peps/pep-0343.txt?r… SQL like search http://fisheye.cenqua.com/search/python/?ql= Note that normally FishEye is pretty close to real time (i.e. when run on a local repository), but we can only get SF updates once a day, and they are potentially a bit old when we grab them. Feel free to mail me direct pete _at_ cenqua _dot_ com if you want to tweak the python instance or have any other questions/comments not appropriate for this list. Cheers, Pete. --- +61-414-587637 www.cenqua.com/fisheye

13 years, 9 months

problem installing current cvs

by Irmen de Jong

Hi, I'm having 2 problems with the current cvs : During compilation this warning occurs: *** WARNING: renaming "dbm" since importing it failed: build/lib.linux-i686-2.5/ dbm.so: undefined symbol: dbm_firstkey and the 'dbm' module is unavailable. I'm running MandrakeLinux 2005 (10.2) gcc 3.4.3 (I'm also having this problem when compiling python 2.3.5 or 2.4.1) furthermore the 'make install' of current cvs fails halfway trough with the following errors: ..... ..... Compiling /opt/python25/lib/python2.5/bsddb/test/test_associate.py ... Sorry: TabError: ('inconsistent use of tabs and spaces in indentation', ('/opt/python25/lib/python2.5/bsddb/test/test_associate.py', 97, 23, '\t os.mkdir(homeDir)\n')) Compiling /opt/python25/lib/python2.5/bsddb/test/test_basics.py ... Sorry: TabError: ('inconsistent use of tabs and spaces in indentation', ('/opt/python25/lib/python2.5/bsddb/test/test_basics.py', 400, 26, '\t if get_raises_error:\n')) Compiling /opt/python25/lib/python2.5/bsddb/test/test_compare.py ... Sorry: TabError: ('inconsistent use of tabs and spaces in indentation', ('/opt/python25/lib/python2.5/bsddb/test/test_compare.py', 167, 5, '\t"""\n')) ..... ..... Compiling /opt/python25/lib/python2.5/bsddb/test/test_recno.py ... Sorry: TabError: ('inconsistent use of tabs and spaces in indentation', ('/opt/python25/lib/python2.5/bsddb/test/test_recno.py', 38, 46, '\tget_returns_none = d.set_get_returns_none(2)\n')) ..... ..... make: *** [libinstall] Error 1 $ And then it quits. Fixing the tab indentation errors locally makes the problem go away. Regards, Irmen de Jong

13 years, 11 months

Terminology for PEP 343

by Raymond Hettinger

With 343 accepted, we can now add __enter__() and __exit__() methods to objects. What term should describe those objects in the documentation? For instance, if the new magic methods are added to decimal.Context(), do we then describe Context objects as "withable" ;-) The PEP itself doesn't provide much of a clue. The terms "template", "wrapper", and "block" are over-broad and do not provide a metaphor for setup/teardown, r entry/exit, or initialization/finalization. Whatever term is selected, it should be well thought-out and added to the glossary. The choice of words will likely have a great effect on learnability and on how people think about the new tool. Raymond

14 years, 7 months

Some RFE for review

by Reinhold Birkenfeld

Hi, while bugs and patches are sometimes tricky to close, RFE can be very easy to decide whether to implement in the first place. So what about working a bit on this front? Here are several RFE reviewed, perhaps some can be closed ("should" is always from submitter's point of view): 1193128: str.translate(None, delchars) should be allowed and only delete delchars from the string. Review: may be a good feature, although string._idmap can be passed as the first parameter too. 1226256: The "path" module by Jason Orendorff should be in the standard library. http://www.jorendorff.com/articles/python/path/ Review: the module is great and seems to have a large user base. On c.l.py there are frequent praises about it. 1216944: urllib(2) should gain a dict mapping HTTP status codes to the correspondig status/error text. Review: I can't see anything speaking against it. 1214675: warnings should get a removefilter() method. An alternative would be to fully document the "filters" attribute to allow direct tinkering with it. Review: As mwh said in a comment, this isn't Java, so the latter may be the way to go. 1205239: Shift operands should be allowed to be negative integers, so e.g. a << -2 is the same as a >> 2. Review: Allowing this would open a source of bugs previously well identifiable. 1152248: In order to read "records" separated by something other than newline, file objects should either support an additional parameter (the separator) to (x)readlines(), or gain an additional method which does this. Review: The former is a no-go, I think, because what is read won't be lines. The latter is further complicating the file interface, so I would follow the principle that not every 3-line function should be builtin. 1110010: A function "attrmap" should be introduced which is used as follows: attrmap(x)['att'] == getattr(x, 'att') The submitter mentions the use case of new-style classes without a __dict__ used at the right of %-style string interpolation. Review: I don't know whether this is worth it. 1052098: An environment variable should be supported to set the default encoding. Review: If one wants this for a single application, he can still implement it himself. 985094: getattr should support a callable as the second argument, used as follows: getattr(obj, func) == func(obj) Review: Highly unintuitive to me. That's all for today; sorry if it was too much ;) Reinhold -- Mail address is perfectly valid!

14 years, 8 months

Is PEP 237 final -- Unifying Long Integers and Integers

by Raymond Hettinger

IIRC, there was a decision to not implement phase C and to keep the trailing L in representations of long integers. If so, I believe the PEP can be marked as final. We've done all we're going to do. Raymond

14 years, 8 months

Inconsistent API for sets.Set and build-in set

by Barry Warsaw

I've been looking at the API for sets.Set and built-in set objects in Python 2.4.1 and I think I have found some minor inconsistencies. Background: we have an object that is very similar to "sets" and we originally modeled the API after sets.Set since we started with Python 2.3. Now I'm trying to update the API so that it's consistent with Python 2.4's built-in set object and I've noticed the following discrepancies. set.Set has both an .update() method and a .union_update() method. They appear to be completely identical, accepting either a Set object or an arbitrary sequence. This is the case despite the docstring difference for these two methods and the fact that Set.update() isn't documented on the texinfo page. Built-in set has only .update() though but it acts just like the set.Set methods above. Note that of all these methods, only Set.update() is documented to take an iterable argument. These inconsistencies could prove a bit problematic when porting Py2.3 applications using sets.Set to Py2.4 using built-in set. I'd like to fix this for Python 2.4.2, and I think the changes are pretty minimal. If there are no objections, I propose to do the following (only in Python 2.4 and 2.5): * Add set.union_update() as an alias for set.update(). * Add to docstrings for all methods that 't' can be any iterable. * Update texinfo documentation to add Set.update() and set.union_update() and explain that all can take iterables I consider this a bug in 2.4, not a new feature, because without it, it makes more work in porting applications. Thoughts? I'm willing to Just Fix It, but if someone wants to see a patch first, I'll be happy to generate it and post it to SF. -Barry

14 years, 9 months

Possible C API problem?

by Gary Robinson

Hello, I was asking about a problem I was having over on the C++-python list, and they suggested I report it here as a possible Python problem. I was getting bus errors with a C module I was linking to, so factored it down too a very small example that reproduced the problem. Here it is: #include <Python.h> static double gfSumChiSquare = 123.0; static PyObject * getSumChiSquare(PyObject *self, PyObject *args){ return Py_BuildValue("d", gfSumChiSquare); } static PyMethodDef SimMethods[] = { {"getSumChiSquare", getSumChiSquare, METH_NOARGS, "Return fSumChiSquare"}, {NULL, NULL, 0, NULL} /* Sentinel */ }; PyMODINIT_FUNC inittestfloat(void) { (void) Py_InitModule("testfloat", SimMethods); } That caused a bus error 100% of the time when I simply imported the module into Python and called getSumChiSquare(), i.e.: >>> import testfloat >>> testfloat.getSumChiSquare() However, the problem seems to go away if I use METH_VARARGS, and parse the non-existent args with PyArg_ParseTuple: #include <Python.h> static double gfSumChiSquare = 123.0; static PyObject * getSumChiSquare(PyObject *self, PyObject *args){ if (!PyArg_ParseTuple(args, "", NULL)) return NULL; return Py_BuildValue("d", gfSumChiSquare); } static PyMethodDef SimMethods[] = { {"getSumChiSquare", getSumChiSquare, METH_VARARGS, "Return fSumChiSquare"}, {NULL, NULL, 0, NULL} /* Sentinel */ }; PyMODINIT_FUNC inittestfloat(void) { (void) Py_InitModule("testfloat", SimMethods); } This approach seems to work reliably -- at least variations I've tried haven't caused a bus error. But I haven't been able to discern an explanation from the docs as to why this would be better. The docs say that both METH_VARARGS and METH_NOARGS expect a PyCFunction. So if I am calling the function with no arguments, why can't I use METH_NOARGS and skip the call to PyArg_ParseTuple? Could it be that this is a python bug? Or am I doing something wrong? Note: this is using Python 2.3 on OS X: Python 2.3 (#1, Sep 13 2003, 00:49:11) Thanks in advance for any help or insight you can give, Gary -- Gary Robinson CTO Emergent Music, LLC grobinson(a)goombah.com 207-942-3463 Company: http://www.goombah.com Blog: http://www.garyrobinson.net

14 years, 9 months

using pyhon from the MSYS shell

by lode leroy

I was trying to compile a python plugin (for gimp) using the MSYS shell and the MINGW compiler. I now have put this in 'sitecustomize' $ cat /c/Python24/lib/site-packages/sitecustomize.py import sys import os import re if os.environ.get("MSYSTEM") == "MINGW32": os.sep='/' os.pathsep='/' sys.prefix = re.sub('\\\\','/',sys.prefix) sys.exec_prefix = re.sub('\\\\','/',sys.exec_prefix) It would probably be better to have python detect that it's running from inside the msys shell, and output '/'es instead of '\'es. maybe someone could extend os.path to do this in the standard distribution: implement an msyspath.py, which calls ntpath for each function, and does a replace at the end of the evaluation. Unfortunately, I'm just starting to look into python, so I do not know what the cleanest implementation of this would be...

14 years, 9 months

getch() in msvcrt does not accept extended characters.

by Darryl Dixon

Hi, I'm sorry, I don't seem to have done a very good job at explaining the situation. I'll try again: 'getch()' is a low-level function provided on Windows to capture a single character of input from a user, /without echoing it to the screen/. As far as I can tell there's no other way of doing this with Python on Windows. The Python interface to this function is in the C code in msvcrtmodule.c, which has a (very thin) wrapper around the raw OS system call. Microsoft provide a way of accepting both normal ASCII codes, and extended characters via this system call. Currently, the Python wrapper in msvcrtmodule.c only supports the semantics of getting the bare ASCII codes, and not the extended characters. I would strongly suggest that it should support both. So, I guess in answer to the questions raised below; 1) I can't do it in Python code without getch() (hence the original email) 2) I would argue that in fact getch() is 'broken' in its current Python implementation, as it doesn't support what the OS implies /should/ be supported (and, indeed, if I input an extended character in response to a 'getch()' call, all I get back currently is an empty string). Hope this helps, D Fredrik wrote: >Darryl Dixon wrote: > >> Microsoft support capturing extended characters via _getch, but it requires making a >> second call to getch() if one of the 'magic' returns is encountered in the first call (0x00 >> or 0xE0). > >so why not do that in your python code? > >> The relevant chunk of code in Python that would probably need to be >> changed to support this appears to be in msvcrtmodule.c: > >if you change msvcrt, you'll break all the programs that uses getch() in >the prescribed way... > ></F> -- Darryl Dixon <esrever_otua(a)pythonhacker.is-a-geek.net>

14 years, 9 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev June 2005