Mailman 3 May 2008 - Python-Dev

A smarter shutil.copytree ?
by Tarek Ziadé May 21, 2008

May 21, 2008

Hi, shutil.copytree is very convenient to make recursive copies, but os.walk has to be used everytime some filtering has to be done on the files copied., if you want to avoid copying some files. The code pattern with os.walk is pretty talkative : --------------------- copying a source folder to a target folder, but the pyc/pyo files os.mkdir(target) for root, dirs, filenames in os.walk(source): root_target = root.replace(source, target) for dir_ in dirs: target_dir = join(root_target, dir_) if os.path.exists(target_dir): continue os.mkdir(target_dir) for filename in filenames: filename_source = join(root, filename) filename_target = join(root_target, filename) if (os.path.exists(filename_target) or os.path.splitext(filename) in ('.pyc', '.pyo')): continue shutil.copyfile(filename_source, filename_target) -------------------- If we could provide a callable argument to shutil.copytree, this would allow simplifying a lot the code: --------------------- copying a source to a target, but the pyc/pyo file def filtering(source, target): return os.path.splitext(filename) not in ('.pyc', '.pyo') shutil.copytree(source, target, filter_=filtering) --------------------- This is a very current pattern in my code, and I think this could be an interesting enhancement to shutil. So if people think it is a good idea, I can work on a patch and submit it to the tracker. Regards Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

6 13

Proposal: new environment variable PYTHONSTDOUTENCODING
by "Martin v. Löwis" May 21, 2008

May 21, 2008

I'd like to propose a new environment variable PYTHONSTDOUTENCODING. This is meant to solve various problems that people had with Python not detecting their terminal encoding correctly; it would override any detection that Python would use for determining the encoding of stdout (and stdin - but that's less relevant in 2.x). In particular, setting this environment variable would also disable the detection of whether stdout is a terminal. This is desirable for cases as the pydev eclipse plugin, where Python currently fails to detect that the output is a terminal (and technically, what Eclipse provides is not a terminal, but just a pipe, as you can't do pseudoterms in Java). This would have the additional effect that the encoding also gets in effect when redirecting stdout to a file. Whether or not this is a good thing might be debatable; giving the user the control over it (to set or clear that variable) is a good thing, IMO. Naming contest: it probably would be the longest of the PYTHON* variables. I would not want to call it PYTHONENCODING, or PYTHONSTDENCODING, though, because people might infer that it affects sys.getdefaultencoding(), which it shouldn't. Regards, Martin

7 16

disappearing exceptions
by Benjamin Peterson May 21, 2008

May 21, 2008

In several places in the C code, there are instances where all exceptions can be completely ignored (akin to a bare except statement.) after a PyObject_GetAttr call. A week ago, I fixed one of these in hasattr (issue 2196) by propagating exceptions that don't inherit Exception (SystemExit, KeyboardInterrupt). However, there is another patch that propagates anything that is not a AttributeError (issue 1574217). How should we approach this? -- Cheers, Benjamin Peterson "There's no place like 127.0.0.1."

4 7

Possible bug in re module?
by Dmitry Vasiliev May 20, 2008

May 20, 2008

I've just found a strange re behavior: >>> import re >>> re.sub("(?:ab|b|a)", "+", "cbacbabcabc") 'c++c++c+c' >>> re.sub("(?:ab|b|a){2}", "+", "cbacbabcabc") 'c+c+c+c' In the last case |-separated expressions seems don't tried from left to right. Is it bug or just me? -- Dmitry Vasiliev <dima at hlabs.spb.ru> http://hlabs.spb.ru

3 2

Re: [Python-Dev] Addition of "pyprocessing" module to standard lib.
by Leif Walsh May 20, 2008

May 20, 2008

On Mon, 19 May 2008, Bill Janssen wrote: > > On Mon, May 19, 2008 at 5:15 PM, Bill Janssen <janssen(a)parc.com> wrote: > > >> If you can run a pure Python module > > >> that does not depend on any C extension, then that platform has the > > >> support needed to run Python. > > > > > > This is certainly a point of view. One that many end-users wouldn't > > > understand :-). > > > > Perhaps, but it's clear-cut. Is OS X not properly supported because it > > can't run the _winreg module? I just don't like labeling a platform as > > unsupported just because ctypes doesn't compile on it. > > I assume _winreg runs everywhere Python is said to run, and there's a > Windows registry to examine, so I think it's a different kettle of > fish. ctypes doesn't run everywhere Python is said to run, and there > are dynamic libraries to call into. > > I think it would be great if we could get some AIX, HP-UX, and Solaris > boxes for Thomas to work on. What would motivate IBM, H-P, and Sun to > donate such gear? Perhaps each of the companies have an office > somewhere that could help with this resource allocation? For > instance, talking to the "AIX Collaboration Center" of IBM > (aixcc(a)us.ibm.com)? > > And these are all SYSV derivatives, aren't they? So perhaps it's some > common fix for all three? I just spent a semester in "Advanced Systems Programming in Unix/C", and all we did was port old software to solaris, hpux, irix, and friends (I don't think we used AIX, but I believe he has one running). Most of these were virtual machines, but they adequately represented the annoyances present in their respective architectures. I've cc'd my professor, hopefully he will let me know if there are any tips he has for setting up a VM. -- Cheers, Leif

1 0

Module renaming and pickle mechanisms
by M.-A. Lemburg May 20, 2008

May 20, 2008

I'd like to bring a potential problem to attention that is caused by the recent module renaming approach: Object serialization protocols like e.g. pickle usually store the complete module path to the object class together with the object. They access this module path by looking at the __module__ attribute of the object classes. With the renaming, all objects which use classes from the renamed modules will now refer to the renamed modules in their serialized form, e.g. queue.Queue instead of Queue.Queue (just to name one example). While this is nice for forward compatibility, it causes rather serious problems for making object serialization backwards compatible, since the older Python versions can no longer unserialize objects due to missing modules. This can happen in client-server setups where e.g. the server uses Python 2.6 and the clients some other Python version (e.g. Python 2.5). It can also happen in storage setups where Python objects are stored using e.g. pickle, ZODB being a prominent example. As soon as a Python 2.6 application starts writing to such storages, Python 2.5 and lower versions will no longer be able to read back all the data. Now, I think there's a way to solve this puzzle: Instead of renaming the modules (e.g. Queue -> queue), we leave the code in the existing modules and packages and instead add the new module names and package structure with pointers and redirects to the existing 2.5 modules. Code can (and probably should) still be changed to try to import the new module name. In cases where backwards compatibility is needed, this can also be done using try: import newname except ImportError: import oldname Later on, when porting applications to 3.0, the 2to3 script can then apply the final renaming in the source code. Example: queue.py: --------- import sys, Queue sys.modules[__name__] = Queue -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611

14 22

Unaccompanied Patch
by Graham Horler May 20, 2008

May 20, 2008

Hi all, I created a patch on 2007-08-15: http://bugs.python.org/issue1775025 I wonder, will it just stay in patches unnoticed until it has its very own associated bug report to keep it company? Thanks, Graham

4 4

Symbolic errno values in error messages
by Yannick Gingras May 19, 2008

May 19, 2008

Hi, I spent some time googleing for "OSError 4" before it occurred to me that "4" was actually an irrelevant implementation detail. As soon as I searched for "EINTR", I found exactly what I was looking for. (not really but this is another story) I jumped to the conclusion that OSError.__str__() should return the symbolic error number instead of the integer value. I was about to write a patch but I just noticed that OSError and friends are implemented in C so I guess it will be a bit harder than expected. The error message comes from EnvironmentError.__str__() which is implemented by EnvironmentError_str() in Objects/exceptions.c. A Python implementation could have replaced self.errno in the format string args by errno.errorcode[self.errno]. However, it's not clear to me if there is such a mapping available in C. Since my fix is not as trivial as I expected, I ask for advice regarding the following questions: 1) Should OSError.__str__() print the symbolic name of errno? 2) Where can I find the symbolic name in C? Best regards, -- Yannick Gingras

8 13

Re: [Python-Dev] Symbolic errno values in error messages
by Yannick Gingras May 19, 2008

May 19, 2008

"Guido van Rossum" <guido(a)python.org> writes: > Have you considered whether this works on all platforms? (E.g. > Windows, or embedded non-Unix-based.) Yes but I guess I didn't comment it properly. The line printed_errno = errno_str ? errno_str : self->myerrno; ensures that we gracefully fallback to numeric errno when here is no symbolic value available. The code is only invoked if there is an errno value, which takes care of most non-Unix platforms. -- Yannick Gingras

1 0

Conditional For Statements
by Ryan Hitchman May 19, 2008

May 19, 2008

I'd like to propose an addition to the for statement's syntax: for {variable} in {iterable} if {condition}: {block} which is equivalent to for {variable} in {iterable}: if not {condition}: continue {block} and for {variable} in filter(lambda: {condition}, iterable): {block} This would make the syntax closer to that of generators, which have 'for variable in iterable if condition', and would improve code clarity by increased brevity and not negating boolean expressions. Following are examples of current code with what the new code would be, taken from the Python 3.0a5 tarball. Demo/tkinter/guido/ss1.py:163: for (x, y), cell in self.cells.items(): if x <= 0 or y <= 0: continue for (x, y), cell in self.cells.items() if x > 0 and y > 0: Lib/encodings/__init__.py:91: for modname in modnames: if not modname or '.' in modname: continue for modname in modnames if modname and '.' not in modname: Lib/idlelib/AutoExpand.py:70: for w in wafter: if dict.get(w): continue for w in wafter if w not in dict: Lib/Cookie.py:483: for K,V in items: if V == "": continue if K not in attrs: continue for K,V in items if V != "" and K not in attrs: Lib/hashlib.py:108: for opensslFuncName in filter(lambda n: n.startswith('openssl_'), dir(_hashlib)): for opensslFuncName in dir(_hashlib) if opensslFuncName.startswith('openssl_'): There are many more examples of this in the standard library, and likely even more in production code. I am not familiar with LL(1) parsing, so this may impossible under that constraint.

8 8