Mailman 3 May 2013 - Python-Dev

2.7.5 baking
by Benjamin Peterson 15 May '13

15 May '13

The long anticipated "emergency" 2.7.5 release has now been tagged. It will be publicly announced as binaries arrive. Originally, I was just going to cherrypick regression fixes onto the 2.7.4 release and release those as 2.7.5. I started to this but ran into some conflicts. Since we don't have buildbot testing of release branches, I decided it would be best to just cut from the maintenance branch. -- Regards, Benjamin

7 9

How to debug python crashes
by Philippe Fremy 15 May '13

15 May '13

Hi, I have a reproducable crash on Windows XP with Python 2.7 which I would like to investigate. I have Visual Studio 2008 installed and I downloaded the pdb files. However I could not find any instructions on how to use them and was unsuccessful at getting anything out of it. I checked the developer guide but could not find anything on debugging crashes. On internet, this seems to be also an underdocumented topic. So, a few questions : - is there some documentation to help debugging crashes ? - are the pdb files released along python usable with Visual Studio and stock Python ? Or do you need a hand-compiled version ? cheers, Philippe

6 6

Pickling failure on Enums
by Ethan Furman 15 May '13

15 May '13

On 05/13/2013 11:32 AM, Guido van Rossum wrote: > > But now you enter a different phase of your project, or one of your collaborators does, or perhaps you've released your code on PyPI and one of your users does. So someone tries to pickle some class instance that happens to contain an unpicklable enum. That's not a great experience. Pickling and unpickling errors are often remarkably hard to debug. (Especially the latter, so I have privately admonished Ethan to ensure that if the getframe hack doesn't work, the pickle failure should happen at pickling time, not at unpickle time.) I can get pickle failure on members created using the functional syntax with no module set; I cannot get pickle failure on those same classes; I cannot get pickle failure on class syntax enums that inherit complex types (such as the NEI class in the tests). If anybody has any insight on how to make that work, I'm all ears. -- ~Ethan~

3 6

First post
by Carlos Nepomuceno 14 May '13

14 May '13

Hi guys! This is my first post on this list. I'd like have your opinion on how to safely implement WSGI on a production server. My benchmarks show no performance differences between our PHP and Python environments. I'm using mod_wsgi v3.4 with Apache 2.4. Is that ok or can it get faster? Thanks in advance. Regards, Carlos

3 2

Issue 11406: adding os.scandir(), a directory iterator returning stat-like info
by Ben Hoyt 14 May '13

14 May '13

A few of us were having a discussion at http://bugs.python.org/issue11406 about adding os.scandir(): a generator version of os.listdir() to make iterating over very large directories more memory efficient. This also reflects how the OS gives things to you -- it doesn't give you a big list, but you call a function to iterate and fetch the next entry. While I think that's a good idea, I'm not sure just that much is enough of an improvement to make adding the generator version worth it. But what would make this a killer feature is making os.scandir() generate tuples of (name, stat_like_info). The Windows directory iteration functions (FindFirstFile/FindNextFile) give you the full stat information for free, and the Linux and OS X functions (opendir/readdir) give you partial file information (d_type in the dirent struct, which is basically the st_mode part of a stat, whether it's a file, directory, link, etc). Having this available at the Python level would mean we can vastly speed up functions like os.walk() that otherwise need to make an os.stat() call for every file returned. In my benchmarks of such a generator on Windows, it speeds up os.walk() by 9-10x. On Linux/OS X, it's more like 1.5-3x. In my opinion, that kind of gain is huge, especially on Windows, but also on Linux/OS X. So the idea is to add this relatively low-level function that exposes the extra information the OS gives us for free, but which os.listdir() currently throws away. Then higher-level, platform-independent functions like os.walk() could use os.scandir() to get much better performance. People over at Issue 11406 think this is a good idea. HOWEVER, there's debate over what kind of object the second element in the tuple, "stat_like_info", should be. My strong vote is for it to be a stat_result-like object, but where the fields are None if they're unknown. There would be basically three scenarios: 1) stat_result with all fields set: this would happen on Windows, where you get as much info from FindFirst/FindNext as from an os.stat() 2) stat_result with just st_mode set, and all other fields None: this would be the usual case on Linux/OS X 3) stat_result with all fields None: this would happen on systems whose readdir()/dirent doesn't have d_type, or on Linux/OS X when d_type was DT_UNKNOWN Higher-level functions like os.walk() would then check the fields they needed are not None, and only call os.stat() if needed, for example: # Build lists of files and directories in path files = [] dirs = [] for name, st in os.scandir(path): if st.st_mode is None: st = os.stat(os.path.join(path, name)) if stat.S_ISDIR(st.st_mode): dirs.append(name) else: files.append(name) Not bad for a 2-10x performance boost, right? What do folks think? Cheers, Ben. P.S. A few non-essential further notes: 1) As a Windows guy, a nice-to-have addition to os.scandir() would be a keyword arg like win_wildcard which defaulted to '*.*', but power users can pass in to utilize the wildcard feature of FindFirst/FindNext on Windows. We have plenty of other low-level functions that expose OS-specific features in the OS module, so this would be no different. But then again, it's not nearly as important as exposing the stat info. 2) I've been dabbling with this concept for a while in my BetterWalk library: https://github.com/benhoyt/betterwalk Note that the benchmarks there are old, and I've made further improvements in my local copy. The ctypes version gives speed gains for os.walk() of 2-3x on Windows, but I've also got a C version, which is giving 9-10x speed gains. I haven't yet got a Linux/OS X version written in C. 3) See also the previous python-dev thread on BetterWalk: http://mail.python.org/pipermail/python-ideas/2012-November/017944.html

12 40

PEP 379 Python launcher for Windows - behaviour for #!/usr/bin/env python line is wrong
by Paul Moore 14 May '13

14 May '13

While reviewing the behaviour of Vinay's "distil" installer tool (see distutils-sig for details, but it's not relevant here) I have found what I think is a flaw in the behaviour of the py.exe launcher for Windows. To recap for people unfamiliar with the launcher, it emulates #! line interpretation on Windows, interpreting commonly used forms from Unix and launching the appropriate installed Python interpreter (finding its location from the registry, as python.exe may not be on PATH). The problem is with the interpretation of #!/usr/bin/env python. The launcher treats this the same as #!/usr/bin/python, launching the "default" Python. But that is *not* what the equivalent line does on Unix, where it launches the *currently active* Python (a crucial difference when there is an active virtualenv). The result is that a script written to run with the active Python works on Unix as expected, but can use an unexpected version of Python on Windows. This is particularly unpleasant when the program in question is an (un)installer like distil! I would propose that the behaviour of the launcher on Windows should be changed when it encounters specifically the hashbang line #!/usr/bin/env python. In that case, it should search PATH for a copy of python.exe, and if it finds one, use that. If there is no python.exe on PATH, it should fall back to the same version of Python as would have been used if the line were #!/usr/bin/python. This will mean that scripts written with #!/usr/bin/env python will behave the same on Unix and Windows in the presence of activated virtualenvs. Would people be happy with this change? If so I will open an issue on bugs.python.org. I can look at producing a patch, as well. Paul

4 8

weak refs in descriptors (http://bugs.python.org/issue17950)
by Kristján Valur Jónsson 13 May '13

13 May '13

Hello python-dev. I'm working on a patch to remove reference cycles from heap-allocated classes: http://bugs.python.org/issue17950 Part of the patch involves making sure that descriptors in the class dictionary don't contain strong references to the class itself. This is item 2) in the defect description. I have implemented this via weak references and hit no issues at all when running the test suite. But I'd like to ask the oracle if there is anything I may be overlooking with this approach? Any hidden problems we might encounter? K

5 4

Tightening up the specification for locals()
by Nick Coghlan 13 May '13

13 May '13

An exchange in one of the enum threads prompted me to write down something I've occasionally thought about regarding locals(): it is currently severely underspecified, and I'd like to make the current CPython behaviour part of the language/library specification. (We recently found a bug in the interaction between the __prepare__ method and lexical closures that was indirectly related to this underspecification) Specifically, rather than the current vague "post-modification of locals may not work", I would like to explicitly document the expected behaviour at module, class and function scope (as well as clearly documenting the connection between modules, classes and the single- and dual-namespace variants of exec() and eval()): * at module scope, as well as when using exec() or eval() with a single namespace, locals() must return the same thing as globals(), which must be the actual execution namespace. Subsequent execution may change the contents of the returned mapping, and changes to the returned mapping must change the execution environment. * at class scope, as well as when using exec() or eval() with separate global and local namespaces, locals() must return the specified local namespace (which may be supplied by the metaclass __prepare__ method in the case of classes). Subsequent execution may change the contents of the returned mapping, and changes to the returned mapping must change the execution environment. For classes, this mapping will not be used as the actual class namespace underlying the defined class (the class creation process will copy the contents to a fresh dictionary that is only accessible by going through the class machinery). * at function scope, locals() must return a *snapshot* of the current locals and free variables. Subsequent execution must not change the contents of the returned mapping and changes to the returned mapping must not change the execution environment. Rather than adding this low level detail to the library reference docs, I would suggest adding it to the data model section of the language reference, with a link to the appropriate section from the docs for the locals() builtin. The warning in the locals() docs would be softened to indicate that modifications won't work at function scope, but are supported at module and class scope. Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

7 9

make a Windows installation package (.msi) for Python 3.3
by Jianfeng Mao 10 May '13

10 May '13

To Python Windows Release Managers: My name is Jianfeng Mao and I am a software developer at the U2 group in Rocket Software (http://u2.rocketsoftware.com/). I am currently working on a project to embed a slightly customized Python interpreter in our product. For easy installation and setup, we hope to be able to do the standard Python installation during the installation of our software. Basically I want to create a .msi file that can be called to install the full Python if the user needs this new feature. Brian Curtin (brian(a)python.org<mailto:brian@python.org>) pointed me to Tools/msi/msi.py for the Windows MSI builder. I tried to follow the instructions in the README but couldn't make it to work after a few twists and turns. Brian mentioned that few people needs to do this and only release managers handle the packaging of Python. I have listed the steps I have done in my attempt to create the .msi file. Please let me know if I have missed anything or done anything wrong. 1. hg clone http://hg.python.org/cpython 2. cd cpython 3. hg update 3.3 4. cd tools\buildbot, edit build.bat to change the configuration from Debug to Releaes; edit external.bat, change DEBUG=1 to DEBUG=0 5. go back to cpython\ and run tools\buildbot\build.bat 6. cd PC, then do 'nmake -f icons.mak' 7. cd ..\tools\msi 8. c:\python27\python msi.py WARNING: nm did not run successfully - libpythonXX.a not built cl /O2 /D WIN32 /D NDEBUG /D _WINDOWS /MT /W3 /c msisupport.c Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 16.00.40219.01 for 80x86 Copyright (C) Microsoft Corporation. All rights reserved. msisupport.c link.exe /OUT:msisupport.dll /INCREMENTAL:NO /NOLOGO /DLL /SUBSYSTEM:WIN DOWS /OPT:REF /OPT:ICF msisupport.obj msi.lib kernel32.lib Creating library msisupport.lib and object msisupport.exp Traceback (most recent call last): File "msi.py", line 1336, in <module> add_files(db) File "msi.py", line 961, in add_files generate_license() File "msi.py", line 914, in generate_license raise ValueError, "Could not find "+srcdir+"/../"+pat ValueError: Could not find C:\temp\cpython/../tcl8*

2 1

PEP 0 maintenance - deferring some currently open PEPs
by Nick Coghlan 10 May '13

10 May '13

I'd like to mark a few PEPs that are not currently being actively considered for 3.4 as Deferred: S 286 Enhanced Argument Tuples von Löwis S 337 Logging Usage in the Standard Library Dubner S 368 Standard image protocol and class Mastrodomenico I 396 Module Version Numbers Warsaw S 400 Deprecate codecs.StreamReader and codecs.StreamWriter Stinner S 419 Protecting cleanup statements from interruptions Colomiets I 423 Naming conventions and recipes related to packaging Bryon I 444 Python Web3 Interface McDonough, Ronacher S 3124 Overloading, Generic Functions, Interfaces, and ... Eby S 3142 Add a "while" clause to generator expressions Britton S 3143 Standard daemon process library Finney S 3145 Asynchronous I/O For subprocess.Popen Pruitt, McCreary, Carlson S 3152 Cofunctions Ewing Obviously, they can be reactivated at any time, but I think it would be beneficial to have the "Open" list more accurately reflect proposals that are currently being championed. I'd also like to mark this one as rejected by Guido at PyCon US 2013 in favour of an updated PEP 436: S 437 A DSL for specifying signatures, annotations and ... Krah Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

7 10