Mailman 3 May 2013 - Python-Dev

2.7.5 baking
by Benjamin Peterson May 15, 2013

May 15, 2013

The long anticipated "emergency" 2.7.5 release has now been tagged. It will be publicly announced as binaries arrive. Originally, I was just going to cherrypick regression fixes onto the 2.7.4 release and release those as 2.7.5. I started to this but ran into some conflicts. Since we don't have buildbot testing of release branches, I decided it would be best to just cut from the maintenance branch. -- Regards, Benjamin

7 9

How to debug python crashes
by Philippe Fremy May 15, 2013

May 15, 2013

Hi, I have a reproducable crash on Windows XP with Python 2.7 which I would like to investigate. I have Visual Studio 2008 installed and I downloaded the pdb files. However I could not find any instructions on how to use them and was unsuccessful at getting anything out of it. I checked the developer guide but could not find anything on debugging crashes. On internet, this seems to be also an underdocumented topic. So, a few questions : - is there some documentation to help debugging crashes … [View More]

6 6

Pickling failure on Enums
by Ethan Furman May 14, 2013

May 14, 2013

On 05/13/2013 11:32 AM, Guido van Rossum wrote: > > But now you enter a different phase of your project, or one of your collaborators does, or perhaps you've released your code on PyPI and one of your users does. So someone tries to pickle some class instance that happens to contain an unpicklable enum. That's not a great experience. Pickling and unpickling errors are often remarkably hard to debug. (Especially the latter, so I have privately admonished Ethan to ensure that if the … [View More]

3 6

First post
by Carlos Nepomuceno May 14, 2013

May 14, 2013

Hi guys! This is my first post on this list. I'd like have your opinion on how to safely implement WSGI on a production server. My benchmarks show no performance differences between our PHP and Python environments. I'm using mod_wsgi v3.4 with Apache 2.4. Is that ok or can it get faster? Thanks in advance. Regards, Carlos

3 2

Issue 11406: adding os.scandir(), a directory iterator returning stat-like info
by Ben Hoyt May 14, 2013

May 14, 2013

A few of us were having a discussion at http://bugs.python.org/issue11406 about adding os.scandir(): a generator version of os.listdir() to make iterating over very large directories more memory efficient. This also reflects how the OS gives things to you -- it doesn't give you a big list, but you call a function to iterate and fetch the next entry. While I think that's a good idea, I'm not sure just that much is enough of an improvement to make adding the generator version worth it. But what … [View More]would make this a killer feature is making os.scandir() generate tuples of (name, stat_like_info). The Windows directory iteration functions (FindFirstFile/FindNextFile) give you the full stat information for free, and the Linux and OS X functions (opendir/readdir) give you partial file information (d_type in the dirent struct, which is basically the st_mode part of a stat, whether it's a file, directory, link, etc). Having this available at the Python level would mean we can vastly speed up functions like os.walk() that otherwise need to make an os.stat() call for every file returned. In my benchmarks of such a generator on Windows, it speeds up os.walk() by 9-10x. On Linux/OS X, it's more like 1.5-3x. In my opinion, that kind of gain is huge, especially on Windows, but also on Linux/OS X. So the idea is to add this relatively low-level function that exposes the extra information the OS gives us for free, but which os.listdir() currently throws away. Then higher-level, platform-independent functions like os.walk() could use os.scandir() to get much better performance. People over at Issue 11406 think this is a good idea. HOWEVER, there's debate over what kind of object the second element in the tuple, "stat_like_info", should be. My strong vote is for it to be a stat_result-like object, but where the fields are None if they're unknown. There would be basically three scenarios: 1) stat_result with all fields set: this would happen on Windows, where you get as much info from FindFirst/FindNext as from an os.stat() 2) stat_result with just st_mode set, and all other fields None: this would be the usual case on Linux/OS X 3) stat_result with all fields None: this would happen on systems whose readdir()/dirent doesn't have d_type, or on Linux/OS X when d_type was DT_UNKNOWN Higher-level functions like os.walk() would then check the fields they needed are not None, and only call os.stat() if needed, for example: # Build lists of files and directories in path files = [] dirs = [] for name, st in os.scandir(path): if st.st_mode is None: st = os.stat(os.path.join(path, name)) if stat.S_ISDIR(st.st_mode): dirs.append(name) else: files.append(name) Not bad for a 2-10x performance boost, right? What do folks think? Cheers, Ben. P.S. A few non-essential further notes: 1) As a Windows guy, a nice-to-have addition to os.scandir() would be a keyword arg like win_wildcard which defaulted to '*.*', but power users can pass in to utilize the wildcard feature of FindFirst/FindNext on Windows. We have plenty of other low-level functions that expose OS-specific features in the OS module, so this would be no different. But then again, it's not nearly as important as exposing the stat info. 2) I've been dabbling with this concept for a while in my BetterWalk library: https://github.com/benhoyt/betterwalk Note that the benchmarks there are old, and I've made further improvements in my local copy. The ctypes version gives speed gains for os.walk() of 2-3x on Windows, but I've also got a C version, which is giving 9-10x speed gains. I haven't yet got a Linux/OS X version written in C. 3) See also the previous python-dev thread on BetterWalk: http://mail.python.org/pipermail/python-ideas/2012-November/017944.html [View Less]

12 40

PEP 379 Python launcher for Windows - behaviour for #!/usr/bin/env python line is wrong
by Paul Moore May 14, 2013

May 14, 2013

While reviewing the behaviour of Vinay's "distil" installer tool (see distutils-sig for details, but it's not relevant here) I have found what I think is a flaw in the behaviour of the py.exe launcher for Windows. To recap for people unfamiliar with the launcher, it emulates #! line interpretation on Windows, interpreting commonly used forms from Unix and launching the appropriate installed Python interpreter (finding its location from the registry, as python.exe may not be on PATH). The … [View More]

4 8

weak refs in descriptors (http://bugs.python.org/issue17950)
by Kristján Valur Jónsson May 13, 2013

May 13, 2013

Hello python-dev. I'm working on a patch to remove reference cycles from heap-allocated classes: http://bugs.python.org/issue17950 Part of the patch involves making sure that descriptors in the class dictionary don't contain strong references to the class itself. This is item 2) in the defect description. I have implemented this via weak references and hit no issues at all when running the test suite. But I'd like to ask the oracle if there is anything I may be overlooking with this approach? … [View More]

5 4

Tightening up the specification for locals()
by Nick Coghlan May 13, 2013

May 13, 2013

An exchange in one of the enum threads prompted me to write down something I've occasionally thought about regarding locals(): it is currently severely underspecified, and I'd like to make the current CPython behaviour part of the language/library specification. (We recently found a bug in the interaction between the __prepare__ method and lexical closures that was indirectly related to this underspecification) Specifically, rather than the current vague "post-modification of locals may not … [View More]

7 9

make a Windows installation package (.msi) for Python 3.3
by Jianfeng Mao May 10, 2013

May 10, 2013

To Python Windows Release Managers: My name is Jianfeng Mao and I am a software developer at the U2 group in Rocket Software (http://u2.rocketsoftware.com/). I am currently working on a project to embed a slightly customized Python interpreter in our product. For easy installation and setup, we hope to be able to do the standard Python installation during the installation of our software. Basically I want to create a .msi file that can be called to install the full Python if the user needs … [View More]

2 1

PEP 0 maintenance - deferring some currently open PEPs
by Nick Coghlan May 10, 2013

May 10, 2013

I'd like to mark a few PEPs that are not currently being actively considered for 3.4 as Deferred: S 286 Enhanced Argument Tuples von Löwis S 337 Logging Usage in the Standard Library Dubner S 368 Standard image protocol and class Mastrodomenico I 396 Module Version Numbers Warsaw S 400 Deprecate codecs.StreamReader and codecs.StreamWriter Stinner S 419 Protecting … [View More]

7 10