Mailman 3 April 2008 - Python-Dev

New project : Spyke python-to-C compiler
by Rahul Garg April 7, 2008

April 7, 2008

Note this message has been posted to numpy-discussion and python-dev. Sorry for the multiple posting but I thought both python devs and numpy users will be interested. If you believe your list should not receive this email, let me know. Also I just wanted to introduce myself since I may ask doubts about Python and Numpy internals from time to time :) Hi. I am a student at Univ of Alberta doing my masters in computing science. I am writing a Python-to-C compiler as one part of my thesis. The compiler, named Spyke, will be made available in a couple of weeks and is geared towards scientific applications and will therefore focus mostly on needs of scientific app developers. What is Spyke? In many performance critical projects, it is often necessary to rewrite parts of the application in C. However writing C wrappers can be time consuming. Spyke offers an alternative approach. You add annotations to your Python code as strings. These strings are discarded by the Python interpreter but these are interpreted as types by Spyke compiler to convert to C. Example : "int -> int" def f(x): return 2*x In this case the Spyke compiler will consider the string "int -> int" as a decalration that the function accepts int as parameter and returns int. Spyke will then generate a C function and a wrapper function. This idea is directly copied from PLW (Python Language Wrapper) project. Once Python3k arrives, much of these declarations will be moved to function annotations and class decorators. This way you can do all your development and debugging interactively using the standard Python interpreter. When you need to compile to C, you just add type annotations to places that you want to convert and invoke spyke on the annotated module. This is different from Pyrex because Pyrex does not accept Python code. With Spyke, your code is 100% pure python. Spyke has basic support for functions and classes. Spyke can do very basic type inference for local variables in function bodies. Spyke also has partial support for homogenous lists and dictionaries and fixed length tuples. One big advantage of Spyke is that it understands at least part of numpy. Numpy arrays are treated as fundamental types and Spyke knows what C code to generate for slicing/indexing of numpy arrays etc. This should help a lot in scientific applications. Note that Spyke can handle only a subset of Python. Exceptions, iterators, generators, runtime code generation of any kind etc is not handled. Nested functions will be added soon. I will definitely add some of these missing features based on what is actually required for real world Python codes. Currently if Spyke does not understand a function, it just leaves it as Python code. Classes can be handled but special methods are not currently supported. The support of classes is a little brittle because I am trying to resolve some issues b/w old and new style of classes. Where is Spyke? Spyke will be available as a binary only release in a couple of weeks. I intend to make it open source after a few months. Spyke is written in Python and Java and should be platform independant. I do intend to make the source open in a few months. Right now its undergoing very rapid development and has negligible amounts of documentation so the source code right now is pretty useless to anyone else anyway. I need help: However I need a bit of help. I am having a couple of problems : a) I am finding it hard to get pure Python+NumPy testing codes. I need more codes to test the compiler. Developing a compiler without a test-suite is kind of useless. If you have some pure Python codes which need better performance, please contact me. I guarantee that your codes will not be released to public without your permission but might be referenced in academic publications. I can also make the compiler available to you hopefully after 10th of April. Its kind of unstable currently. I will also need your help in annotating the provided testing codes since I probably wont know what your application is doing. b) Libraries which interface with C/C++ : Many codes in SciPy for instance have mixed language codes. Part of the code is written in C/C++. Spyke only knows how to annotated Python codes. For C/C++ libraries wrapped into Python modules, Spyke will therefore need to know at least 2 things : i) The mapping of a C function name/struct etc to Python ii) The type information of the said C function. There are many many ways that people interact with C code. People either write wrappers manually, or use autogenerated wrappers using SWIG or SIP Boost.Python etc., use Pyrex or Cython while some people use ctypes. I dont have the time or resources to support these multitude of methods. I considered trying to parse the C code implementing wrappers but its "non-trivial" to put it mildly. Parsing only SWIG generated code is another possibility but its still hard. Another approach that I am seriously considering is to support a subset of ctypes (with additional restriction) instead. But my question is : Is ctypes good enough for most of you? Ctypes cannot interface with C++ code but its pure Python. However I have not seen too many instances of people using ctypes. c) Strings as type declarations : Do you think I should use decorators instead at least for function type declarations? thanks for patiently reading this, comments and inquiries sought. rahul

4 4

python source code
by Avi Kohn April 7, 2008

April 7, 2008

I am interested in understanding the python source code. Can someone direct me to resources (documentation,book,archive of mailling lists,etc) that will assist me ? Thank you, Avi -- Avi Kohn

2 1

Python 3000: Special type for object attributes & map keys
by Henrik Vendelbo April 7, 2008

April 7, 2008

It appears to me that if you can make mapping mechanisms faster in Python you can make significant overall speed improvements. I also think the proposed concept could add flexibility to persistence formats and RMI interfaces. My basic idea is to have a constant string type with an interpreter globally unique hash. If the original constant is created in a manner different from string constants, it can be tracked and handled differently by the interpreter. Obviously most object attribute references are done with the dot operator, so I guess the interpreter already has an efficient mapping mechanism. But there must be a crossover with __getattr__ etc, where a map of some sort is used. I imagine that having a global namespace to translate attribute names into integers could be used for several purposes by the interpreter as well as an application exchanging objects with other applications. I imagine these expressions to be supported: * attrname(string) - creates an attrname value from the string * int(attrname) - gets the hash value * string(attrname) - gets the string value Hope this makes sense Henrik P.S. I originally thought of this in a JavaScript context so forgive me if this would make little difference in Python.

6 5

socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows)
by Trent Nelson April 7, 2008

April 7, 2008

I started looking into this: http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/289/step-t… Pertinent part: test_asyncore <snip> test_asynchat command timed out: 1200 seconds without output SIGKILL failed to kill process using fake rc=-1 program finished with exit code -1 remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last): Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process ] I tried to replicate it on the buildbot in order to debug, which, surprisingly, I could do consistently by just running rt.bat -q -d -uall test_asynchat. As the log above indicates, the python process becomes completely and utterly wedged, to the point that I can't even attach a remote debugger and step into it. Digging further, I noticed that if I ran the following code in two different python consoles, EADDRINUSE was *NOT* being raised by socket.bind(): import socket s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.bind(('127.0.0.1', 54322)) However, take out the setsockopt line, and wallah, the second s.bind() will raise EADDRINUSE, as expected. This manifests into a really bizarre issue with test_asynchat in particualr, as subsequent sock.accept() calls on the socket put python into the uber wedged state (can't even ctrl-c out at the console, need to kill the process directly). Have to leave the office and head home so I don't have any more time to look at it tonight -- just wanted to post here for others to mull over. Trent.

3 6

Remove "current" Windows executables from Lib/distutils/command in svn?
by Mark Hammond April 7, 2008

April 7, 2008

As I was preparing to check-in my cross-compilation patch, which includes a new x64 executable in the Lib/distutils/command directory, it occurs to me that we don't actually need it in subversion - and nor do we need wininst-9.0.exe. I believe these executables are in svn for historical reasons. The build process for these executables are now integrated into the core Python build process (although they are disabled by default). There is a good use-case for keeping executables built with older MSVC versions, but I don't see one for executables built with the current version. I'd like to propose we delete Lib/Distutils/command/wininst-9.0.exe, and enable the building of that project by default in the standard build process (and I'll setup the x64 build of the executable similarly). They will then need to be re-added when a future version moves away from Visual Studio 2008, but that seems like a reasonable way to approach things. Are there any objections to this, or should we stick with the status-quo and I'll add the new x64 executable? Or do things that way *just* for x64? Thanks, Mark.

5 6

Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbot/bui...
by Trent Nelson April 6, 2008

April 6, 2008

> I don't like the part where the solution kills the Python process during > a rebuild. It's too surprising for the user. Hmmm. When you say rebuild, I assume you mean the change I made to the pythoncore project's pre-link step to call kill_python.exe, and not to the post-build step of kill_python that runs itself? Personally, when I'm doing development, if I've got the pcbuild\python_d.exe console open, it's usually to test one liners, I'm not using it to do anything important. If I forget to close it before I kick off a build, it's annoying running into errors at the link stage, I'd certainly prefer the build system to kill off anything that'll inhibit a successful link before actually linking. What do others that do Windows development think? I don't have a problem changing the build behaviour if the approach I've taken is generally disliked. Trent. ________________________________________ From: Christian Heimes [lists(a)cheimes.de] Sent: 04 April 2008 09:25 To: python-checkins(a)python.org; Trent Nelson Subject: Re: r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbot/bui... trent.nelson schrieb: > Author: trent.nelson > Date: Thu Apr 3 20:27:06 2008 > New Revision: 62129 > > Added: > python/trunk/PCbuild/kill_python.c (contents, props changed) > python/trunk/PCbuild/kill_python.vcproj > Removed: > python/trunk/Tools/buildbot/Makefile > python/trunk/Tools/buildbot/kill_python.bat > python/trunk/Tools/buildbot/kill_python.c > python/trunk/Tools/buildbot/kill_python.mak > Modified: > python/trunk/PCbuild/debug.vsprops > python/trunk/PCbuild/pcbuild.sln > python/trunk/PCbuild/pythoncore.vcproj > python/trunk/PCbuild/release.vsprops > python/trunk/Tools/buildbot/build-amd64.bat > python/trunk/Tools/buildbot/build.bat > python/trunk/Tools/buildbot/buildmsi.bat > Log: > Reimplement kill_python. The existing version had a number of flaws, namely, it didn't work for x64 and it wasn't precise about which python_d.exe it was killing -- it just killed the first one it came across that happened to have 'pcbuild\python_d.exe' or 'build\python_d.exe' in it's path. The new version has been rewritten from the ground up and now lives in PCbuild, instead of Tools\buildbot, and it has also been incorporated into the Visual Studio solution (pcbuild.sln) as 'kill_python'. The solution has also been altered such that kill_python is called where necessary in the build process in order to prevent any linking errors due to open file locks. In lieu of this, all of the existing bits and pieces in Tools\buildbot that called out to kill_python at various points have also be > en removed as they are now obsolete. Tested on both Win32 and x64. I don't like the part where the solution kills the Python process during a rebuild. It's too surprising for the user. Christian

2 3

Thread-safe file objects, the return
by Antoine Pitrou April 6, 2008

April 6, 2008

Hello, It seems this subject has had quite a bit of history. Tim Peters demonstrated the problem in 2003 in this message: http://mail.python.org/pipermail/python-dev/2003-June/036537.html In short, Python file objects release the GIL before calling any C stdlib function on their embedded FILE pointer. Unfortunately, if another thread calls fclose on the FILE pointer concurrently, the contents pointed to can become garbage and the interpreter process crashes. Just by using the same file object in two threads running pure Python code, you can crash the interpreter. (another, easier-to-solve problem is that the FILE pointer stored in the file object could become NULL at the point it is used by another thread. If that was the only problem you could just store the FILE pointer in a local variable before releasing the GIL et voilà) There was some discussion at the time about the possible resolution. I've tried to fix the problem, and I've come to what I think is a satisfying solution, which I can sum up as the following bullet points: * Each file object gets a dedicated counter, which is incremented before the bject releases the GIL and decremented after the GIL is taken again; thus this counter keeps track of how many running "unlocked" sections of code are using that particular file object. (please note the counter doesn't need its own lock, since it is only modified in GIL-protected sections) * In the close() method, if the aforementioned counter is greater than 0, we refuse to call fclose and instead raise an IOError. This may seem like a worrying semantic change, but I don't think it is, for the following reasons: 1) if we closed the FILE pointer anyway, the interpreter would likely crash because another thread would be using garbage data (that's what we are trying to fix after all!) 2) if close() raises an IOError, it can be called again later, or at worse fclose will be called when the file object is garbage collected 3) close() can already raise an IOError if fclose fails for whatever reason (although for sure it's probably very rare) 4) it doesn't seem wrong to notify the programmer that his code is very unsafe The patch is attached at http://bugs.python.org/issue815646 . It addresses (or at least I hope it does) all potential problems with pure Python code, threads, and the file object. It doesn't try to fix C extensions using the PyFile_AsFile API and doing their own dirty things with the FILE pointer. It could be a second step if the approach is accepted, but as noted in the 2003 discussions it would probably involve a new API. Whether we want to introduce such an API in Python 2.x while Python 3.0 has a different IO model anyway is left open to discussion :) Regards Antoine.

3 5

inittimezone - shouldn't it be static?
by skip＠pobox.com April 5, 2008

April 5, 2008

Looks to me like the inittimezone function in timemodule.c should be static. I don't see it called from any code other than within that module. If it's not supposed to be static it needs a more pythonic name. (Can't dig into this just now as I'm flying from Denver to Chicago a.t.m...) Skip

2 2

__eq__ vs hash
by Ralf Schmitt April 5, 2008

April 5, 2008

Hi all, the news file for python 2.6 does not mention that you need to define __hash__ in case you define __eq__ for a class. This breaks some code (for me: mercurial and pyparsing). Shouldn't this be documented somewhere (I also cannot find it in the whatsnew file). - Ralf

7 9

Antony Joseph wants to chat
by Antony Joseph April 5, 2008

April 5, 2008

----------------------------------------------------------------------- Antony Joseph wants to stay in better touch using some of Google's coolest new products. If you already have Gmail or Google Talk, visit: http://mail.google.com/mail/b-29833b0a8-b7e4c018a7-1e5ec176e7509c87 You'll need to click this link to be able to chat with Antony Joseph. To get Gmail - a free email account from Google with over 2,800 megabytes of storage - and chat with Antony Joseph, visit: http://mail.google.com/mail/a-29833b0a8-b7e4c018a7-e7105acb3e Gmail offers: - Instant messaging right inside Gmail - Powerful spam protection - Built-in search for finding your messages and a helpful way of organizing emails into "conversations" - No pop-up ads or untargeted banners - just text ads and related information that are relevant to the content of your messages All this, and its yours for free. But wait, there's more! By opening a Gmail account, you also get access to Google Talk, Google's instant messaging service: http://www.google.com/talk/ Google Talk offers: - Web-based chat that you can use anywhere, without a download - A contact list that's synchronized with your Gmail account - Free, high quality PC-to-PC voice calls when you download the Google Talk client Gmail and Google Talk are still in beta. We're working hard to add new features and make improvements, so we might also ask for your comments and suggestions periodically. We appreciate your help in making our products even better! Thanks, The Google Team To learn more about Gmail and Google Talk, visit: http://mail.google.com/mail/help/about.html http://www.google.com/talk/about.html (If clicking the URLs in this message does not work, copy and paste them into the address bar of your browser).

1 0