Mailman 3 February 2007 - Python-Dev

pypy-0.99.0: new object spaces, optimizations, configuration ...
by Samuele Pedroni Feb. 17, 2007

Feb. 17, 2007

================================================================= pypy-0.99.0: new object spaces, optimizations, configuration ... ================================================================= Welcome to the PyPy 0.99.0 release - a major snapshot and milestone of the last 8 months of work and contributions since PyPy-0.9.0 came out in June 2006! Main entry point for getting-started/download and documentation: http://codespeak.net/pypy/dist/pypy/doc/index.html Further below you'll find some notes about PyPy, the 0.99.0 highlights and our aims for PyPy 1.0. have fun, the PyPy team, Samuele Pedroni, Carl Friedrich Bolz, Armin Rigo, Michael Hudson, Maciej Fijalkowski, Anders Chrigstroem, Holger Krekel, Guido Wesdorp and many others: http://codespeak.net/pypy/dist/pypy/doc/contributor.html What is PyPy? ================================ Technically, PyPy is both a Python Interpreter implementation and an advanced Compiler, actually a framework for implementing dynamic languages and generating virtual machines for them. The Framework allows for alternative frontends and for alternative backends, currently C, LLVM and .NET. For our main target "C", we can can "mix in" different Garbage Collectors and threading models, including micro-threads aka "Stackless". The inherent complexity that arises from this ambitious approach is mostly kept away from the Python interpreter implementation, our main frontend. Socially, PyPy is a collaborative effort of many individuals working together in a distributed and sprint-driven way since 2003. PyPy would not have gotten as far without the coding, feedback and general support from numerous people. Formally, many of the current developers are involved in executing an EU contract with the goal of exploring and researching new approaches to Language/Compiler development and software engineering. This contract's duration is about to end March 2007 and we are working and preparing the according final review which is scheduled for May 2007. Key 0.99.0 Features ===================== * new object spaces: - Tainting: a 270-line proxy object space tracking and boxing sensitive information within an application. A tainted object is completely barred from crossing an I/O barrier, such as writing to files, databases or sockets. This allows to significantly reduce the effort of e.g. security reviews to the few places where objects are "declassified" in order to send information across I/O barriers. - Transparent proxies: allow to customize both application and builtin objects from application level code. Works as an addition to the Standard Object Space (and is translatable). For details see http://codespeak.net/pypy/dist/pypy/doc/proxy.html * optimizations: - Experimental new optimized implementations for various built in Python types (strings, dicts, lists) - Optimized builtin lookups to not require any dictionary lookups if the builtin is not shadowed by a name in the global dictionary. - Improved inlining (now also working for higher level backends) and malloc removal. - twice the speed of the 0.9 release, overall 2-3 slower than CPython * High level backends: - It is now possible to translate the PyPy interpreter to run on the .NET platform, which gives a very compliant (but somewhat slow) Python interpreter. - the JavaScript backend has evolved to a point where it can be used to write AJAX web applications with it. This is still an experimental technique, though. For demo applications see: http://play1.codespeak.net:8008/ * new configuration system: There is a new comprehensive configuration system that allows fine-grained configuration of the PyPy standard interpreter and the translation process. * new and improved modules: Since the last release, the signal, mmap, bz2 and fcntl standard library modules have been implemented for PyPy. The socket, _sre and os modules have been greatly improved. In addition we added a the pypymagic module that contains PyPy-specific functionality. * improved file implementation: Our file implementation was ported to RPython and is therefore faster (and not based on libc). * The stability of stackless features was greatly improved. For more details see: http://codespeak.net/pypy/dist/pypy/doc/stackless.html * RPython library: The release contains our emerging RPython library that tries to make programming in RPython more pleasant. It contains an experimental parser generator framework. For more details see: http://codespeak.net/pypy/dist/pypy/doc/rlib.html * improved documentation: - extended documentation about stackless features: http://codespeak.net/pypy/dist/pypy/doc/stackless.html - PyPy video documentation: eight hours of talks, interviews and features: http://codespeak.net/pypy/dist/pypy/doc/video-index.html - technical reports about various aspects of PyPy: http://codespeak.net/pypy/dist/pypy/doc/index-report.html The entry point to all our documentation is: http://codespeak.net/pypy/dist/pypy/doc/index.html What about 1.0? ====================== In the last week leading up to the release, we decided to go for tagging the release as 0.99.0, mainly because we have some efforts pending to integrate and complete research and coding work: * the JIT Compiler Generator is ready, but not fully integrated with the PyPy interpreter. As a result, the JIT does not give actual speed improvements yet, so we chose to leave it out of the 0.99 release: the result doesn't meet yet the speed expectations that we set for ourselves - and which some blogs and people have chosen as the main criterium for looking at PyPy. * the extension enabling runtime changes of the Python grammar is not yet integrated. This will be used to provide Aspect-Oriented Programming extensions and Design by Contract facilities in PyPy. * the Logic object space, which provides Logic Variables in PyPy, needs to undergo a bit more testing. A constraint problem solver extension module is ready, and needs to be integrated with the codebase. PyPy 0.99 is the start for getting to 1.0 end of March 2007, which we intend to become a base for a longer (and more relaxed :) time to come. Funding partners and organisations ===================================================== PyPy development and activities happen as an open source project and with the support of a consortium partially funded by a 28 months European Union IST research grant. The full partners of that consortium are: Heinrich-Heine University (Germany), Open End (Sweden) merlinux GmbH (Germany), tismerysoft GmbH (Germany) Logilab Paris (France), DFKI GmbH (Germany) ChangeMaker (Sweden), Impara (Germany)

1 0

Recent experience with the _ast module
by Collin Winter Feb. 17, 2007

Feb. 17, 2007

I've been working this past week at converting the stdlib's compiler package to use 2.5's new _ast module instead of the package's own AST. Overall, _ast was a joy to work with, save for the following nitpicks: 1) There are times when the _fields attribute on some AST nodes is None; if this was done to indicate that a given node has no AST-related attributes, it would be much easier if _fields was [] in this case. As it is, I had to special-case "node._fields is None" in the visitor so that I don't accidentally iterate over it. 2) {BinOp,AugAssign,BoolOp,etc}.op is an instance of, eg, Add, Sub, Mult, Mod, meaning you have to dispatch based on tests like "isinstance(node.op, x)" or "type(node.op) is x". I would much, much prefer to spell this "node.op is x", ie, use "node.op = Add" rather than the current "node.op = Add()" when constructing the nodes. 3) I'd like there to be an Else() node for If.orelse, While.orelse, etc. My motivation is that I need the lineno and col_offset values of the "else" statement for a code-coverage utility; as it is, I have to find the last lineno of the if/while/etc suite and the first lineno of the "else" suite and then search between those two for the "else" statement. Thoughts? Thanks, Collin Winter

4 17

Re: [Python-Dev] generic async io (was: microthreading vs. async io)
by Nick Maclaren Feb. 17, 2007

Feb. 17, 2007

dustin(a)v.igoro.us wrote: > > Knowing the history of something like this is very helpful, but I'm not > sure what you mean by this first paragraph. I think I'm most unclear > about the meaning of "The 'threading' approach to asynchronous I/O"? > Its opposite ("separating asynchronous I/O from threading") doesn't > illuminate it much more. Could you elaborate? I'll try. Sorry about being unclear - it is one of my failings. Here is an example draft of some interfaces: Threading --------- An I/O operation passes a buffer, length, file and action and receives a token back. This token can be queried for completion, waited on and so on, and is cancelled by waiting on it and getting a status back. I.e. it is a thread-like object. This is the POSIX-style operation, and is what I say cannot be made to work effectively. Streaming --------- An I/O operation either writes some data to a stream or reads some data from it; such actions are sequenced within a thread, but not between threads (even if the threads coordinate their I/O). Data written goes into limbo until it is read, and there is no way for a reader to find the block boundaries it was written with or whether data HAS been written. A non-blocking read merely tests if data are ready for reading, which is not the same. There are no positioning operations, and only open, close and POSSIBLY a heavyweight synchronise or rewind (both equivalent to close+open) force written data to be transferred. Think of Fortran sequential I/O without BACKSPACE or C I/O without ungetc/ungetchar/fseek/fsetpos. Transactions ------------ An I/O operation either writes some data to a file or reads some data from it. There is no synchronisation of any form until a commit. If two transfers between a pair of commits overlap (including file length changes), the behaviour is undefined. All I/O includes its own positioning, and no positioning is relative. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1(a)cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679

3 2

Re: [Python-Dev] urllib2 EP + decr. startup time
by Phillip J. Eby Feb. 17, 2007

Feb. 17, 2007

At 07:29 PM 2/16/2007 +0200, KoDer wrote: >2007/2/16, Phillip J. Eby <pje(a)telecommunity.com>: > > At 04:38 PM 2/16/2007 +0200, KoDer wrote: >..... > > > > > > Also, are you aware that putting a zipped version of the standard library > > on sys.path already speeds up startup considerably? Python since 2.3 > > automatically includes an appropriate entry in sys.path: > > > >zipped version has one weakness - you can't put .so(or dll) files inside. >In my system 19 from 25 installed egg add directories ,not archives >(because it's contain dll ). There's something wrong there. You can put .so, .dll, or .pyd files in eggs - they get extracted to a cache directory automatically, but those directories *don't* get added to sys.path. The mere presence of such files is not enough to cause an egg to be installed unzipped. The package has to be using things like __file__, the package author has to have marked it zip-unsafe, or easy_install was invoked with a request to install unzipped. As long as none of these conditions apply, the egg should be installed zipped, with dynamic libraries automatically extracted on-demand to the $PYTHON_EGG_CACHE. >But even without egg directories >> >['', >'C:\\Python25\\Scripts', >'C:\\WINDOWS\\system32\\python25.zip', >'C:\\Python25\\DLLs', >'C:\\Python25\\lib', >'C:\\Python25\\lib\\plat-win', >......... >'C:\\Python25\\lib\\site-packages\\wx-2.8-msw-unicode'] >len(sys.path) == 18 (without eggs) near 18 / 2 = 9 'file not found' errors > for every first module import. That depends on what the module is. In the path you've shown, having a python25.zip would allow only 2 failures before each stdlib import. (Note that an failed zipfile imports are roughly equivalent to a dictionary lookup in time - they don't access the filesystem once the zipfile index is loaded). >So improvement of setuptools will help, but not solve this problem . Right -- most of your problem will be solved by creating 'C:\\WINDOWS\\system32\\python25.zip', containing the contents of C:\\Python25\\lib\\. Hm. Interesting, actually. Does anybody know why it's looking for 'C:\\WINDOWS\\system32\\python25.zip'? That seems wrong to me.

4 3

Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
by glyph＠divmod.com Feb. 17, 2007

Feb. 17, 2007

On 16 Feb, 06:30 pm, larry(a)hastings.org wrote: >I suggest it is possible to implement a PerfectReactor. I don't think this constitutes a sufficient existence proof. Perhaps you could write a prototype? There are a bunch of existing reactors you could either import or copy/paste from to bootstrap it, so you wouldn't necessarily need to do many of the platform-specific hacks it has already implemented for those loops. Personally I am highly skeptical that this is possible, and I didn't find any of your arguments convincing, but I would be ecstatic if it worked; it isn't *fun* maintaining 12 or so subtly incompatible implementations of the same interface :). I also think this discussion would be more appropriate to continue on the Twisted list, as we have veered pretty far afield of stdlib enhancements and are now talking about multiplexing method implementations, but I'll follow it wherever it continues.

1 0

Re: [Python-Dev] Py2.6 ideas
by Delaney, Timothy (Tim) Feb. 16, 2007

Feb. 16, 2007

Guido van Rossum wrote: > On 2/15/07, Raymond Hettinger <raymond.hettinger(a)verizon.net> wrote: >> * Add a pure python named_tuple class to the collections module. >> I've been using the class for about a year and found that it greatly >> improves the usability of tuples as records. >> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 > > Hm, but why would they still have to be tuples? Why not just have a > generic 'record' class? Hmm - possibilities. "record" definitely has greater connotations of heterogeneous elements than "tuple", which would put paid to the constant arguments that "a tuple is really just an immutable list". list - primarily intended for homogeneous elements record - primarily intended for heterogeneous elements, elements are (optionally?) named and have mutable and immutable versions of each. Maybe the current list syntax would then continue to create a mutable list, and the current tuple syntax would create an immutable record (with no element names) i.e. the current tuple. Tim Delaney

10 15

Alternative Python VM
by Sokolov Yura Feb. 16, 2007

Feb. 16, 2007

It could be interesting. """ - pyvm is * 2 times * faster than Python 2.4. In the source code there is a collection of benchmarks which includes 65 python scripts collected from the internet. At average if Python 2.4 needs 24 hours to do some job, pyvm can do it in 12 hours. - pyvm is a virtual machine that can run Python 2.3/2.4 bytecode. There is a compiler written in python (the 'pyc' program) which is executed by the vm to compile source code to bytecode. It is very easy to do advanced optimizations in python and the compiler produces bytecode of high quality (speed and size). """ http://students.ceid.upatras.gr/~sxanth/pyvm/ Sokolov Yura (funny_falcon)

2 1

Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
by glyph＠divmod.com Feb. 16, 2007

Feb. 16, 2007

On 08:52 pm, theller(a)ctypes.org wrote: >When I last looked at twisted (that is several years ago), there were >several reactors - win32reactor, wxreactor, maybe even more. Yes. That's intentional. Each of those systems offers its own event loop, and each reactor implements the basic operations in terms of those loops. They all have the same API. Application code does 'from twisted.internet import reactor; reactor.listenTCP', 'reactor.callLater', etc. Only the very top-most level decides which reactor the application will use. >And they >didn't even work too well. The problems I remember were that the win32reactor >was limited to a only handful of handles, the wxreactor didn't process >events when a wx modal dialog boy was displayed, and so on. Has this changed? win32eventreactor is limited to 64 handles because WaitForMultipleObjects is limited to 64 handles. wxreactor's event loop is ghastly and actually did pause when a modal dialog box is displayed (that has since been fixed). Process support on win32 now works in the default select reactor as well as the gtk reactor, so win32reactor is mostly a curiosity at this point (useful mainly if you want to implement your own GDI-based GUI, as PyUI did at one point), and its limitations are not as serious for Twisted as a whole. In other words, Twisted exposes the platform limitations in its platform-specific event loop implementations, and only works around them where it's possible to do so without masking platform functionality. For servers, the epoll, poll, and select reactors work just fine. The select reactor does have a maximum of FD_SETSIZE simultaneous sockets as well, but it is very easy to switch reactors if you need something more scalable. For clients, the best GUI toolkit for Twisted applications at this point is GTK, but WX, QT and Cocoa can all be made to work with a minimum of hassle.

13 28

Re: [Python-Dev] urllib2 EP + decr. startup time
by Phillip J. Eby Feb. 16, 2007

Feb. 16, 2007

At 04:38 PM 2/16/2007 +0200, KoDer wrote: > 'strace' command shows next: most of startup time the interpreter >try to find imported modules. > And most of them finished with 'not found' error, because of large >size of sys.path variable. > In future this time will be increase - setuptools adds many dirs to >search path > using pth files (to manage installed modules and eggs). Actually, under normal circumstances, most eggs installed are .zip files, which the interpreter already caches the indexes of. Eggs installed as directories should be increasing in rarity, except for in-development packages installed via the "develop" command. Also, I plan to make setuptools 0.7's "nest" packaging tool support managing packages the "old" way, i.e., as a single flat directory structure, using manifest files to support uninstallation and the like. So it should not really be the case that this will keep increasing over time. Also, are you aware that putting a zipped version of the standard library on sys.path already speeds up startup considerably? Python since 2.3 automatically includes an appropriate entry in sys.path: Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] on win32 >>> import sys >>> sys.path ['', 'C:\\WINDOWS\\system32\\python23.zip', '', 'c:\\Python23\\DLLs', 'c:\\Pytho n23\\lib', 'c:\\Python23\\lib\\plat-win', 'c:\\Python23\\lib\\lib-tk', 'c:\\Pyth on23'] Creating the zip file that's already listed in the default sys.path will allow most startup imports to be handled without a lot of path checking. > I propose to add something like .so caching which used in modern >*nix sytems to load > shared libraries. > > a) Add to python interpreter --build-modules-index option. When python > found > this opts it scans all dirs in paths and build dictionary >{module_name:module_path}. > Dict will be saved in external file (save only top-dir for packages >and path for one-file modules). > Also it saves in this file mtime for all pth files and dirs from >path and path variable. Unless you mean something more abstract by "dirs" than just filesystem directories, this isn't going to help eggs or other zipped modules any, compared to how they are now.

2 1

urllib2 EP + decr. startup time
by KoDer Feb. 16, 2007

Feb. 16, 2007

Hello to all. During more than two years i widely use urllib2 to write commercial applications (almost for extracting data from web sites to excel sheets) and here is some enhanced enhanced for it: 1) Add support for 'HEAD' request (and maybe some other). This needs small changes. a)Add request_type = 'GET' to urllib2.Request class constructor. b)Then put request_type value pass to http header, except Request has data - in this case it's change to 'POST'. The results of such request will be the same as in case of 'GET' request, except zero size of body. 2)HTTH keep-alive opener. Almost complete realizations can be found in urlgrabber (http://linux.duke.edu/projects/urlgrabber)(used by yum, so tested well enough, i think). It's use urllib2 opener protocol and well integrated in urllib2 structure. They need just little change to properly support some headers. 3) Save HTTP exchange history. Now there is no suitable way to obtain all sent and received headers. Received headers are saved only for last response in redirection chain and sent headers are not saved at all. I use run-time patching of httplib to intercept of the sent and received data (may be i missed something?). Proposal is to add property 'history' to object returned from urllib2.urlopen - list of objects which contain send/recv headers for all redirect chain. 4) Add possibilities to obtain underlying socket, used for recv http data. Now it's impossible to work with http connection in async matter (or i miss something again?). If connection hangs then whole program hangs too and i don't known way to fix this. Of cause if you obtain such socket then you respond for compression and etc. Now i use following code: x = urrlib2.urlopen(.....) sock = x.fp._sock.fp._sock. There only one problem, as i know, - chunked encoding. In case of chunked encoding need to return socket-like object which do all work to assemble chunks in original stream. I already use such object for two years and it's ok. 5) And now for something completely different ;)). This is just initial proposal and it needs enhancement. May be i need put it to python-ideas list? At last Goggle SOC there was one of problem to solve - the decrease of interpreter's startup time. 'strace' command shows next: most of startup time the interpreter try to find imported modules. And most of them finished with 'not found' error, because of large size of sys.path variable. In future this time will be increase - setuptools adds many dirs to search path using pth files (to manage installed modules and eggs). I propose to add something like .so caching which used in modern *nix sytems to load shared libraries. a) Add to python interpreter --build-modules-index option. When python found this opts it scans all dirs in paths and build dictionary {module_name:module_path}. Dict will be saved in external file (save only top-dir for packages and path for one-file modules). Also it saves in this file mtime for all pth files and dirs from path and path variable. b) When interpreter is started-up it, as usually, scans all path dirs for pth files, and using data saved in cache file check is new modules or search dirs added or old modified. Then it read cache file and compares mtimes and path dirs. If it isn't modified then cache data used for fast module loading. If imported module didn't found in cache - interpreter falls back to standard scheme. Also it is necessary to add some options to control of using cache like --disable-cache, --clear-cache,disable cashing some dirs, etc. --- K.Danilov aka KoDer

1 0