Mailman 3 February 2007 - Python-Dev

pypy-0.99.0: new object spaces, optimizations, configuration ...
by Samuele Pedroni Feb. 17, 2007

Feb. 17, 2007

================================================================= pypy-0.99.0: new object spaces, optimizations, configuration ... ================================================================= Welcome to the PyPy 0.99.0 release - a major snapshot and milestone of the last 8 months of work and contributions since PyPy-0.9.0 came out in June 2006! Main entry point for getting-started/download and documentation: http://codespeak.net/pypy/dist/pypy/doc/index.html Further below you'll … [View More]find some notes about PyPy, the 0.99.0 highlights and our aims for PyPy 1.0. have fun, the PyPy team, Samuele Pedroni, Carl Friedrich Bolz, Armin Rigo, Michael Hudson, Maciej Fijalkowski, Anders Chrigstroem, Holger Krekel, Guido Wesdorp and many others: http://codespeak.net/pypy/dist/pypy/doc/contributor.html What is PyPy? ================================ Technically, PyPy is both a Python Interpreter implementation and an advanced Compiler, actually a framework for implementing dynamic languages and generating virtual machines for them. The Framework allows for alternative frontends and for alternative backends, currently C, LLVM and .NET. For our main target "C", we can can "mix in" different Garbage Collectors and threading models, including micro-threads aka "Stackless". The inherent complexity that arises from this ambitious approach is mostly kept away from the Python interpreter implementation, our main frontend. Socially, PyPy is a collaborative effort of many individuals working together in a distributed and sprint-driven way since 2003. PyPy would not have gotten as far without the coding, feedback and general support from numerous people. Formally, many of the current developers are involved in executing an EU contract with the goal of exploring and researching new approaches to Language/Compiler development and software engineering. This contract's duration is about to end March 2007 and we are working and preparing the according final review which is scheduled for May 2007. Key 0.99.0 Features ===================== * new object spaces: - Tainting: a 270-line proxy object space tracking and boxing sensitive information within an application. A tainted object is completely barred from crossing an I/O barrier, such as writing to files, databases or sockets. This allows to significantly reduce the effort of e.g. security reviews to the few places where objects are "declassified" in order to send information across I/O barriers. - Transparent proxies: allow to customize both application and builtin objects from application level code. Works as an addition to the Standard Object Space (and is translatable). For details see http://codespeak.net/pypy/dist/pypy/doc/proxy.html * optimizations: - Experimental new optimized implementations for various built in Python types (strings, dicts, lists) - Optimized builtin lookups to not require any dictionary lookups if the builtin is not shadowed by a name in the global dictionary. - Improved inlining (now also working for higher level backends) and malloc removal. - twice the speed of the 0.9 release, overall 2-3 slower than CPython * High level backends: - It is now possible to translate the PyPy interpreter to run on the .NET platform, which gives a very compliant (but somewhat slow) Python interpreter. - the JavaScript backend has evolved to a point where it can be used to write AJAX web applications with it. This is still an experimental technique, though. For demo applications see: http://play1.codespeak.net:8008/ * new configuration system: There is a new comprehensive configuration system that allows fine-grained configuration of the PyPy standard interpreter and the translation process. * new and improved modules: Since the last release, the signal, mmap, bz2 and fcntl standard library modules have been implemented for PyPy. The socket, _sre and os modules have been greatly improved. In addition we added a the pypymagic module that contains PyPy-specific functionality. * improved file implementation: Our file implementation was ported to RPython and is therefore faster (and not based on libc). * The stability of stackless features was greatly improved. For more details see: http://codespeak.net/pypy/dist/pypy/doc/stackless.html * RPython library: The release contains our emerging RPython library that tries to make programming in RPython more pleasant. It contains an experimental parser generator framework. For more details see: http://codespeak.net/pypy/dist/pypy/doc/rlib.html * improved documentation: - extended documentation about stackless features: http://codespeak.net/pypy/dist/pypy/doc/stackless.html - PyPy video documentation: eight hours of talks, interviews and features: http://codespeak.net/pypy/dist/pypy/doc/video-index.html - technical reports about various aspects of PyPy: http://codespeak.net/pypy/dist/pypy/doc/index-report.html The entry point to all our documentation is: http://codespeak.net/pypy/dist/pypy/doc/index.html What about 1.0? ====================== In the last week leading up to the release, we decided to go for tagging the release as 0.99.0, mainly because we have some efforts pending to integrate and complete research and coding work: * the JIT Compiler Generator is ready, but not fully integrated with the PyPy interpreter. As a result, the JIT does not give actual speed improvements yet, so we chose to leave it out of the 0.99 release: the result doesn't meet yet the speed expectations that we set for ourselves - and which some blogs and people have chosen as the main criterium for looking at PyPy. * the extension enabling runtime changes of the Python grammar is not yet integrated. This will be used to provide Aspect-Oriented Programming extensions and Design by Contract facilities in PyPy. * the Logic object space, which provides Logic Variables in PyPy, needs to undergo a bit more testing. A constraint problem solver extension module is ready, and needs to be integrated with the codebase. PyPy 0.99 is the start for getting to 1.0 end of March 2007, which we intend to become a base for a longer (and more relaxed :) time to come. Funding partners and organisations ===================================================== PyPy development and activities happen as an open source project and with the support of a consortium partially funded by a 28 months European Union IST research grant. The full partners of that consortium are: Heinrich-Heine University (Germany), Open End (Sweden) merlinux GmbH (Germany), tismerysoft GmbH (Germany) Logilab Paris (France), DFKI GmbH (Germany) ChangeMaker (Sweden), Impara (Germany) [View Less]

1 0

Recent experience with the _ast module
by Collin Winter Feb. 17, 2007

Feb. 17, 2007

I've been working this past week at converting the stdlib's compiler package to use 2.5's new _ast module instead of the package's own AST. Overall, _ast was a joy to work with, save for the following nitpicks: 1) There are times when the _fields attribute on some AST nodes is None; if this was done to indicate that a given node has no AST-related attributes, it would be much easier if _fields was [] in this case. As it is, I had to special-case "node._fields is None" in the visitor so that I … [View More]

4 17

Re: [Python-Dev] generic async io (was: microthreading vs. async io)
by Nick Maclaren Feb. 17, 2007

Feb. 17, 2007

dustin(a)v.igoro.us wrote: > > Knowing the history of something like this is very helpful, but I'm not > sure what you mean by this first paragraph. I think I'm most unclear > about the meaning of "The 'threading' approach to asynchronous I/O"? > Its opposite ("separating asynchronous I/O from threading") doesn't > illuminate it much more. Could you elaborate? I'll try. Sorry about being unclear - it is one of my failings. Here is an example draft of some interfaces: … [View More]

3 2

Re: [Python-Dev] urllib2 EP + decr. startup time
by Phillip J. Eby Feb. 17, 2007

Feb. 17, 2007

At 07:29 PM 2/16/2007 +0200, KoDer wrote: >2007/2/16, Phillip J. Eby <pje(a)telecommunity.com>: > > At 04:38 PM 2/16/2007 +0200, KoDer wrote: >..... > > > > > > Also, are you aware that putting a zipped version of the standard library > > on sys.path already speeds up startup considerably? Python since 2.3 > > automatically includes an appropriate entry in sys.path: > > > >zipped version has one weakness - you can't put .so(or dll) … [View More]

4 3

Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
by glyph＠divmod.com Feb. 17, 2007

Feb. 17, 2007

On 16 Feb, 06:30 pm, larry(a)hastings.org wrote: >I suggest it is possible to implement a PerfectReactor. I don't think this constitutes a sufficient existence proof. Perhaps you could write a prototype? There are a bunch of existing reactors you could either import or copy/paste from to bootstrap it, so you wouldn't necessarily need to do many of the platform-specific hacks it has already implemented for those loops. Personally I am highly skeptical that this is possible, and I didn't … [View More]

1 0

Re: [Python-Dev] Py2.6 ideas
by Delaney, Timothy (Tim) Feb. 16, 2007

Feb. 16, 2007

Guido van Rossum wrote: > On 2/15/07, Raymond Hettinger <raymond.hettinger(a)verizon.net> wrote: >> * Add a pure python named_tuple class to the collections module. >> I've been using the class for about a year and found that it greatly >> improves the usability of tuples as records. >> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 > > Hm, but why would they still have to be tuples? Why not just have a > generic 'record' class? Hmm -… [View More]

10 15

Alternative Python VM
by Sokolov Yura Feb. 16, 2007

Feb. 16, 2007

It could be interesting. """ - pyvm is * 2 times * faster than Python 2.4. In the source code there is a collection of benchmarks which includes 65 python scripts collected from the internet. At average if Python 2.4 needs 24 hours to do some job, pyvm can do it in 12 hours. - pyvm is a virtual machine that can run Python 2.3/2.4 bytecode. There is a compiler written in python (the 'pyc' program) which is executed by the vm to compile source code to bytecode. It is very easy to do … [View More]

2 1

Re: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib)
by glyph＠divmod.com Feb. 16, 2007

Feb. 16, 2007

On 08:52 pm, theller(a)ctypes.org wrote: >When I last looked at twisted (that is several years ago), there were >several reactors - win32reactor, wxreactor, maybe even more. Yes. That's intentional. Each of those systems offers its own event loop, and each reactor implements the basic operations in terms of those loops. They all have the same API. Application code does 'from twisted.internet import reactor; reactor.listenTCP', 'reactor.callLater', etc. Only the very top-most level … [View More]

13 28

Re: [Python-Dev] urllib2 EP + decr. startup time
by Phillip J. Eby Feb. 16, 2007

Feb. 16, 2007

At 04:38 PM 2/16/2007 +0200, KoDer wrote: > 'strace' command shows next: most of startup time the interpreter >try to find imported modules. > And most of them finished with 'not found' error, because of large >size of sys.path variable. > In future this time will be increase - setuptools adds many dirs to >search path > using pth files (to manage installed modules and eggs). Actually, under normal circumstances, most eggs installed are .zip files, which the … [View More]

2 1

urllib2 EP + decr. startup time
by KoDer Feb. 16, 2007

Feb. 16, 2007

Hello to all. During more than two years i widely use urllib2 to write commercial applications (almost for extracting data from web sites to excel sheets) and here is some enhanced enhanced for it: 1) Add support for 'HEAD' request (and maybe some other). This needs small changes. a)Add request_type = 'GET' to urllib2.Request class constructor. b)Then put request_type value pass to http header, except Request has data - in this case it's change to 'POST'. The results of such … [View More]request will be the same as in case of 'GET' request, except zero size of body. 2)HTTH keep-alive opener. Almost complete realizations can be found in urlgrabber (http://linux.duke.edu/projects/urlgrabber)(used by yum, so tested well enough, i think). It's use urllib2 opener protocol and well integrated in urllib2 structure. They need just little change to properly support some headers. 3) Save HTTP exchange history. Now there is no suitable way to obtain all sent and received headers. Received headers are saved only for last response in redirection chain and sent headers are not saved at all. I use run-time patching of httplib to intercept of the sent and received data (may be i missed something?). Proposal is to add property 'history' to object returned from urllib2.urlopen - list of objects which contain send/recv headers for all redirect chain. 4) Add possibilities to obtain underlying socket, used for recv http data. Now it's impossible to work with http connection in async matter (or i miss something again?). If connection hangs then whole program hangs too and i don't known way to fix this. Of cause if you obtain such socket then you respond for compression and etc. Now i use following code: x = urrlib2.urlopen(.....) sock = x.fp._sock.fp._sock. There only one problem, as i know, - chunked encoding. In case of chunked encoding need to return socket-like object which do all work to assemble chunks in original stream. I already use such object for two years and it's ok. 5) And now for something completely different ;)). This is just initial proposal and it needs enhancement. May be i need put it to python-ideas list? At last Goggle SOC there was one of problem to solve - the decrease of interpreter's startup time. 'strace' command shows next: most of startup time the interpreter try to find imported modules. And most of them finished with 'not found' error, because of large size of sys.path variable. In future this time will be increase - setuptools adds many dirs to search path using pth files (to manage installed modules and eggs). I propose to add something like .so caching which used in modern *nix sytems to load shared libraries. a) Add to python interpreter --build-modules-index option. When python found this opts it scans all dirs in paths and build dictionary {module_name:module_path}. Dict will be saved in external file (save only top-dir for packages and path for one-file modules). Also it saves in this file mtime for all pth files and dirs from path and path variable. b) When interpreter is started-up it, as usually, scans all path dirs for pth files, and using data saved in cache file check is new modules or search dirs added or old modified. Then it read cache file and compares mtimes and path dirs. If it isn't modified then cache data used for fast module loading. If imported module didn't found in cache - interpreter falls back to standard scheme. Also it is necessary to add some options to control of using cache like --disable-cache, --clear-cache,disable cashing some dirs, etc. --- K.Danilov aka KoDer [View Less]

1 0