Mailman 3 October 2008 - pypy-dev

Possible student project on PyPy
by Paolo Giarrusso 17 Nov '08

17 Nov '08

Hi, for a student project, we are evaluating the possibility to experiment with some ideas on PyPy. Even if it is a student project, we have high expectations (our aim is to really improve PyPy's speed, if possible). But we are still trying to choose which features to implement (we started with some nice ideas, but part of them is already implemented). We will work for something like 2 months on a separate branch - maybe we can give readonly access to the source code repository, but write access would be a problem for our exam, for obvious reasons. Do you have any possibility to host a development branch for this project? To let you know who we are, I'll present us: - I, Paolo 'Blaisorblade' Giarrusso, am a past Linux kernel hacker, I worked also with Java, C++, and a bit of Python, and I am currently a graduate student at Aarhus University, Denmark; - Sigurd is a PhD candidate at Aarhus University, Denmark, currently working, among other things, on a cryptography research project in Python (viff.dk). - our professor is Lars Bak, the lead architect of the Google V8 Javascript engine, on which we implemented various optimizations in the previous months. We are obviously open to suggestions, and we have been looking at the status blog and at various blogs. It seems that there is still space for improvement in the space of garbage collectors, as mentioned here: - http://codespeak.net/pypy/extradoc/talk/osdc2008/paper.txt working on that could be interesting. The main idea we wanted to apply to Python, based on a suggestion from Lars, is the main peculiar V8 idea, the one of hidden classes with transitions. That allows avoiding dictionary lookups for property accesses on objects, using instead a Java-like data representation in memory, without any visible effects on the Python semantics. While revising this email, we saw that this is already partially implemented: for what the blog says, the memory savings where done but not the memory speedups. Also, the blog does not mention whether and how class transitions are used. Some of the initial ideas below may already have been implemented, so point out what is there and what can be interesting. I've also read the discussion about the JIT, which will be missing in the next release. Obviously, this would require partial reworking the core object model of PyPy. Still, we think that specialization will especially benefit from this, even because this allows to specialize better also on class types, not only on primitive types. It is not clear to me how do you handle changes (like addition of a property) to a single object - is it possible to specialize the code on the (anonymous) type of this new object? Can you give me pointers inside the documentation, if this is already explained? I've read the EU reports about this and partial compilation, but it is still unclear to us how things work. Btw, the first thing to do is studying the object model and reflection capability of Python. Since Lars said that the target of a VM designer is to allow programmers work in the most pleasant way and use advanced features without incurring any cost, our development plan should try to optimize also reflective features as much as possible. Just to make an example, in Python you can register an handler for unknown method calls on an object (I don't remember how, but I'm pretty sure something like this exists). This idea means that our design should be able to do inline caching by storing a call to this handler (maybe we'll defer the implementation, but the initial design should allow doing that IMHO). Any comments? -- Paolo Giarrusso

3 3

Problems getting Translate.py and Pylint to work
by Andrew Francis 03 Nov '08

03 Nov '08

Hello Colleagues: I am not sure if this is the right place to ask.... I am trying to get RPython to work. I get the following problem: [translation:ERROR] from compiler import parse, ast, pycodegen [translation:ERROR] ImportError: cannot import name parse I also installed Pylint including the Logilab-astng package. I get the error: no module called common.compat. I read a post stating the problem may be with an __init__.py file. I am using Xandros on an ASUS eee (similar to Ubuntu). Any help would be appreciated. Cheers, Andrew

4 5

bugday
by Antonio Cuni 27 Oct '08

27 Oct '08

Hi pypy-dev, we are trying to organize the first "PyPy Bugday", but it seems it's hard to find a proper date for it; so, we've set up a poll on doodle: http://doodle.com/73tepx6ktg9cyd29 the poll is mainly targeted at core pypy developers, but of course everyone can vote. Please spend two minutes for voting, so that we can pick the best date for it. ciao & a bientot, Anto & Armin

3 2

C++ library bindings
by Geoffrey Irving 27 Oct '08

27 Oct '08

Hello, I posted a response to your blog post on C++ library bindings, and wanted to continue the discussion further via email if anyone's interested. I just signed up for the mailing list, so apologies if I missed a lot of previous discussion. I'll say up front that it's unlikely that I'll be able to devote any actual coding effort to this, so feel free to tell me to get lost if you have plenty of ideas and not enough manpower. :) I started out writing C++ bindings using Boost.Python, and was very happy with it for a long time. It's strongest point is the ability to wrap libraries that were never designed with python in mind, specifically code with poor and inflexible ownership semantics. Internally, this means that C++ objects are exposed indirectly through a holder object containing either an inline copy of the C++ object or any type of pointer holding the object. Every access to the object has to go through runtime dispatch in order to work with any possible holder type. The holder also contains the logic for ownership and finalization. For example, Boost.Python can return a reference to a field inside another object, in which case the holder will keep a reference to the parent object to keep it alive as long as the field reference lives. The problem with this generality is that it produces a huge amount of object code (wrapping a single function in Boost.Python can add 10k to the object file), and adds a lot of runtime indirection. Assuming that one is writing C++ bindings because of speed issues, it'd be nice if this extra layer of memory indirection and runtime dispatch was exposed to the (eventual) JIT. In order to do that, pypy would have to be capable of handling pointers to raw memory containing non-python objects (is already true due to ctypes stuff?), with separate information about type and ownership. For example, if you have bindings for a C++ vector class and a C++ array containing the vectors, a "reference" to an individual vector in the array is really three different pieces: 1. The actual pointer to the vector. 2. A type structure containing functions to be called with the pointer (1) as an argument. 3. A list of references to other objects that need to stay alive while this reference lives. If pypy and the JIT ends up able to treat these pieces separately, it'd be a significant performance win over libraries wrapped with CPython. The other main source of slowness and complexity in Boost.Python is overloading support, but I think that part is fairly straightforward to handle in the python level. All Boost.Python does internally is loop over the set of functions registered for a given name, and for each one loop over the arguments calling into its converter registry to see if the python object can be converted to the C++ type. As I mentioned in the blog comment, a lot of these issues come up in contexts outside C++, like numpy. Internally numpy represents operations like addition as a big list of optimized routines to call depending on the stored data type. Functions in these tables are called on raw pointers to memory, which is fundamental since numpy arrays can refer to memory inside objects from C++, Fortran, mmap, etc. It'd be really awesome if the type dispatch step could be written in python but still call into optimized C code for the final arithmetic. The other major issue is safety: if a lot of overloading and dispatch code is going to be written in python, it'd be nice to shield that code from segfaults. I think you can get a long way there just by having a consistent scheme for boxing the three components above (pointer, type, and reference info), a way to label C function pointers with type information, a small RPython layer that did simple type-checked calls (with no support for overloading or type conversion). I just wrote a C++ analogue to this last part as a minimal replacement for Boost.Python, so I could try to formulate what I mean in pseudocode if there's interest. There'd be some amount of duplicate type checking if higher level layers such as overload resolution were written in application level python, but that duplication should be amenable to elimination by the JIT. That's enough for now. I'll look forward to the discussion. Most of my uses of python revolve heavily around C++ bindings, so it's exciting to see that you're starting to think about it even if it's a long way off. Geoffrey

3 3

Re: [pypy-dev] Sprint Discussions: C++ Library Bindings
by David Wilson 23 Oct '08

23 Oct '08

Hi there, I just read about this on the PyPy status blog and find myself wondering about the strategies you considered before settling on the Reflex approach. I've had an extension to ctypes in mind for quite some time that involves simply leaving the debugging segments present in a shared library in order to be able to introspect them and provide type-safe access to the code within. The idea is simple: implement a DWARF/stabs/etc. interpreter that can construct Python types and function proxies to represent what is present in the shared library. Initially I wanted this for C, so that a Python programmer needn't manually declare structs using quirky ctypes class definitions, but I can't see why a (rather more complex) implementation couldn't reconstruct C++ classes with their associated class/member functions. The obvious benefit is that a C/C++ library need not be modified at all in order to benefit, other than not being stripped of debugging information during its build. Linux distributions are slowly gaining infrastructure that makes this almost zero cost (e.g. the Debian python-dbg package). I've had the DWARF specification sitting on my desktop for about 2 months now, waiting for me to validate my idea. I'm wondering if you've considered this approach, and if so, why you discarded it. Thanks, David.

3 2

py2php
by Jurgis Pralgauskis 23 Oct '08

23 Oct '08

Hello, I'd like to make life easier for writing simple php scripts (as I am teaching python and web technologies, but my students (and I) complain, that python web hosting is still not widelly affordable). I imagine, that php is mostly similar to javascript for what's in pypy now first I need very basic things: variables, loops, some string_methods, echo and read/write to file. I looked around https://codespeak.net/viewvc/pypy/dist/pypy/translator/js/ as I see, things would be easy to change as in jts.py and sbuiltin.py, but I dont' see where loop structure representation is generated... maybe smb could construct a bare php translator structure, which I could start filling up and experimenting if this smb is busy right now, I could wait :) Thanks in advance ps.: another idea: can pypy quite easily help to visualize the list (or other structure) manipulation for teaching purposes it would be kind of watchvariables functionality but more the way of https://launchpad.net/algorithms-animator -- Jurgis Pralgauskis tel: 8-616 77613; jabber: jurgis(a)akl.lt; skype: dz0rdzas; Don't worry, be happy and make things better ;)

2 1

switched dist -> trunk
by Armin Rigo 22 Oct '08

22 Oct '08

Hi all, We finally created the path http://codespeak.net/svn/pypy/trunk, which contains the in-development 2.5-compatible version of PyPy. This closes our many-year-old issue 30 :-) The http://codespeak.net/svn/pypy/dist path is still available, and it's fine to continue checking in there for some more days, but please switch to using 'trunk' for development when convenient. The plan is to merge the changes from 'dist' into 'trunk' in a few days, after which we'll use 'dist' only as a kind of mirror of 'trunk' containing read-only snapshots of stable revisions. A bientot, Armin

1 1

Re: [pypy-dev] [pypy-svn] r59181 - pypy/branch/cbuild-refactor/pypy/translator/platform
by Maciej Fijalkowski 18 Oct '08

18 Oct '08

On Fri, Oct 17, 2008 at 3:19 PM, <afa(a)codespeak.net> wrote: > Author: afa > Date: Fri Oct 17 15:19:27 2008 > New Revision: 59181 > > Modified: > pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py > Log: > Log message about updated environment > > > Modified: pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py > ============================================================================== > --- pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py (original) > +++ pypy/branch/cbuild-refactor/pypy/translator/platform/windows.py Fri Oct 17 15:19:27 2008 > @@ -38,6 +38,7 @@ > key, value = line.split('=', 1) > if key.upper() in ['PATH', 'INCLUDE', 'LIB']: > os.environ[key] = value > + log.msg("Updated environment with %s" % (vcvars,)) > > try: > _install_msvc_env() > _______________________________________________ > pypy-svn mailing list > pypy-svn(a)codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn > Can you please make sure that os.environ is cleaned afterwards? thanks

1 1

Re: [pypy-dev] [sympy] Re: sympy tests: pypy vs cpython incompatibilities
by Ondrej Certik 16 Oct '08

16 Oct '08

On Sat, Oct 11, 2008 at 8:13 PM, didier deshommes <dfdeshom(a)gmail.com> wrote: > > On Sat, Oct 11, 2008 at 1:50 PM, Ondrej Certik <ondrej(a)certik.cz> wrote: >>> as soon as someone realizes that in CPython it is possible to speed up >>> iteration over strings by creating a stringiterator type, then CPython >>> will grow a 'str.__iter__' as well, and the same infinite recursion will >>> occur in sympy... >>> >>> A cleaner way to say "is x iterable?" would be to try to call iter(x) >>> and see if it raises TypeError or not. >> >> Indeed, thanks very much for the tip. We'll fix that, that's >> definitely something that should be fixed in sympy, I created a new >> issue for that: > > Hi, > I've faced the same problem and checking if x has the __getitem__ > attribute has worked for me. The rationale being that any iterable has > a way to access its individual items. Thanks Didier for the tip. Both iter("p") and "p".__getitem__ behaves the same in both python and pypy. But "p".__iter__ behaves differently and our code assumed that "p" is not iterable, so I just implemented a function: http://code.google.com/p/sympy/issues/detail?id=1153#c1 +def is_iterable(x): + try: + iter(x) + if isinstance(x, str) and len(x) == 1: + return False + return True + except TypeError: + return False Which fixes the problem above. Unfortunately, then there are other unrelated problems. Armin, is there some way to tell pypy-c to stop on ctrl-C? I am getting hangups and the only way out is to kill the pypy-c program. It'd be much more convenient if I could just do ctrl-C and it would print the stacktrace. I'll be continuing porting sympy to pypy some other time, so anyone feel free to take over it. You just need to apply the patch in the issue 1153 and then try to make all tests to pass. Thanks, Ondrej

2 1

buildbot problem/misunderstanding.
by Maciej Fijalkowski 15 Oct '08

15 Oct '08

I've been starring at report (here: http://codespeak.net:8099/builders/own-linux-x86-32/builds/16/steps/shell_2…) and I don't understand why there is only one test in translator/c. Am I missing something? (search for translator/c/test/test_support to see what I mean). I'm confused. Cheers, fijal

1 1