[Python-Dev] wpython is back

[Cesare Di Mauro]

Hi Mart

I'm back with some news about wpython. I completed all the work that I was
committed to do till the end of the year. I made a lot of changes to the
code, that I'll report here.

First, I added several conditional compilation sections that enable or
disable almost every optimization I introduced into the project. Everything
is controlled by a new include file, wpython.h, which holds a lot of
#DEFINEs for each one of them.
Every #DEFINE has a brief explanation, and some report an example with
Python code disassembled, showing what happens.
It can be useful both to document the code (also to access to the interested
parts), and to let people test the effect of all optimizations. There are
also a couple of #DEFINEs which are useful to enable or disable all
superinstructions, or to make wpython work like CPython (with all new
optimizations and superinstructions disabled).

Full tracing support required a big effort, due to the missing
SETUP_LOOP/POP_BLOCK instructions used in FOR_ITER blocks. It was a pain in
the neck to let them work, but I think I have found a good solution fot it.
If I remember correctly, Collin asked in the past about performance with
testing enabled. I believe that speed is comparable to CPython, since I can
trace FOR_ITER blocks enter/exit with very little time spent intercepting
them; stack unrolling (for forward jumps cases) is fast too.

Restoring Python object model required much of the work. I reverted all the
changes that I made to many PyObjects, and just added some accessory code
only to a few of them. There are no more hacks, and code is quite polite;
only CodeObject required one line of code change in the hash function, to
let it calculate hash correctly for the constants tuple (because it can hold
lists and dictionaries now, which usally aren't hashable).
Every file in Include/ and Objects/ that I modified has only 1 diff (except
frameobject.c, for tracing code), so it's easy so see what is changed and
the extra helper functions that I added to introduce lists and dictionaries
in the consts tuple.

In the meanwhile I've added a little optimization for lists and dictionaries
used in for loops. Writing this:

def f():
    for x in ['a', 'b', 'c']: print x

generates the following (word)code with the previous wpython:

LOAD_CONST (['a', 'b', 'c'])
DEEP_LIST_COPY
GET_ITER
FOR_ITER

because ['a', 'b', 'c'] is a mutable object, and a copy must be made before
using it.

Now it'll be:

LOAD_CONST (['a', 'b', 'c'])
GET_ITER
FOR_ITER

So code is reduced and memory consumption too, because there's no need clone
the list. The trick works only for lists and dictionaries that holds
non-mutable objects, but I found it's a common pattern in Python code.

I've also updated the source to the latest Python 2.x version, 2.6.4.

All tests pass, both with Debug and Release code, on Visual Studio Express
with 32 bit code (I can't compile 64 bits versions with it).

There are only a few open issues.

test_syntax.py required some changes in the doctest (adding full filesystem
path) to let them pass correctly. It's really strange, but... works now!

test_compile.py has 2 tests disabled in test_compile_ast:

#['<forblock>', """for n in [1, 2, 3]:\n print n\n"""],
#[fname, fcontents],

that's because there's no support for constants (except Num_kind and
Str_kind) in the current ASTs code. However code compiles well, except that
it cannot make use of the new constant folding code.

I haven't updated Doc/library/dis.rst, which is exactly the same of CPython.
I'll do it when I stop introducing or changing opcodes.

Right now wpython requires manual patching of Include/Python-ast.h, with the
following lines:

enum _expr_kind {BoolOp_kind=1, BinOp_kind=2, UnaryOp_kind=3, Lambda_kind=4,
[...]
List_kind=18, Tuple_kind=19, Const_kind=20};

enum _expr_const {no_const=0, mutable_const=1, content_const=3,
pure_const=7};

struct _expr {

    enum _expr_kind kind;
    union {
[...]

        struct {
            object c;
            enum _expr_const constant;
        } Const;

    } v;
    int lineno;
    int col_offset;
};

They are need to let ast.c handle constants for the new constant folding
code.
I greatly appreciate any help to let it be generated automatically with ASDL
grammar.


That's all about the new code. Now the weird and stupid part. A few days I
got a new gmail account, but accidentally I removed the google account that
I've used to create the wpython at Google Code. I definitely lost project
ownership, so I can't tag the old code and put the new one in trunk.
I'll thank very much if someone that works or has contacts with Google can
ask for moving ownership from my old account (cesare at pronto do it) to my
new (the one which I've using now to write this mail), so I'll commit ASAP.
Alternatively, I need to create a new project at Google Code.

I hope that the community will appreciate the work (when I'll upload it :-).
I know that it's a young project, but I think it's mature enough to take a
look at it.
Last but not least, think about it like a starting point. I have many ideas
on how to optimize several other parts of Python, and the wordcode structure
gives me rooms to do it in an elegant and efficient way thanks to the
superinstructions (when needed).

For the next release I plan to cleanup opcode.h and ceval.c, grouping some
instructions into single superinstructions (CALL_FUNCTIONs and IMPORT_NAME),
adding a few opcodes, and tweaking a bit the VM main loop (primarily
targeted to reduce the jump-table that compilers produce for the big switch
statement).

Then I'll consider porting it to python 2.7 and/or python 3.1/3.2 if
there'll be interest and feedbacks about it.

I'm also at your disposal to discuss any detail about wpython source code,
since I know that it isn't a simple patch to apply to some files.

Cheers,
Cesare

2009/11/4 Mart Sõmermaa <mrts.pydev at gmail.com>

> Thanks for the recap and for the good work on wpython!
>
> Best, eagerly waiting for the results of your work to land in mainline
> python,
> MS
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20091126/8108b7b2/attachment-0001.htm>