[pypy-dev] rffi and rpython layer consistency

Christian Tismer tismer at stackless.com
Tue Oct 16 14:52:54 CEST 2007

[a lengthy brain dump from last night which was meant for pypy-dev]
[today I would be much more concise, but well, it is like it is]

Dear Friends,

I am talking about different layers of RPython
and how coherently they respond. ATM, they don't.

I am writing because I feel the need for it.
We (you) have reached quite some state with PyPy
and RPython.

At the current state, I digged in and worked with
the windows branch of Maciey's rmmap implementation.

It went quite smooth, until I followed some XXX saying
that the native windows types should be used.

Not sure if they should be used, but I wanted to be
consistent, and to try a show-case. And this caused me grief!

Internal reasons for the trouble

Unfortunately, there are different flavors of the windows
API. Most of them are consistent, having an agreement on
what a HANDLE should be, for instance.
But there are lower-level, also public functions with no
general equivalent, which simply define a Handle as an int.

This is not nice, but given by the gods of MS.

Now I wanted to use exactly those inconsistent declarations
with RFFI. Exactly!

Why did I want this?

Well, ideally an extension writer should not bother about
any types, just use them and pass them around as necessary.

But there is this inconsistency. In order to make functions
compatible, I tried to insert casts, a bad thing, I know.
But how are casts worse that modifying an original definition?
I wanted to provide a clean interface, as clean as can be,
with no modifications. But rffi stopped me doing this.

Putting it short:

I don't want to have cast restrictions in llinterp, which don't
exist in ctypes and low level modes.

After trying this for hours and already days, I came up with the
following limitations of our type system:

1) things are running fine on top of cpython, with the ctypes
    emulation in place

2) things do not run well with llinterp in place.
    llinterp has grown into a large handler of different
    cases, no objection, but it does not do what it should do
    in this particular case.

3) things in a full translation are behaving like expected.
    We should move away from a full trabslation requirement
    as a proof of concept.

What I can agree with would be a strong hierarchy, getting more
restrictive as we become more specific. But the current situation
just feels badly inconsistent and unsorted.

rffi has a cast operation, which is implemented by force_cast.
This should be unconditional. For every mode of interpretation
or translatition. But it is not.

Good but then bad:
Using ctypes, everything works great. This should be the point
of trust: it ctypes takes it, there should be no more testing

Worse, much worse:
llinterp barfs at me, moaning that it is only able to cast an odd
int. Hey why that!!!

Crazy, in the end: I can produce a perfect extension to PyPy.

Yes, I know, this was a contribution for testing tagged pointers.
I do hate this argument. This is not PyPy. We do not implement
something that is needed in some place. It is either explicitly
enabled and used in a controlled manner, or it is disabled.

So what I'm saying is this:

We should not support casting to any odd pointers at all, unless
we are doing a tagged pointer implementation.
By default, these operations should be hidden.

If we do so, then we should emit a special operation for all pointers in
that domain, and it should not be value dependent, but type dependent,
only for that domain (whatever a domain should be, in this context).

We also should not support casting to any pointer at all but
for good reasons. If we have to cast, then for inconsistent, given
annotations. That is just the case for rffi!
I just want it there -- rffi.cast does a cast. Maybe restricted
to raw, no idea. But it should be there, and only there!


This is what I'd like to do.

- make pointer coercion functions as restrictive as possible

- drop the llinterp odd pointer rule; instead

- implement tagging exactly for those kinds of pointers where we want
   to support pointer tagging

- make casting rules consistent through all layers of abstraction.
   If interpreted pypy allows/forbids casts, llinterp must adhere
   to it, and backends must follow the rules, too.

- testing should have a defined hierarchy of things that may break
   when moving from one layer to the other. Ideally, they all would
   be the same. But they should be monotone:
   If a passes level 3, it also should pass level 1 and 2.

- I would like to add tests for these meta-test levels, if seems

Yes, this was what I wanted to communicate since days.
I'm open to work on these things, while I hope that these issues are
not just my problem, after some thought ouf your's. You will know.

G'nite -- chris

Christian Tismer             :^)   <mailto:tismer at stackless.com>
tismerysoft GmbH             :     Have a break! Take a ride on Python's
Johannes-Niemeyer-Weg 9A     :    *Starship* http://starship.python.net/
14109 Berlin                 :     PGP key -> http://wwwkeys.pgp.net/
work +49 30 802 86 56  mobile +49 173 24 18 776  fax +49 30 80 90 57 05
PGP 0x57F3BF04       9064 F4E1 D754 C2FF 1619  305B C09C 5A3B 57F3 BF04
       whom do you want to sponsor today?   http://www.stackless.com/

More information about the Pypy-dev mailing list