Mailman 3 July 2007 - pypy-dev

about breaking RPython, RCTypes
by Martijn Faassen Sept. 20, 2007

Sept. 20, 2007

Hi there, I posted this to a longer thread the other day but have received no answers to it whatsoever. Perhaps it got lost, so I'll do a repost. I spent a bit of time writing this post, and I ask some questions in it that I think should be answered. Perhaps by telling me I don't understand what I'm talking about or that my analysis is entirely incorrect. Perhaps it was simply too long? Whatever it is, I do think the PyPy community needs to be aware of how technical decisions have an impact on attracting or repelling potential contributors. "answering by silence", which happened so far, is one way to make potential contributors worried. :) I only understand a little of the technical discussion. I understand that RCTypes has problems, and that people anticipate having to break it and reimplement it in a different way in order to solve these problems. I understand that one feature that RCTypes offers, namely the ability to test your RPython code on CPython, might also go away. There are currently, as far as I can see it, two reasons to use RPython for CPython extensions: a) benefit from performance gains by rewriting your CPython code into RPython. This is what we want to do with the page template interpreter. In case of language implementations, this might lead to the ability to use other PyPy technology later (such as JIT generation and so on). Some others might purely be interested in performance though and aren't implementing a language. b) better ways to write extensions that use C libraries in CPython. The typical Pyrex use case, though it can do the former as well. This would use RCTypes. Use case a) probably has less interested people than use case b). Eventually JIT technologies will help here, which will make use case a) less important. I happen to be interested in use case a) though. It is quite exciting to me that we could have a faster template language interpreter implementation that we can use in CPython today, with the potential benefits of PyPy for interpreter translation in the future as well. We are basically language implementors, one of your core target audiences. We just feel more comfortable doing work if we know we have payoffs in the near future, i.e. use in CPython. If we were implementing this template language in isolation and had no hope we could deploy it in production any time soon, we would be less interested. I am actually quite interested in use case b) as well. While developing lxml, I've done quite a bit of work with Pyrex and I can see how RPython and RCTypes might potentially help projects like that. I've been taking a wait and see attitude here though, as the technology needed more maturing. As I understand it, the proposed changes wouldn't affect use case a). Or is the ability to generate CPython extensions also going to be broken due to this work? Please let us know, as if CPython support is going to be broken, we would likely stop our efforts for the time being. Use case b) would certainly be broken due to this work. What's more, I understand that testing your extension code in CPython first wouldn't work anymore (at least if it uses CTypes). This is a very nice feature of RPython that Pyrex doesn't have. You mentioned that there are potential ways to reimplement this feature again, but I'm not sure whether you're committing to such implementation work. If I were writing C extension modules with PyPy I'd like to know things like: * what are the reasons this change is going to be made? * would this benefit just core PyPy or would it benefit extension module writers too? * when do you expect the new way to be finalized? * will you make a real commitment to make the "Test in CPython first" feature work again? Or is did you just sketched out possible ways to implement this in the new system and you hope some contributor is going to do the work? * will you make a commitment to make this new way useful and supported for extension module writers, or should they invest their time in some other technology? I.e. what are the changes a future shift will make me change things again? * How well documented will this be for CPython extension developers? Will you market this feature to developers? Things like this indicate your commitment in another way. :) Please note that I'm not actually *asking* for anything but information. I'm not telling the PyPy developers they should make commitments and do various kinds of work. That's up to you. I'm just trying to guess what other developers might want to know. Of course, the quality of your communication and the answers you actually give will make people decide whether to continue to use PyPy or move on. This will affect the way your open source community will grow and develop. I would urge the PyPy developers to consider investing in the open source community by supporting features like this and promoting them. This would mean serving admittedly self-interested developers with short term goals. The promise of immediate payoff is a great way to attract people to an open source project. If there is the hope of continuing payoff (and PyPy has tremendous potential here), some of those new people can be expected to become valuable contributors to the project as a whole. And yes, I'm self-interested in saying all of this - that's my point. Regards, Martijn

5 5

rffi strict typing
by Simon Burton Aug. 24, 2007

Aug. 24, 2007

The lltype's are very strict about types. If I declare an llexternal that takes a Float arg, I cannot call it with an int arg. Well maybe that's not so bad. But in rlib.rsdl there are functions that take Unsigned, and these don't work even if called with a non-negative constant int... Can we loosen up these restrictions ? Maybe put an "is_compatable(self, other)" method on LowLevelType's ? Simon. using py lib: /home/simon/local/pypy-latest/py <rev 45205> test/test_SDL.py[2] FF __________________________________________________________________________________________________________________________________________________________________________________________________________ entrypoint: test_run ____________________________________________________________ def test_run(): > demo() [/home/simon/local/pypy-latest/pypy/rlib/rsdl/test/test_SDL.py:48] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def demo(): > if SDL.Init(SDL.INIT_VIDEO) < 0: [/home/simon/local/pypy-latest/pypy/rlib/rsdl/test/test_SDL.py:13] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __call__(self, *args): if isinstance(self._T, FuncType): if len(args) != len(self._T.ARGS): raise TypeError,"calling %r with wrong argument number: %r" % (self._T, args) for a, ARG in zip(args, self._T.ARGS): if typeOf(a) != ARG: E raise TypeError,"calling %r with wrong argument types: %r" % (self._T, args) > TypeError: calling <Func ( Unsigned ) -> Signed> with wrong argument types: (32,) ...

3 7

scheme interpreter [status report]
by Jakub Gustak Aug. 18, 2007

Aug. 18, 2007

Hello. Scheme interpreter is starting look like its going to do some serious stuff in the near future, or maybe I am just too enthusiastic about it. What we have now: - We can execute simple expressions like: (+ 1 2 1.3 -2) Nested ones also works. - (define <var> <value>) works with a given context, so: (define var 2) (+ var 40) => 42 - Conditional (if <cond> <then> <else>) seem to work fine, <else> is not arbitrary. pypy/lang/scheme/test/test_eval.py Gives you the best overview. What will be done in the nearest future: - comparison primitives: = < > eq? eqv? - (cons <car> <cdr>) (car <pair>) (cdr <pair>) so we can play with lists. That kind of stuff. You can take a look at: pypy/lang/scheme/TODO.txt Will postpone quotations and symbols implementation for a while. I have to think about it, how much of it is parsing matter. Then i would like to work on (lambda ...). This will probably take some time to think again execution context, make a hard distinction between procedures and macros and a lot of stuff, that I don't even dare to think about now :-). Before EP I would like to make lambdas work and also have some tuning done (whatever it means), so on the sprint we could focus on really interesting stuff instead of refactoring and bug fixes. I would like to refine what I would like to do during the sprint, but not sure right now. Any ideas, what would be worth trying then? Cheers, Jakub Gustak

4 10

A quick question
by Maciek Fijalkowski July 30, 2007

July 30, 2007

Can we move jit tests to the end of nightly test run? It'll help quick-checking whether there are no breakages spread all around. Cheers, fijal :.

1 0

partial evaluation links
by Maciek Fijalkowski July 28, 2007

July 28, 2007

not sure, if this is of any interest: http://partial-eval.org/pe-impl.html list of partial evaluation implementations (mostly scheme, but also C). Cheers, fijal :.

1 0

Design question
by Maciek Fijalkowski July 26, 2007

July 26, 2007

I'm approaching "how to make cache failures not crash on you every time you import anything". So in short, file like ll_os.py looks right now: if cache.defined('XXX'): TYPE = cache.inttype(...) register_external(...) Idea is that if any of the functions involved in creation of external (cache functions...), show a warning and than create a register_external stub which will explode if referenced (by ie a specific extregistry entry). The question is how it should look like than. My idea is: class Os(registeringclass): def register_os_uname(...): ... def register_os_other(...): ... and so on. This will create huge indentation. Any other ideas? Cheers, fijal :.

1 0

Re: [pypy-dev] [pypy-svn] r45269 - in pypy/dist/pypy/rpython/module: . tes
by Amaury Forgeot d'Arc July 25, 2007

July 25, 2007

Hello again, > Author: fijal > Date: Mon Jul 23 13:52:26 2007 > New Revision: 45269 > > Modified: > pypy/dist/pypy/rpython/module/ll_os.py > pypy/dist/pypy/rpython/module/test/test_posix.py > Log: > Add os.uname. RPython level. segfaults ll2ctypes in some strange way, > I'm not sure I want to know > > > Modified: pypy/dist/pypy/rpython/module/ll_os.py > ============================================================================== > --- pypy/dist/pypy/rpython/module/ll_os.py (original) > +++ pypy/dist/pypy/rpython/module/ll_os.py Mon Jul 23 13:52:26 2007 > @@ -110,6 +110,34 @@ > register_external(os.setsid, [], int, export_name="ll_os.ll_os_setsid", > llimpl=setsid_lltypeimpl) > > +# ------------------------------- os.uname ------------------------------ > + > +if hasattr(os, 'uname'): > + UTSNAMEP = rffi.CStruct('utsname', ('sysname', rffi.CCHARP), > + ('nodename', rffi.CCHARP), > + ('release', rffi.CCHARP), > + ('version', rffi.CCHARP), > + ('machine', rffi.CCHARP), > + ('stuff', rffi.CCHARP)) I think the error comes from the ctypes interpretation of the structure. utsname is not made of char* pointers, but (at least on the debian machine I have access to) the members are actually fixed arrays of chars, the lengths of which are not public, but well specified in sys/utsname.h. It should make no difference when translated, because the code looks like v->sysname etc. ctypes on the other hand needs to know the precise sizeof and offsets of each member. I tried to modify the _socket module to also use rffi, and had the same problem. I think that we will need a tool similar to pypy.rpython.rctypes.tool.ctypes_platform, which generates and compiles C code to get the different sizes and offsets. (We will need support for C defines as well) Hope this helps, -- Amaury Forgeot d'Arc

4 5

sprint report
by Carl Friedrich Bolz July 25, 2007

July 25, 2007

Hi all! Since I didn't manage to come to Europython and the sprint afterwards, I would really appreciate it if somebody wrote a sprint report. Since I guess that is kind of unlikely to happen now, could at least everybody write a paragraph about what he worked on? Cheers, Carl Friedrich

6 5

Re: [pypy-dev] Proposal of timeline
by Antonio Cuni July 23, 2007

July 23, 2007

Hi Paul, I'm cc-ing pypy-dev. Really, don't be shy and don't worry to bother the list with your questions. They are not spam. I'm sure that the others are happy to hear your questions and also to answer in case I can't do it immediately (as happened in the last weekend). Paul deGrandis wrote: > Antonio, > > I've been working on the JVM backend. When I trace the float tests that > look at uint handling, weird NEGATIVE values pop up. I originally > associated these with overflow, but it's a different issue. Any > insight? The way to do unsigned values in Java is to use the next > primary type larger, for example, a uint is stored in a double... but > currently no checks exist for negative values. as Niko pointed out unsigned types are stored in the corresponding signed types in the JVM: this is ok becasue they occupy the same amount of bits, but you have to been careful when dealing with them because the JVM thinks you want to represent a signed value. In particular, if you try to print one to stdout it will be printed as a signed, and so this might be the reason why you see negative values. You have to write a custom procedure to format unsigned values into the right human readable implementation. Please note that this is necessary only during testing or when doing I/O: as long as your unsigned values are kept inside the JVM, they works just fine (apart from comparisons). The simplest way I can think to format an unsigned into an human readable form is to convert it into a double, as I already wrote in a previous email: double from_uint(int x) { double res = x; if (x < 0) // i.e., the leftmost but is 1 res += 4294967294; return res; } But again, you should use this *only* for I/O and testing, not for doing any real operation. > I looked at the test_overflow but these tests don't make sense for the > jvm. You have overflow built in to CLI so you want to make sure that > ovf_check from the rpython definition works, but the JVM defines these > operations. What exactly are these tests aiming to illustrate and test? RPython is a language with a (more or less :-)) well defined semantics, so every backend must take care of implementing this semantics correctly, even if it's not natively supported by the underlying platform. In this particular case, RPython programs expect overflow to be detected correctly. If the JVM does not do this for you, you have to handle those cases by yourself. GenC has the same problem: the C platform does not detect any overflow, so it must explicitly check for that; you can easily reuse the same logic; for example, this macro is taken from translator/c/src/int.h: #define OP_INT_ADD_OVF(x,y,r) \ OP_INT_ADD(x,y,r); \ if ((r^(x)) >= 0 || (r^(y)) >= 0); \ else FAIL_OVF("integer addition") for genjvm, you could e.g. write a java function doing the same (and raising the appropriate exception in case of overflow). > I haven't tackled overriding default values (as skipped in test_class) > yet, but that's really all there is needed left to pass the test_class > requirements I think you could skip this for now. IIRC it's only needed to translate pypy with the thunk objspace, it's not a priority. > I'm still working on the others, but I don't think I'll have trouble > having these first bugs fixed by the end of the work day on the 27th. ok, that's cool! :-) > Lastly, what is the easy approach to I/O and external functions? the easy approach is to forward the external functions to the backend, which will take care of rendering them as appropriate (usually by calling an helper function written in java). This approach is already used to implement time.time, time.clock and os.write: you can find their implementation in jvm/src/pypy/PyPy.java (ll_time_time, ll_time_clock, ll_os_write). The code that calls these functions is in jvm/generator.py, method 'call_primitive'. ciao Anto

2 1

Strange annotation problem.
by Maciek Fijalkowski July 23, 2007

July 23, 2007

Discovered by exarkun. following diff breaks translation :. Index: ../../module/posix/__init__.py =================================================================== --- ../../module/posix/__init__.py (revision 45246) +++ ../../module/posix/__init__.py (working copy) @@ -83,6 +83,9 @@ # interpleveldefs['uname'] = 'interp_posix.uname' if hasattr(os, 'ttyname'): interpleveldefs['ttyname'] = 'interp_posix.ttyname' + if hasattr(os, 'setsid'): + interpleveldefs['xxx'] = 'interp_posix.xxx' + for name in w_star: if hasattr(os, name): interpleveldefs[name] = 'interp_posix.' + name Index: ../../module/posix/interp_posix.py =================================================================== --- ../../module/posix/interp_posix.py (revision 45246) +++ ../../module/posix/interp_posix.py (working copy) @@ -499,6 +499,22 @@ raise OperationError(space.w_TypeError, space.wrap(msg)) utime.unwrap_spec = [ObjSpace, str, W_Root] +def xxx(space): + """xxx() -> pid + + Stuff + """ + return 3 +xxx.unwrap_spec = [ObjSpace] + +#def setsid(space): +# """setsid() -> pid +# +# Creates a new session with this process as the leader. +# """ +# return 3 +#setsid.unwrap_spec = [ObjSpace] + def declare_new_w_star(name): if name in w_star_returning_int: def WSTAR(space, status):

2 2