Mailman 3 June 2007 - pypy-dev

about breaking RPython, RCTypes
by Martijn Faassen Sept. 19, 2007

Sept. 19, 2007

Hi there, I posted this to a longer thread the other day but have received no answers to it whatsoever. Perhaps it got lost, so I'll do a repost. I spent a bit of time writing this post, and I ask some questions in it that I think should be answered. Perhaps by telling me I don't understand what I'm talking about or that my analysis is entirely incorrect. Perhaps it was simply too long? Whatever it is, I do think the PyPy community needs to be aware of how technical decisions have an impact on attracting or repelling potential contributors. "answering by silence", which happened so far, is one way to make potential contributors worried. :) I only understand a little of the technical discussion. I understand that RCTypes has problems, and that people anticipate having to break it and reimplement it in a different way in order to solve these problems. I understand that one feature that RCTypes offers, namely the ability to test your RPython code on CPython, might also go away. There are currently, as far as I can see it, two reasons to use RPython for CPython extensions: a) benefit from performance gains by rewriting your CPython code into RPython. This is what we want to do with the page template interpreter. In case of language implementations, this might lead to the ability to use other PyPy technology later (such as JIT generation and so on). Some others might purely be interested in performance though and aren't implementing a language. b) better ways to write extensions that use C libraries in CPython. The typical Pyrex use case, though it can do the former as well. This would use RCTypes. Use case a) probably has less interested people than use case b). Eventually JIT technologies will help here, which will make use case a) less important. I happen to be interested in use case a) though. It is quite exciting to me that we could have a faster template language interpreter implementation that we can use in CPython today, with the potential benefits of PyPy for interpreter translation in the future as well. We are basically language implementors, one of your core target audiences. We just feel more comfortable doing work if we know we have payoffs in the near future, i.e. use in CPython. If we were implementing this template language in isolation and had no hope we could deploy it in production any time soon, we would be less interested. I am actually quite interested in use case b) as well. While developing lxml, I've done quite a bit of work with Pyrex and I can see how RPython and RCTypes might potentially help projects like that. I've been taking a wait and see attitude here though, as the technology needed more maturing. As I understand it, the proposed changes wouldn't affect use case a). Or is the ability to generate CPython extensions also going to be broken due to this work? Please let us know, as if CPython support is going to be broken, we would likely stop our efforts for the time being. Use case b) would certainly be broken due to this work. What's more, I understand that testing your extension code in CPython first wouldn't work anymore (at least if it uses CTypes). This is a very nice feature of RPython that Pyrex doesn't have. You mentioned that there are potential ways to reimplement this feature again, but I'm not sure whether you're committing to such implementation work. If I were writing C extension modules with PyPy I'd like to know things like: * what are the reasons this change is going to be made? * would this benefit just core PyPy or would it benefit extension module writers too? * when do you expect the new way to be finalized? * will you make a real commitment to make the "Test in CPython first" feature work again? Or is did you just sketched out possible ways to implement this in the new system and you hope some contributor is going to do the work? * will you make a commitment to make this new way useful and supported for extension module writers, or should they invest their time in some other technology? I.e. what are the changes a future shift will make me change things again? * How well documented will this be for CPython extension developers? Will you market this feature to developers? Things like this indicate your commitment in another way. :) Please note that I'm not actually *asking* for anything but information. I'm not telling the PyPy developers they should make commitments and do various kinds of work. That's up to you. I'm just trying to guess what other developers might want to know. Of course, the quality of your communication and the answers you actually give will make people decide whether to continue to use PyPy or move on. This will affect the way your open source community will grow and develop. I would urge the PyPy developers to consider investing in the open source community by supporting features like this and promoting them. This would mean serving admittedly self-interested developers with short term goals. The promise of immediate payoff is a great way to attract people to an open source project. If there is the hope of continuing payoff (and PyPy has tremendous potential here), some of those new people can be expected to become valuable contributors to the project as a whole. And yes, I'm self-interested in saying all of this - that's my point. Regards, Martijn

5 5

scheme interpreter [status report]
by Jakub Gustak Aug. 17, 2007

Aug. 17, 2007

Hello. Scheme interpreter is starting look like its going to do some serious stuff in the near future, or maybe I am just too enthusiastic about it. What we have now: - We can execute simple expressions like: (+ 1 2 1.3 -2) Nested ones also works. - (define <var> <value>) works with a given context, so: (define var 2) (+ var 40) => 42 - Conditional (if <cond> <then> <else>) seem to work fine, <else> is not arbitrary. pypy/lang/scheme/test/test_eval.py Gives you the best overview. What will be done in the nearest future: - comparison primitives: = < > eq? eqv? - (cons <car> <cdr>) (car <pair>) (cdr <pair>) so we can play with lists. That kind of stuff. You can take a look at: pypy/lang/scheme/TODO.txt Will postpone quotations and symbols implementation for a while. I have to think about it, how much of it is parsing matter. Then i would like to work on (lambda ...). This will probably take some time to think again execution context, make a hard distinction between procedures and macros and a lot of stuff, that I don't even dare to think about now :-). Before EP I would like to make lambdas work and also have some tuning done (whatever it means), so on the sprint we could focus on really interesting stuff instead of refactoring and bug fixes. I would like to refine what I would like to do during the sprint, but not sure right now. Any ideas, what would be worth trying then? Cheers, Jakub Gustak

4 10

Fwd: Work plan for PyPy
by Richard Emslie June 30, 2007

June 30, 2007

waaaah - forget to cc everyone :-( Begin forwarded message: > From: Richard Emslie <rxe(a)ukshells.co.uk> > Date: June 27, 2007 10:56:39 AM PDT > To: Armin Rigo <arigo(a)tunes.org> > Subject: Re: [pypy-dev] Work plan for PyPy > > Hi Armin! > > On Jun 27, 2007, at 4:40 AM, Armin Rigo wrote: > >> Hi Richard, >> >> On Mon, Jun 25, 2007 at 01:01:35PM -0700, Richard Emslie wrote: >>>> Just a sidenote - rffi supports (and uses) macros. Not sure how >>>> this >>>> will look like in a ctypes-based solution. >>> >>> if you are talking c macros - this could be very problematic for the >>> llvm backend. >> >> Anton (which I add to the CC's of this mail) proposed a solution for >> calling C APIs from llvm bytecode in a portable way, which would work >> for functions whose argument types are not exactly specified by the >> standards as well as for some kind of macros. The idea is to >> generate >> stub "helper" functions in C and compile them with llvm-gcc. The >> llvm >> bytecode that genllvm produces would then call the helper functions >> instead of calling directly the external C API. Shouldn't be a >> performance problem as llvm will inline the helpers agressively. >> > > ok, that is somewhat what we have now - except the stub functions > are high level and shared *somewhat* with genc. > > Generating c code in llvm backend would sort of defeat the point of > it IMHO. We already have a backend that generates c. On the other > hand, if we hand write the c code - it may be an neverending task > of adding hacks to the backend. For instance just getting at errno > is particularly problematic right now - and would involve a > function call from a constant value. which means injecting a few > lines assembly and a temporary variable. Easy to do, but it is > nice right now that we pretty much have a direct mapping from our > our lli form to llvm assembly. > > >> Of course, it points again to the question of whether it's really >> worthwhile to have an llvm backend in PyPy or if just producing C and >> using llvm-gcc on everything wouldn't work just as well. On the >> other >> hand, it's interesting to note that there are other reasons for which >> exactly the same kind of helper functions would be useful; for >> example, >> to wrap a C library full of macros and unspecified types and >> constants, >> ctypes alone is not enough. > > > Like you say, llvm-gcc (well at least v4) would probably produce > equivalent results with genc. The way we are using llvm right now > (as a static compiler) i'm not sure of its worthiness either of > keeping the backend alive. Other than the potential of accurate > gc, potential of jit (which is sort of orthogonal to this), > potential of producing vectorized code from rpython... > > > Cheers, > Richard > > > > > >

3 5

Re: [pypy-dev] [pypy-svn] r44599 - in pypy/dist/pypy/lang/scheme: . test
by Carl Friedrich Bolz June 30, 2007

June 30, 2007

Hi Jacub! 2007/6/28, jlg(a)codespeak.net <jlg(a)codespeak.net>: > Author: jlg > Date: Thu Jun 28 15:49:30 2007 > New Revision: 44599 > > Modified: > pypy/dist/pypy/lang/scheme/TODO.txt > pypy/dist/pypy/lang/scheme/object.py > pypy/dist/pypy/lang/scheme/test/test_parser.py > Log: > RPython direction for object.py > > Modified: pypy/dist/pypy/lang/scheme/object.py > ============================================================================== > --- pypy/dist/pypy/lang/scheme/object.py (original) > +++ pypy/dist/pypy/lang/scheme/object.py Thu Jun 28 15:49:30 2007 > @@ -139,10 +139,16 @@ > raise NotImplementedError > > def add_lst(ctx, lst): > - return apply_lst(ctx, lambda x, y: x + y, lst) > + def adder(x, y): > + return x + y > + > + return apply_lst(ctx, adder, lst) > > def mul_lst(ctx, lst): > - return apply_lst(ctx, lambda x, y: x * y, lst) > + def multiplier(x, y): > + return x * y > + > + return apply_lst(ctx, multiplier, lst) > > def apply_lst(ctx, fun, lst): > acc = None > This doesn't make mul_lst and add_lst any more rpython than before. In fact, lambda is fully rpython. The problem with this code is rather that you cannot have local functions in RPython. As I said, don't worry about making your code RPython too much yet. Cheers, Carl Friedrich

4 3

Re: [pypy-dev] Work plan for PyPy
by Richard Emslie June 28, 2007

June 28, 2007

Hi Anton, On Jun 27, 2007, at 11:07 PM, Anton Korobeynikov wrote: > Hello, Richard. > >> Generating c code in llvm backend would sort of defeat the point of >> it IMHO. We already have a backend that generates c. On the other >> hand, if we hand write the c code - it may be an neverending task >> of adding hacks to the backend. For instance just getting at errno >> is particularly problematic right now - and would involve a >> function call from a constant value. which means injecting a few >> lines assembly and a temporary variable. Easy to do, but it is >> nice right now that we pretty much have a direct mapping from our >> our lli form to llvm assembly. ^^^ ahem, "ssi form" > I've talked a little bit with Chris Lattner about this (calling C API) > issue. Except stubs/wrappers it seems there are ways to call C API > > 1. Use more or less portable runtime, which will "unify" such > calls. For > example, one can use APR (HLVM, in fact, will use it) > > 2. Delegate stubs resolution to specialised version of lli. I think, > LLVM's itself OS abstraction library (libSystem) can be partly used. > thanks for these ideas. i guess Armin has replied already. I am guess the biggest concern is how to wrap libraries outside what these support. > I personally like the second variant more. Anyway, some specialised > lli > variant (or some application, which will call LLVM's execution engine) > will be needed, if pypy would want some extended features (like > accurate > GC, vectorisation, etc) from LLVM in the future. > > We never managed to run pypy under lli. In was one of the early goals, but it were waiting until gc to mature in the pypy infrastructure. Since that has happened, the llvm backend hasn't had much active development. Right now it only works with boehm, and not surprisingly the interaction of llvm's jit and boehm doesn't go very well in hand. I still think it would be fun and interesting to see pypy-llvm working under lli. Cheers, Ricahrd

1 0

ep 2007 talks
by Simon Burton June 28, 2007

June 28, 2007

I'd like to synchronize with Maciek and Antonio with regards talking about rpython at EP this year. My talk comes right after your rpython talk - i think we can perhaps share the burden (joy?) of talking about rpython a bit. Simon.

3 3

Work plan for PyPy
by Armin Rigo June 28, 2007

June 28, 2007

Hi all, Here is a summary of what we discussed today on #pypy-sync. We tried to list the tasks that now lie ahead of us. There are a number of relatively independent tasks, but there is also a knot of dependent tasks that we'll have to untie bit by bit. RCtypes and the extension compiler are somewhat of a blocker for other tasks. A controversial solution is to drop them. This means: * To support calling external libraries, we would instead shift the focus on supporting the full ctypes at app-level. This means that we could support out of the box the same ctypes-based pure Python modules as CPython. There are issues here (e.g. is the regular ctypes really suitable to wrap large libraries?) but they are not worse than with rctypes. * Of course, we keep the general support for compiling stand-alone RPython programs, even if rctypes dies. * In RPython, fijal worked on a more basic replacement for rctypes, called rtti (almost finished already). It's a PyPy-specific interface to call external functions in RPython. Calls written in this interface can be compiled to good C code, but testing is a bit less convenient than with rctypes because rtti calls cannot be directly executed on top of CPython. * The CPy Object Space is going away initially too because it is based on rctypes. It should not be too hard to port it to rtti in order to continue supporting compiling extension modules for CPython. We need to think a bit more here - how can we compile ext modules for Jython and IronPython? - but it's likely not a high priority at the moment. Any comments or objections? Are there people that already make heavy use of rctypes, or the extension compiler? The graph of dependent tasks ("A --- B" means that B depends on A): merge rtti branch ---- app-level ctypes ---- kill rctypes and extcompiler \ / \ \-- port our modules --- kill keepalives \ \ -- pyrepl -- moving GCs / _______________________________/ / / malloc cleanup ----- gc transform polishing / \ fix weakrefs -----/ \ \ -- c/llvm backend sharing Additionally, some of the independent tasks mentioned: * move some app-level parts of the Python interpreter to interp-level * complete the support for Python 2.5 * switch to LLVM 2.0 * port the stackless transform to ootypesystem * JIT work A bientot, Armin.

11 25

Re: [pypy-dev] Fwd: Work plan for PyPy
by Armin Rigo June 28, 2007

June 28, 2007

Hi Anton, On Thu, Jun 28, 2007 at 10:07:27AM +0400, Anton Korobeynikov wrote: > 1. Use more or less portable runtime, which will "unify" such calls. For > example, one can use APR (HLVM, in fact, will use it) > > 2. Delegate stubs resolution to specialised version of lli. I think, > LLVM's itself OS abstraction library (libSystem) can be partly used. Unless I'm missing something, it seems that neither of these approaches would work to call generic C APIs - only the ones that are re-exported from either the APR or libSystem. Doesn't sound too attractive from my point of view. A bientot, Armin.

1 0

Some bugfixes to the multilistimplementation
by Elmo Mäntynen June 25, 2007

June 25, 2007

Hi. I've made the different [slow] list implementations even slower ! While integrating the blist implementation, I found out that the chosen implementation isn't even used when instantiating with "list()", and neither in some other usecases. By correcting that, a bug in the chunked implementation got uncovered, which I also fixed. I also added "make_implementation_with_one_item()" to have the instantiation of the implementations in one place so that it's easier to add new ones. If they seem good, could someone commit these changes (is that diff format good?). Would you trust me with commit rights, or should I still send patches? The test demonstrates the bugs, but I thougth of later adding a BaseAppTest_ListMultiObject class with magic to test if the implementations that the subclasses test are really used. Does that sound good? Index: objspace/std/listmultiobject.py =================================================================== --- objspace/std/listmultiobject.py (revision 44498) +++ objspace/std/listmultiobject.py (working copy) @@ -12,7 +12,7 @@ # An empty list always is an EmptyListImplementation. # -# RDictImplementation -- standard implementation +# RListImplementation -- standard implementation # StrListImplementation -- lists consisting only of strings # ChunkedListImplementation -- when having set the withchunklist option # SmartResizableListImplementation -- when having set the @@ -87,7 +87,7 @@ for i in range(slicelength): res_w[i] = self.getitem(start) start += step - return RListImplementation(self.space, res_w) + return make_implementation(self.space, res_w) def delitem_slice_step(self, start, stop, step, slicelength): n = self.length() @@ -368,8 +368,8 @@ length = self._length self._grow() - for j in range(length - 1, 0, -1): - self.setitem(j + 1, self.getitem(j)) + for j in range(length, i, -1): + self.setitem(j, self.getitem(j-1)) self.setitem(i, w_item) return self @@ -399,13 +399,7 @@ class EmptyListImplementation(ListImplementation): def make_list_with_one_item(self, w_item): space = self.space - if space.config.objspace.std.withfastslice: - return SliceTrackingListImplementation(space, [w_item]) - w_type = space.type(w_item) - if space.is_w(w_type, space.w_str): - strlist = [space.str_w(w_item)] - return StrListImplementation(space, strlist) - return RListImplementation(space, [w_item]) + return make_implementation_with_one_item(space, w_item) def length(self): return 0 @@ -834,6 +828,27 @@ else: return RListImplementation(space, list_w) +def make_implementation_with_one_item(space, w_item): + if space.config.objspace.std.withfastslice: + return SliceTrackingListImplementation(space, [w_item]) + if space.config.objspace.std.withsmartresizablelist: + from pypy.objspace.std.smartresizablelist import \ + SmartResizableListImplementation + impl = SmartResizableListImplementation(space) + impl.append(w_item) + return impl + if space.config.objspace.std.withchunklist: + return ChunkedListImplementation(space, [w_item]) + if space.config.objspace.std.withblist: + from pypy.objspace.std.blistimplementation import \ + BListImplementation + return BListImplementation(space, [w_item]) + w_type = space.type(w_item) + if space.is_w(w_type, space.w_str): + strlist = [space.str_w(w_item)] + return StrListImplementation(space, strlist) + return RListImplementation(space, [w_item]) + def convert_list_w(space, list_w): if not list_w: impl = space.fromcache(State).empty_impl @@ -884,7 +899,7 @@ if w_iterable is not EMPTY_LIST: list_w = space.unpackiterable(w_iterable) if list_w: - w_list.implementation = RListImplementation(space, list_w) + w_list.implementation = make_implementation(space, list_w) return w_list.implementation = space.fromcache(State).empty_impl @@ -1304,9 +1319,8 @@ sorterclass = CustomKeySort else: sorterclass = SimpleSort - impl = w_list.implementation.to_rlist() - w_list.implementation = impl - items = impl.list_w + impl=w_list.implementation + items = impl.get_list_w() sorter = sorterclass(items, impl.length()) sorter.space = space sorter.w_cmp = w_cmp @@ -1349,8 +1363,7 @@ mucked = w_list.implementation.length() > 0 # put the items back into the list - impl.list_w = sorter.list - w_list.implementation = impl + w_list.implementation = make_implementation(space, sorter.list) if mucked: raise OperationError(space.w_ValueError, Index: objspace/std/test/test_listmultiobject.py =================================================================== --- objspace/std/test/test_listmultiobject.py (revision 44498) +++ objspace/std/test/test_listmultiobject.py (working copy) @@ -144,3 +144,15 @@ def teardown_class(cls): _set_chunk_size_bits(cls.chunk_size_bits) + def test_chunklist_is_implementation_used(self): + import __pypy__ + l = ["1", "2", "3", "4", "5"] + assert "ChunkedListImplementation" in __pypy__.internal_repr(l) + l = list(["1", "2", "3", "4", "5"]) + assert "ChunkedListImplementation" in __pypy__.internal_repr(l) + l=[] + l.append('a') + assert "ChunkedListImplementation" in __pypy__.internal_repr(l) + l = ["6", "8", "3", "1", "5"] + l.sort() + assert "ChunkedListImplementation" in __pypy__.internal_repr(l)

2 1

lltypesystem typecache
by Scott Dial June 25, 2007

June 25, 2007

Greetings, The typecache in subversion repository is missing the 32bit windows platform that I am running the windows tests against, and on occasion (like today) I noticed it caused a update conflict due to Armin's changes in r44398. If someone could add this one, thanks. -Scott Index: pypy/rpython/lltypesystem/typecache.py =================================================================== --- pypy/rpython/lltypesystem/typecache.py (revision 44403) +++ pypy/rpython/lltypesystem/typecache.py (working copy) @@ -1,6 +1,7 @@ # this is automatically generated cache files for c types platforms = { ('', ('32bit', 'ELF'), 'Linux'):{'short': 16, 'int': 32, 'unsigned char': 8, 'long': 32, 'char': 8, 'unsigned short': 16, 'unsigned long': 32, 'long long': 64, 'mode_t': 32, 'unsigned long long': 64, 'size_t': 32, 'unsigned int': 32}, +('', ('32bit', 'WindowsPE'), 'Windows'):{'short': 16, 'int': 32, 'unsigned char': 8, 'long': 32, 'char': 8, 'unsigned short': 16, 'unsigned long': 32, 'long long': 64, 'unsigned long long': 64, 'size_t': 32, 'unsigned int': 32}, ('i386', ('32bit', ''), 'Darwin'):{'short': 16, 'int': 32, 'unsigned char': 8, 'long': 32, 'char': 8, 'unsigned short': 16, 'unsigned long': 32, 'long long': 64, 'mode_t': 16, 'unsigned long long': 64, 'size_t': 32, 'unsigned int': 32}, ('Intel(R) Pentium(R) M processor 1.73GHz', ('32bit', ''), 'Linux'):{'short': 16, 'int': 32, 'unsigned char': 8, 'long': 32, 'char': 8, 'unsigned short': 16, 'unsigned long': 32, 'long long': 64, 'mode_t': 32, 'unsigned long long': 64, 'size_t': 32, 'unsigned int': 32} } -- Scott Dial scott(a)scottdial.com scodial(a)cs.indiana.edu

4 4