Mailman 3 August 2007 - pypy-dev

pypy-dev@codespeak.net
by Armin Rigo 25 Sep '07

25 Sep '07

Hi all, Those that follow IRC already know it, but it's worth being announced a bit more widely: I've been working on a form of sandboxing for RPython programs, which now seems to work for the whole of PyPy. It's "sandboxing" as in "full virtualization", but done in normal C with no OS support at all. It's a two-processes model: we can translate PyPy to a special "pypy-c-sandbox" executable, which is safe in the sense that it doesn't do any library or system call - instead, whenever it would like to perform such an operation, it marshals the operation name and the arguments to its stdout and it waits for the marshalled result on its stdin. This pypy-c-sandbox process is meant to be run by an outer "controller" program that answers to these operation requests. The pypy-c-sandbox program is obtained by adding a transformation during translation, which turns all RPython-level external function calls into stubs that do the marshalling/waiting/unmarshalling. An attacker that tries to escape the sandbox is stuck within a C program that contains no external function call at all except to write to stdout and read from stdin. (It's still attackable, e.g. by exploiting segfault-like situations, but as far as I can tell - unlike CPython - any RPython program is really robust against this kind of attack, at least if we enable the extra checks that all RPython list and string indexing are in range. Alternatively, on Linux there is a lightweight OS-level sandboxing technique available by default - google for 'seccomp'.) The outer controller is a plain Python program that can run in CPython or a regular PyPy. It can perform any virtualization it likes, by giving the subprocess any custom view on its world. For example, while the subprocess thinks it's using file handles, in reality the numbers are created by the controller process and so they need not be (and probably should not be) real OS-level file handles at all. In the demo controller I've implemented there is simply a mapping from numbers to file-like objects. The controller answers to the "os_open" operation by translating the requested path to some file or file-like object in some virtual and completely custom directory hierarchy. The file-like object is put in the mapping with any unused number >= 3 as a key, and the latter is returned to the subprocess. The "os_read" operation works by mapping the pseudo file handle given by the subprocess back to a file-like object in the controller, and reading from the file-like object. Enough explanations, here's a how-to: For now, this lives in a branch at the 'pypy' level: cd ..../pypy-dist/pypy svn switch http://codespeak.net/svn/pypy/branch/pypy-more-rtti-inprogress/ In pypy/translator/goal: ./translate.py --sandbox --source targetpypystandalone.py --withoutmod-gc --withoutmod-_weakref (the gc and _weakref modules are disabled because they are a bit too unsafe, in the sense that they could allow bogus memory accesses) Then in the directory where the sources were created, compile with the extra RPython-level assertions enabled: make CFLAGS="-O2 -DRPY_ASSERT" mv testing_1 /some/path/pypy-c-sandbox Run it with the tools in the pypy/translator/sandbox directory: ./pypy_interact.py /some/path/pypy-c-sandbox [args...] Just like pypy-c, if you pass no argument you get the interactive prompt. In theory it's impossible to do anything bad or read a random file on the machine from this prompt. (There is no protection against using all the RAM or CPU yet.) To pass a script as an argument you need to put it in a directory along with all its dependencies, and ask pypy_interact to export this directory (read-only) to the subprocess' virtual /tmp directory with the --tmp=DIR option. Not all operations are supported; e.g. if you type os.readlink('...'), the controller crashes with an exception and the subprocess is killed. Other operations make the subprocess die directly with a "Fatal RPython error". None of this is a security hole; it just means that if you try to run some random program, it risks getting killed depending on the Python built-in functions it tries to call. By the way, as you should have realized, it's really independent from the fact that it's PyPy that we are translating. Any RPython program should do. I've successfully tried it on the JS interpreter. The controller is only called "pypy_interact" because it emulates a file hierarchy that makes pypy-c-sandbox happy - it contains (read-only) virtual directories like /bin/lib-python and /bin/pypy/lib and it pretends that the executable is /bin/pypy-c. A bientot, Armin.

2 4

about breaking RPython, RCTypes
by Martijn Faassen 19 Sep '07

19 Sep '07

Hi there, I posted this to a longer thread the other day but have received no answers to it whatsoever. Perhaps it got lost, so I'll do a repost. I spent a bit of time writing this post, and I ask some questions in it that I think should be answered. Perhaps by telling me I don't understand what I'm talking about or that my analysis is entirely incorrect. Perhaps it was simply too long? Whatever it is, I do think the PyPy community needs to be aware of how technical decisions have an impact on attracting or repelling potential contributors. "answering by silence", which happened so far, is one way to make potential contributors worried. :) I only understand a little of the technical discussion. I understand that RCTypes has problems, and that people anticipate having to break it and reimplement it in a different way in order to solve these problems. I understand that one feature that RCTypes offers, namely the ability to test your RPython code on CPython, might also go away. There are currently, as far as I can see it, two reasons to use RPython for CPython extensions: a) benefit from performance gains by rewriting your CPython code into RPython. This is what we want to do with the page template interpreter. In case of language implementations, this might lead to the ability to use other PyPy technology later (such as JIT generation and so on). Some others might purely be interested in performance though and aren't implementing a language. b) better ways to write extensions that use C libraries in CPython. The typical Pyrex use case, though it can do the former as well. This would use RCTypes. Use case a) probably has less interested people than use case b). Eventually JIT technologies will help here, which will make use case a) less important. I happen to be interested in use case a) though. It is quite exciting to me that we could have a faster template language interpreter implementation that we can use in CPython today, with the potential benefits of PyPy for interpreter translation in the future as well. We are basically language implementors, one of your core target audiences. We just feel more comfortable doing work if we know we have payoffs in the near future, i.e. use in CPython. If we were implementing this template language in isolation and had no hope we could deploy it in production any time soon, we would be less interested. I am actually quite interested in use case b) as well. While developing lxml, I've done quite a bit of work with Pyrex and I can see how RPython and RCTypes might potentially help projects like that. I've been taking a wait and see attitude here though, as the technology needed more maturing. As I understand it, the proposed changes wouldn't affect use case a). Or is the ability to generate CPython extensions also going to be broken due to this work? Please let us know, as if CPython support is going to be broken, we would likely stop our efforts for the time being. Use case b) would certainly be broken due to this work. What's more, I understand that testing your extension code in CPython first wouldn't work anymore (at least if it uses CTypes). This is a very nice feature of RPython that Pyrex doesn't have. You mentioned that there are potential ways to reimplement this feature again, but I'm not sure whether you're committing to such implementation work. If I were writing C extension modules with PyPy I'd like to know things like: * what are the reasons this change is going to be made? * would this benefit just core PyPy or would it benefit extension module writers too? * when do you expect the new way to be finalized? * will you make a real commitment to make the "Test in CPython first" feature work again? Or is did you just sketched out possible ways to implement this in the new system and you hope some contributor is going to do the work? * will you make a commitment to make this new way useful and supported for extension module writers, or should they invest their time in some other technology? I.e. what are the changes a future shift will make me change things again? * How well documented will this be for CPython extension developers? Will you market this feature to developers? Things like this indicate your commitment in another way. :) Please note that I'm not actually *asking* for anything but information. I'm not telling the PyPy developers they should make commitments and do various kinds of work. That's up to you. I'm just trying to guess what other developers might want to know. Of course, the quality of your communication and the answers you actually give will make people decide whether to continue to use PyPy or move on. This will affect the way your open source community will grow and develop. I would urge the PyPy developers to consider investing in the open source community by supporting features like this and promoting them. This would mean serving admittedly self-interested developers with short term goals. The promise of immediate payoff is a great way to attract people to an open source project. If there is the hope of continuing payoff (and PyPy has tremendous potential here), some of those new people can be expected to become valuable contributors to the project as a whole. And yes, I'm self-interested in saying all of this - that's my point. Regards, Martijn

5 5

I'm off for 4 weeks
by Maciek Fijalkowski 31 Aug '07

31 Aug '07

I'll be probably completely offline for like next 4 weeks, starting from monday. Cheers, fijal :.

1 0

PyPy cleanups
by Maciej Fijalkowski 31 Aug '07

31 Aug '07

Hello, This is the list of possibly orphaned parts of pypy. We should consider each item here and think in detail what to do with them. They're mostly broken or not actively maintained. If nobody shows an interest to maintain them, deleting would be the best solution to avoid clutter. Also they pose a maintenance burden when we proceed with the needed large refactorings. Maybe it is also a solution to lazily delete them as they are broken by refactoring if nobody steps up. Of course deleted things can easily be brought back from svn history, if there is renewed interest. So the above may sound scarier than it is. * llvm backend - we need a maintainer for that * pyrex backend (llvm depends on it tough) * js interpreter - needs a lot of work, before we might have usecases for that. * common lisp backend * squeak backend * ext compiler - would need a large rewrite from rctypes to rffi * rctypes (both) - delete as soon as they would not be needed * logic objspace, including constraint libraries Cheers, fijal, Samuele, Simon & Carl Friedrich

4 4

Talked at ESUG
by Armin Rigo 31 Aug '07

31 Aug '07

Hi all, As some of you know I've just given a talk at the ESUG (European Smalltalk User Group) conference, in Lugano. I mostly showed up for my talk only, staying only one night. The talk yesterday late afternoon went well; it was the demo part of the talk we gave at Dyla. Thanks to Roel for pushing us to give a presentation here - it seems quite unusual to have presentations that are given by "outsiders" of Smalltalk, and the organizers suggested to the audiance that I should get an extra applause for daring come and talk :-) Some people had hard about PyPy already, even though it's clear that in the Smalltalk situation the whole approach we took in PyPy is a bit overkill: it's a very stable language that existed and had good VMs for a very long time, with a relatively small core and a lot of things merely written in Smalltalk on top of it. This was the argument I received when I tried to discuss with someone working on XTC, a JIT compiler for Squeak (yet another) - the language is too small to bother with more general techniques, and he might be right for now. (I cannot seem to find his work by googling.) Where I failed to convince him was that PyPy was a possibly better alternative to "plans" like: all Smalltalk-like languages (Self, dialects, etc.) should be implemented in a single VM in a way that allows a single JIT for them all. Non-performance-related benefits of PyPy are also not completely clear for Smalltalk given that the language is more flexible than Python to start with. However, people generally see why we'd like such things for Python-like languages - I think it's a very positive thing. I also believe that presenting the PyPy approach to different communities can contribute just a little bit in the long run to push forward some ideas; for example, people may think twice about Slang or equivalents and see if it could be practical to make these languages just a bit higher level, e.g. move them above the GC. A final note is that talking about PyPy in live seems to be very important. Various people had spend some time looking around our web site because they had heard about PyPy, but mostly failed to understand what we were trying to do. For most talks we have given about PyPy I remember getting some feedback of people that now, at least, had got an overview about what we were doing. A bientot, Armin.

1 0

Re: [pypy-dev] About adding a blist implementation...
by Elmo Mäntynen 30 Aug '07

30 Aug '07

Sorry, I got distracted, and now I have commitments (schoolwork) which prevent me from doing anything moderately involved. And no, I have only started porting the implementation and fixed some general multilist bugs (nothing grave). I will return to all this soon, probably after a month or two... If you need a btree, you'd better do it yourself anyway, and we'll see later if a refactoring is possible. Elmo Simon Burton wrote: > Elmo! > did you get anywhere with this ?? > > Simon. > > On Fri, 22 Jun 2007 17:10:44 +0300 > Elmo Mäntynen <elmo13(a)jippii.fi> wrote: > > >> Leonardo Santagada wrote: >> >>> Em 21/06/2007, às 22:10, Simon Burton escreveu: >>> >>> >>> >>>> On Tue, 19 Jun 2007 11:18:59 +0300 >>>> Elmo Mäntynen <elmo13(a)jippii.fi> wrote: >>>> >>>> >>>> >>>>> Since I'm adding a list implementation, I'd like to know what >>>>> benchmarks >>>>> have you used to test the other ones. Also, if you know apps with >>>>> heavy >>>>> usage of lists and some that use very long lists, I'd be >>>>> interested very >>>>> much. >>>>> >>>>> >>>> Can you explain what this blist is ? Is it a btree ? I think I need >>>> an rpython btree. >>>> >>>> Simon. >>>> >>>> >>> I dunno but this guy is probably talking about something like this >>> http://mail.python.org/pipermail/python-3000/2007-May/thread.html#7127 >>> >>> If it is not related I would also like to understand the diferences. >>> >>> -- >>> Leonardo Santagada >>> >>> Sent from my iPhone >>> >>> >>> >> Hi. >> It is an list implementation that has much better performance with very >> big lists, available as an extension to CPython, and soon included in >> pypy by me. >> This cheeseshop entry is a good place to start: >> http://www.python.org/pypi/blist/ >> >> The thread might be about integrading it into CPython or something, >> haven't read. >> >> It is based on btree, but the implementation I'm working on is based on >> the python prototype included in the tar distrib, which won't probably >> be useful if you specifically need a btree. In the future, maybe some >> refactoring would be useful, but for now I'm happy with the way I started. >> >> I'd still be interested in ways to test the different list >> implementations =) >> >> Elmo >> _______________________________________________ >> pypy-dev(a)codespeak.net >> http://codespeak.net/mailman/listinfo/pypy-dev >>

1 0

buffer class and ll2ctypes problem
by Simon Burton 29 Aug '07

29 Aug '07

I've been working with Richard on a new (rpython) buffer class. See attached for two approaches. The richbuf seems much clearer, but i'm not sure how to get other types out of it (like long long or short), or do byte swapping. I don't understand this much, can someone point the right direction ? Ie. which buf is the right direction to go. (I would also like to re-visit the array type in pypy/rpython/numpy, and perhaps think more generally about __getitem__ in rpython.) The rffi buffer (buffers.py) fails on CPython: $ python buffers.py Traceback (most recent call last): File "buffers.py", line 55, in <module> test_run() File "buffers.py", line 51, in test_run assert demo(1234) == 1234 File "buffers.py", line 39, in demo buf.put_int32(value) File "buffers.py", line 31, in put_int32 intp = rffi.cast(INTP, buf) File "/home/simon/site-packages/pypy/rpython/lltypesystem/ll2ctypes.py", line 491, in force_cast cvalue = lltype2ctypes(value) File "/home/simon/site-packages/pypy/rpython/lltypesystem/ll2ctypes.py", line 336, in lltype2ctypes convert_struct(container) File "/home/simon/site-packages/pypy/rpython/lltypesystem/ll2ctypes.py", line 161, in convert_struct convert_struct(parent) File "/home/simon/site-packages/pypy/rpython/lltypesystem/ll2ctypes.py", line 165, in convert_struct cstruct = cls._malloc() File "/home/simon/site-packages/pypy/rpython/lltypesystem/ll2ctypes.py", line 89, in _malloc raise TypeError, "array length must be an int" TypeError: array length must be an int Simon.

1 1

translation broken in pypy-more-rtti-inprogress branch
by Jean-Paul Calderone 24 Aug '07

24 Aug '07

Just a heads up :) Here's the traceback I get: [translation:ERROR] Error: [translation:ERROR] Traceback (most recent call last): [translation:ERROR] File "./translate.py", line 272, in main [translation:ERROR] drv.proceed(goals) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/translator/driver.py", line 748, in proceed [translation:ERROR] return self._execute(goals, task_skip = self._maybe_skip()) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/translator/tool/taskengine.py", line 112, in _execute [translation:ERROR] res = self._do(goal, taskcallable, *args, **kwds) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/translator/driver.py", line 265, in _do [translation:ERROR] res = func() [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/translator/driver.py", line 334, in task_rtype_lltype [translation:ERROR] crash_on_first_typeerror=insist) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 180, in specialize [translation:ERROR] self.specialize_more_blocks() [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 226, in specialize_more_blocks [translation:ERROR] self.specialize_block(block) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 345, in specialize_block [translation:ERROR] self.translate_hl_to_ll(hop, varmapping) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 474, in translate_hl_to_ll [translation:ERROR] resultvar = hop.dispatch() [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 712, in dispatch [translation:ERROR] return translate_meth(self) [translation:ERROR] File "None</home/exarkun/Projects/pypy/trunk/py/code/source.py:213>", line 5, in translate_op_delitem [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/lltypesystem/rdict.py", line 319, in rtype_delitem [translation:ERROR] return hop.gendirectcall(ll_dict_delitem, v_dict, v_key) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 750, in gendirectcall [translation:ERROR] return self.llops.gendirectcall(ll_function, *args_v) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/rtyper.py", line 908, in gendirectcall [translation:ERROR] rtyper.lowlevel_ann_policy) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/rpython/annlowlevel.py", line 103, in annotate_lowlevel_helper [translation:ERROR] return annotator.annotate_helper(ll_function, args_s, policy) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 129, in annotate_helper [translation:ERROR] self.complete_helpers(policy) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 175, in complete_helpers [translation:ERROR] self.complete() [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 249, in complete [translation:ERROR] self.processblock(graph, block) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 475, in processblock [translation:ERROR] self.flowin(graph, block) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 535, in flowin [translation:ERROR] self.consider_op(block.operations[i]) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 737, in consider_op [translation:ERROR] raise_nicer_exception(op, str(graph)) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/annrpython.py", line 734, in consider_op [translation:ERROR] resultcell = consider_meth(*argcells) [translation:ERROR] File "<string>", line 3, in consider_op_call_args [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/unaryop.py", line 169, in call_args [translation:ERROR] return obj.call(getbookkeeper().build_args("call_args", args_s)) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/unaryop.py", line 582, in call [translation:ERROR] return bltn.analyser(*args_s, **kwds_s) [translation:ERROR] File "/home/exarkun/Projects/pypy/trunk/pypy/annotation/builtin.py", line 418, in malloc [translation:ERROR] assert (s_n is None or s_n.knowntype == int [translation:ERROR] AssertionError': [translation:ERROR] .. v1 = call_args((function malloc), ((2, ('zero',), False, False)), v0, new_size_0, (True)) [translation:ERROR] .. '(pypy.rpython.lltypesystem.rdict:475)ll_dict_resize__dicttablePtr' [translation:ERROR] Processing block: [translation:ERROR] block@97 is a <class 'pypy.objspace.flow.flowcontext.SpamBlock'> [translation:ERROR] in (pypy.rpython.lltypesystem.rdict:475)ll_dict_resize__dicttablePtr [translation:ERROR] containing the following operations: [translation:ERROR] v2 = simple_call((function typeOf), old_entries_0) [translation:ERROR] v0 = getattr(v2, ('TO')) [translation:ERROR] v1 = call_args((function malloc), ((2, ('zero',), False, False)), v0, new_size_0, (True)) [translation:ERROR] v3 = setattr(d_0, ('entries'), v1) [translation:ERROR] v4 = setattr(d_0, ('num_items'), (0)) [translation:ERROR] v5 = setattr(d_0, ('num_pristine_entries'), new_size_0) [translation:ERROR] --end-- [translation] start debugger... > /home/exarkun/Projects/pypy/trunk/pypy/annotation/builtin.py(418)malloc() -> assert (s_n is None or s_n.knowntype == int (Pdb+) This is from r45970 Jean-Paul

1 0

away for two weeks
by Carl Friedrich Bolz 24 Aug '07

24 Aug '07

Hi all, I will be away from home (and probably from Internet) for about two weeks, starting from Sunday morning. Cheers, Carl Friedrich

1 0

rffi strict typing
by Simon Burton 24 Aug '07

24 Aug '07

The lltype's are very strict about types. If I declare an llexternal that takes a Float arg, I cannot call it with an int arg. Well maybe that's not so bad. But in rlib.rsdl there are functions that take Unsigned, and these don't work even if called with a non-negative constant int... Can we loosen up these restrictions ? Maybe put an "is_compatable(self, other)" method on LowLevelType's ? Simon. using py lib: /home/simon/local/pypy-latest/py <rev 45205> test/test_SDL.py[2] FF __________________________________________________________________________________________________________________________________________________________________________________________________________ entrypoint: test_run ____________________________________________________________ def test_run(): > demo() [/home/simon/local/pypy-latest/pypy/rlib/rsdl/test/test_SDL.py:48] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def demo(): > if SDL.Init(SDL.INIT_VIDEO) < 0: [/home/simon/local/pypy-latest/pypy/rlib/rsdl/test/test_SDL.py:13] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ def __call__(self, *args): if isinstance(self._T, FuncType): if len(args) != len(self._T.ARGS): raise TypeError,"calling %r with wrong argument number: %r" % (self._T, args) for a, ARG in zip(args, self._T.ARGS): if typeOf(a) != ARG: E raise TypeError,"calling %r with wrong argument types: %r" % (self._T, args) > TypeError: calling <Func ( Unsigned ) -> Signed> with wrong argument types: (32,) ...

3 7