Mailman 3 September 2007 - pypy-dev

pypy with sympy
by Ondrej Certik 16 Oct '07

16 Oct '07

Hi, since pypy is in Debian for some time, I sometimes test it with our project SymPy[1], written in pure Python 2.4 and newer. Generally, it works out of the box and it's very nice and quite fast. You did an awesome job! However, sometimes, it fails, because of the bug[2], that we reported two months ago. Do you think you could please apply the patch provided in there, or just fix it somehow? Also another question - what is your plan with your RPython to javascript converter - are you planning to create something like the Google Web Toolkit, or just leave it as is, if someone becomes interested? I think it would be a very cool feature to have something like this in Python, just a little more documented and officially released. Thanks a lot, Ondrej Certik [1] http://code.google.com/p/sympy/ [2] https://codespeak.net/issue/pypy-dev/issue316

3 10

Gothenburg Sprint Announcement
by Carl Friedrich Bolz 29 Sep '07

29 Sep '07

PyPy Göteborg Cleanup Sprint November 19 - November 25 2007 =========================================================== The next post-EU-project PyPy sprint will be in Gothenburg, Sweden. It will focus on cleaning up the PyPy codebase and making it ready for the next round of improvements. It is a "public" sprint but it will probably be more suitable for people already somewhat acquainted with PyPy. Topics and goals ---------------- Meta-Tasks ++++++++++ - Tracker-gardening - Start slowly thinking about 1.1 plans Translation toolchain +++++++++++++++++++++ - finish rctypes removal - move away from implicit keepalives - raw_malloc should be split, rffi one should not use obmalloc (it's not thread-safe) - kill simulator, run the semi space tests on llinterp (may happen before) - have a moving gc correct version of the gc framework transformation - delegate finding type stuff like vtables etc to GC, cleaner interface for rtti, simplify translator/c/gc.py - think about approaches to id, especially concerning boehm, where the id will keep the object alive and concerning a moving GC - clean up the tangle of including headers in the C backend - review pdbplus, especially the graph commands, also in the light of https://codespeak.net/issue/pypy-dev/issue303 and the fact that we can have more than one translator/annotator around (with the timeshifter) - Kill half concrete wrapper - Kill opaque hacks on the C backend - decide how to implement constants in rffi - think about/remove orphaned parts - finish ll2ctypes, especially rffi_platform Interpreter +++++++++++ - there's a shadow tracking bug it seems - fix the taint space - review the things implemented at applevel whether they are performance-critical JIT +++ - start writing real unit tests for the merging logic Location -------- The sprint will be held in the apartment of Laura Creighton and Jacob Hallén which is at Götabergsgatan 22 in Gothenburg, Sweden. Here is a map_. This is in central Gothenburg. It is between the tram_ stops of Vasaplatsen and Valand, where many lines call. .. _tram: http://www.vasttrafik.se .. _map: http://gulasidorna.eniro.se/query?stq=0&streetname=G%F6tabergsgatan+22%2C+G… Probably cheapest and not too far away is to book accomodation at `SGS Veckobostader`_. There are also hotels to suit every price range and desire for luxury. Just ask in the mailing list. .. _`SGS Veckobostader`: http://www.sgsveckobostader.com Registration ------------ If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be send to that list. We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have codespeak commit rights). .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`people`: http://codespeak.net/pypy/extradoc/sprintinfo/gothenburg-2007/people.html

1 0

PyPy/Squeak Sprint Announcement
by Carl Friedrich Bolz 27 Sep '07

27 Sep '07

Bern Squeak/PyPy sprint October 22 - October 26 2007 ===================================================== We are proud to announce the first Squeak-PyPy-collaboration-sprint. The plan is to bring people from the Squeak/Smalltalk and the PyPy/Python communities together to learn about each others projects and to think about collaboration possibilies. The sprint will be hosted by the Software Composition Group of the University of Bern from the 22nd to the 26th of October 2007. Topics and goals ---------------- The general technical goal of the sprint is to explore implementing the bytecode interpreter as well as the primitives of Squeak in RPython. This should make it eventually possible to auto-generate a JIT for Squeak with the help of PyPy technologies. The other important goal of the sprint is to teach the first-time-sprinters the sprint-driven-development methodology. The first day of the sprint will be a tutorial day, were talks about PyPy, Squeak and sprint-driven-development will be given. The further days will be sprinting days. Of course the most important goal of the sprint is to get to know each other and have some fun :-). Topics of the sprint: - learning about: - PyPy - the internals of Squeaks VM - sprint-driven-development - exploring ways to implement Squeaks bytecode-interpreter and the primitives in Python - or (maybe alternatively) implement a Squeak-backend for PyPy - think about how to load and store of images in PyPy Location -------- The sprint will take place in a seminar room at the University of Bern. Further details of how to get there will be announced before the sprint. Registration ------------ If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be send to that list. We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have codespeak commit rights). If you have no clue where to stay in Bern ask around on the list, maybe we can help you find something or find some other solution. .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`people`: http://codespeak.net/pypy/extradoc/sprintinfo/bern2007/people.html

2 1

Fwd: New on list
by OvermindDL1 27 Sep '07

27 Sep '07

I'm from the USA as well, although I tend to be extremely quiet on here... I'm hoping for PyPy to be a faster Python mostly (would love the exporting to C++), especially with stackless functionality. I know the CPython internals quite well, so if there is anywhere I can use that knowledge in here, feel free to ask, short on time but of what little I have I can give up to this. EDIT: Helps if I send to the right list... Too many too similar lists... On 9/26/07, Paul deGrandis <paul.degrandis(a)gmail.com> wrote: > > > On 9/26/07, Brandon Lewis <brandon_lewis(a)berkeley.edu> wrote: > > Hi, my name is Brandon Lewis. I'm an undergrad Cognitive Science major > > at U.C. Berkeley. I'm currently taking the upper-division compilers class. > > > Finally I'm not the only person from the United > States! :) > Hi Brandon and welcome! I'm a BS/MS student at > Drexel University. My BS is in Software > Engineering. My MS is in Computer Science with my thesis on Dependable > Systems and Autonomic Computing. > > Our final project will be writing a JavaScript interpreter. > > Interestingly enough, we are using Python as the implementation > > language; so, when I discovered this project while looking through some > > python documentation, I was intrigued. > > It sounds like you have a great professor! Enjoy the class. > > Paul > > > Mostly just wanted to introduce myself. > > > > --Brandon Lewis > > _______________________________________________ > > pypy-dev(a)codespeak.net > > http://codespeak.net/mailman/listinfo/pypy-dev > > > > > _______________________________________________ > pypy-dev(a)codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev >

1 0

Replies to list?
by Brandon Lewis 27 Sep '07

27 Sep '07

Just wondering about how this list handles replies. If I get a reply off-list should I reply to it on-list? --Brandon

3 2

New on list
by Brandon Lewis 27 Sep '07

27 Sep '07

Hi, my name is Brandon Lewis. I'm an undergrad Cognitive Science major at U.C. Berkeley. I'm currently taking the upper-division compilers class. Our final project will be writing a JavaScript interpreter. Interestingly enough, we are using Python as the implementation language; so, when I discovered this project while looking through some python documentation, I was intrigued. Mostly just wanted to introduce myself. --Brandon Lewis

6 8

Re: [pypy-dev] PyPy cleanups
by Richard Emslie 27 Sep '07

27 Sep '07

Hi! Sorry it has taken forever to reply... > > This is the list of possibly orphaned parts of pypy. > We should consider each item here and think in detail > what to do with them. They're mostly broken or not > actively maintained. If nobody shows an interest > to maintain them, deleting would be the best solution > to avoid clutter. Also they pose a maintenance burden > when we proceed with the needed large refactorings. > Maybe it is also a solution to lazily delete them as > they are broken by refactoring if nobody steps up. > > Of course deleted things can easily be brought back from > svn history, if there is renewed interest. So the above > may sound scarier than it is. > > * llvm backend - we need a maintainer for that As much I as I agree that deleting code is better than leaving it to rot beyond all recognition, I'm still not sure why we need a "maintainer" for this given that no other backend has an explicit maintainer. But anyways I will take that role if it needs be. Most work to be done is in factoring code out of genc and making rffi work (which I am guessing making changes to rffi, but not sure since haven't been following things for last however many months). > * pyrex backend (llvm depends on it tough) There is no longer this dependency. Cheers, Richard

2 1

rpython numpy
by Simon Burton 26 Sep '07

26 Sep '07

I've been having some fun working on a (tiny) rpython version of numpy. At the heart of numpy is a multi-dimensional array type, ndarray. For example it (the cpython version) can handle processing image arrays like so: >>> import numpy >>> from PIL import Image >>> im=Image.open('jit.png') >>> data=numpy.array(im) now data is a 3 dimensional array, data.ndim==3 and data.shape==(1158, 1173, 3). The last dimension is the color dimension. The cool part of numpy is all the indexing tricks. This is a single pixel: >>> data[0,0,:] array([224, 255, 224], dtype=uint8) this is the red channel: >>> data[:,:,0] array([[224, 224, 224, ..., 224, 224, 224], ... now we mix red and blue: >>> data[:,:,0]+data[:,:,2] (this actually didn't work so well because it's an array of uint8 and we get overflow, but that can be fixed with a cast) The trick with numpy is, every time we use an indexing operation it creates a "view" onto the original data, ie. the memory is aliased. Each ndarray stores a list of "strides" (integers) which specify how many bytes to skip when incrementing an index in a dimension: >>> data.strides (3519, 3, 1) >>> data[:,:,0].strides (3519, 3) Then numpy (at the c level) uses iterators to perform getattr, setattr, and other (pointwise) operations. "broadcasting" allows a smaller array to be used as a larger array, as long as the shape of the smaller array is a suffix of the shape of the larger. >>> green = numpy.array([0,255,0], dtype=numpy.uint8) >>> data[:] = green This sets every pixel to green. I made a design choice with rnumpy to specialize on the number of dimensions (ndim) of an array. This means that we can generate efficient code (more about this below); which was just too tempting to ignore. But also, * it would be impossible to access arbitrary ndim arrays from a pypy wrapped rnumpy. * the shape attribute is read-only. To change the shape of an array, use the reshape method (this does not copy the data): >>> numpy.array(range(12)).reshape((3,4)) array([[ 0, 1, 2, 3], [ 4, 5, 6, 7], [ 8, 9, 10, 11]]) The above mentioned "efficient code" is perhaps a non-issue. Numpy code tends to become memory-bound, not cpu-bound. However it is an interesting use of the rpython optimiser. In general, to iterate over an n-dimensional array requires n-nested for loops, or the use of a "coordinate" array. In this implementation (as in the c implementation) there is an iterator object to hold the coordinates (and the corresponding strides). With malloc removal, all this iterator data become local variables. It's nifty. I don't really know where this is all going; an obvious goal would be to extend all this with a variable dimension array type (and write an applevel numpy). One of the original ideas was to somehow produce cache friendly code by doing lazy operations. Another issue is that it has been mind-boggleingly difficult to directly extend the compiler. I would like to rethink this whole approach; perhaps special methods can help, or some other "low-level to high-level" interaction. Simon.

2 1

pypy-dev@codespeak.net
by Armin Rigo 25 Sep '07

25 Sep '07

Hi all, Those that follow IRC already know it, but it's worth being announced a bit more widely: I've been working on a form of sandboxing for RPython programs, which now seems to work for the whole of PyPy. It's "sandboxing" as in "full virtualization", but done in normal C with no OS support at all. It's a two-processes model: we can translate PyPy to a special "pypy-c-sandbox" executable, which is safe in the sense that it doesn't do any library or system call - instead, whenever it would like to perform such an operation, it marshals the operation name and the arguments to its stdout and it waits for the marshalled result on its stdin. This pypy-c-sandbox process is meant to be run by an outer "controller" program that answers to these operation requests. The pypy-c-sandbox program is obtained by adding a transformation during translation, which turns all RPython-level external function calls into stubs that do the marshalling/waiting/unmarshalling. An attacker that tries to escape the sandbox is stuck within a C program that contains no external function call at all except to write to stdout and read from stdin. (It's still attackable, e.g. by exploiting segfault-like situations, but as far as I can tell - unlike CPython - any RPython program is really robust against this kind of attack, at least if we enable the extra checks that all RPython list and string indexing are in range. Alternatively, on Linux there is a lightweight OS-level sandboxing technique available by default - google for 'seccomp'.) The outer controller is a plain Python program that can run in CPython or a regular PyPy. It can perform any virtualization it likes, by giving the subprocess any custom view on its world. For example, while the subprocess thinks it's using file handles, in reality the numbers are created by the controller process and so they need not be (and probably should not be) real OS-level file handles at all. In the demo controller I've implemented there is simply a mapping from numbers to file-like objects. The controller answers to the "os_open" operation by translating the requested path to some file or file-like object in some virtual and completely custom directory hierarchy. The file-like object is put in the mapping with any unused number >= 3 as a key, and the latter is returned to the subprocess. The "os_read" operation works by mapping the pseudo file handle given by the subprocess back to a file-like object in the controller, and reading from the file-like object. Enough explanations, here's a how-to: For now, this lives in a branch at the 'pypy' level: cd ..../pypy-dist/pypy svn switch http://codespeak.net/svn/pypy/branch/pypy-more-rtti-inprogress/ In pypy/translator/goal: ./translate.py --sandbox --source targetpypystandalone.py --withoutmod-gc --withoutmod-_weakref (the gc and _weakref modules are disabled because they are a bit too unsafe, in the sense that they could allow bogus memory accesses) Then in the directory where the sources were created, compile with the extra RPython-level assertions enabled: make CFLAGS="-O2 -DRPY_ASSERT" mv testing_1 /some/path/pypy-c-sandbox Run it with the tools in the pypy/translator/sandbox directory: ./pypy_interact.py /some/path/pypy-c-sandbox [args...] Just like pypy-c, if you pass no argument you get the interactive prompt. In theory it's impossible to do anything bad or read a random file on the machine from this prompt. (There is no protection against using all the RAM or CPU yet.) To pass a script as an argument you need to put it in a directory along with all its dependencies, and ask pypy_interact to export this directory (read-only) to the subprocess' virtual /tmp directory with the --tmp=DIR option. Not all operations are supported; e.g. if you type os.readlink('...'), the controller crashes with an exception and the subprocess is killed. Other operations make the subprocess die directly with a "Fatal RPython error". None of this is a security hole; it just means that if you try to run some random program, it risks getting killed depending on the Python built-in functions it tries to call. By the way, as you should have realized, it's really independent from the fact that it's PyPy that we are translating. Any RPython program should do. I've successfully tried it on the JS interpreter. The controller is only called "pypy_interact" because it emulates a file hierarchy that makes pypy-c-sandbox happy - it contains (read-only) virtual directories like /bin/lib-python and /bin/pypy/lib and it pretends that the executable is /bin/pypy-c. A bientot, Armin.

2 4

llvm backend status?
by Rahul Garg 20 Sep '07

20 Sep '07

Hi. I recently discovered PyPy and its one cool project in which I would love to be involved in some capacity :) So I have a bunch of questions about the direction and status of the project a) How is the LLVM backend coming along? b) What really are the focus areas of the project currently? c) Any wishlists ? thanks, rahul

2 2