Mailman 3 September 2007 - pypy-dev

transactional memory
by Simon Burton 20 Sep '07

20 Sep '07

Someone on pypy irc posted a link to a talk by Simon Peyton-Jones on transactional memory: http://oscon.blip.tv/file/317758/ it's worth checking out. Also, i noticed on the llvm list someone has added support for TM: http://lists.cs.uiuc.edu/pipermail/llvmdev/2007-August/010365.html Simon.

1 0

about breaking RPython, RCTypes
by Martijn Faassen 19 Sep '07

19 Sep '07

Hi there, I posted this to a longer thread the other day but have received no answers to it whatsoever. Perhaps it got lost, so I'll do a repost. I spent a bit of time writing this post, and I ask some questions in it that I think should be answered. Perhaps by telling me I don't understand what I'm talking about or that my analysis is entirely incorrect. Perhaps it was simply too long? Whatever it is, I do think the PyPy community needs to be aware of how technical decisions have an impact on attracting or repelling potential contributors. "answering by silence", which happened so far, is one way to make potential contributors worried. :) I only understand a little of the technical discussion. I understand that RCTypes has problems, and that people anticipate having to break it and reimplement it in a different way in order to solve these problems. I understand that one feature that RCTypes offers, namely the ability to test your RPython code on CPython, might also go away. There are currently, as far as I can see it, two reasons to use RPython for CPython extensions: a) benefit from performance gains by rewriting your CPython code into RPython. This is what we want to do with the page template interpreter. In case of language implementations, this might lead to the ability to use other PyPy technology later (such as JIT generation and so on). Some others might purely be interested in performance though and aren't implementing a language. b) better ways to write extensions that use C libraries in CPython. The typical Pyrex use case, though it can do the former as well. This would use RCTypes. Use case a) probably has less interested people than use case b). Eventually JIT technologies will help here, which will make use case a) less important. I happen to be interested in use case a) though. It is quite exciting to me that we could have a faster template language interpreter implementation that we can use in CPython today, with the potential benefits of PyPy for interpreter translation in the future as well. We are basically language implementors, one of your core target audiences. We just feel more comfortable doing work if we know we have payoffs in the near future, i.e. use in CPython. If we were implementing this template language in isolation and had no hope we could deploy it in production any time soon, we would be less interested. I am actually quite interested in use case b) as well. While developing lxml, I've done quite a bit of work with Pyrex and I can see how RPython and RCTypes might potentially help projects like that. I've been taking a wait and see attitude here though, as the technology needed more maturing. As I understand it, the proposed changes wouldn't affect use case a). Or is the ability to generate CPython extensions also going to be broken due to this work? Please let us know, as if CPython support is going to be broken, we would likely stop our efforts for the time being. Use case b) would certainly be broken due to this work. What's more, I understand that testing your extension code in CPython first wouldn't work anymore (at least if it uses CTypes). This is a very nice feature of RPython that Pyrex doesn't have. You mentioned that there are potential ways to reimplement this feature again, but I'm not sure whether you're committing to such implementation work. If I were writing C extension modules with PyPy I'd like to know things like: * what are the reasons this change is going to be made? * would this benefit just core PyPy or would it benefit extension module writers too? * when do you expect the new way to be finalized? * will you make a real commitment to make the "Test in CPython first" feature work again? Or is did you just sketched out possible ways to implement this in the new system and you hope some contributor is going to do the work? * will you make a commitment to make this new way useful and supported for extension module writers, or should they invest their time in some other technology? I.e. what are the changes a future shift will make me change things again? * How well documented will this be for CPython extension developers? Will you market this feature to developers? Things like this indicate your commitment in another way. :) Please note that I'm not actually *asking* for anything but information. I'm not telling the PyPy developers they should make commitments and do various kinds of work. That's up to you. I'm just trying to guess what other developers might want to know. Of course, the quality of your communication and the answers you actually give will make people decide whether to continue to use PyPy or move on. This will affect the way your open source community will grow and develop. I would urge the PyPy developers to consider investing in the open source community by supporting features like this and promoting them. This would mean serving admittedly self-interested developers with short term goals. The promise of immediate payoff is a great way to attract people to an open source project. If there is the hope of continuing payoff (and PyPy has tremendous potential here), some of those new people can be expected to become valuable contributors to the project as a whole. And yes, I'm self-interested in saying all of this - that's my point. Regards, Martijn

5 5

Re: [pypy-dev] pypy with sympy
by Laura Creighton 19 Sep '07

19 Sep '07

In a message of Sat, 15 Sep 2007 15:17:34 +0200, "Ondrej Certik" writes: <snip> >So what is your priority now with pypy? I think it has such an >enormous potential, but currently for me it's just a playground. Do >you plan to apply pypy in some area, where people will use it as the >best technology available in there? Or do you plan more to have pypy >just as a means of testing new JIT ideas? > >Ondrej We really want to push this as a real language that people should use to solve real problems they have. Currently, however, things are in a bad need of a cleanup. While it was terrific to have the EU funding which let many of us work on pypy full time, the deadlines for deliverables were such that many things were hacked together in ways that are ok for a prototype but not really robust enough for production use. We are planning our next sprint to work on the roughest of these. Then, of course, the problem is that there are so many things that could be the 'next step' for PyPy. Some of us (me for instance) are actively looking for paying customers who have a business case for wanting PyPy to do X. Tim Parkin, who designed the current python.org site is also interested in working on finding out what should be the next step for PyPy. He is collecting opinions, and people who are willing to put money where their mouth is here: http://wiki.python.org/moin/PyPyDonations It would be nice to get PyPy to such a state that it again had many tasks for those people who don't feel up to hacking the Just in Time Specialiser. A better garbage collector is necessary, but I think that people find that intimidating as well. We're much further along with the new scheme for working with C extensions, and that too is necessary. It would be nice to find out exactly who wants what next from PyPy, and perhaps more importantly, who would be willing to help on PyPy, but cannot now, and what they would need in the way of changes before they could contribute. Besides fixes for the bugs you have reported, is there anything in particular you wish we were doing next? Laura

4 6

Stackless PyPy mini sprint 2007
by Christian Tismer 18 Sep '07

18 Sep '07

Hi Stackless PyPy people. We have been sprinting once, quite a long time, ago. Time to start over, again. I'm thinking of doing a mini sprint, before the PyPy cleanup-sprint. It should be a last sprint of clarifying issues which are not resolved, yet, concerning completeness of app-level basic Stackless implementations, and it should lead to an almost final application of the scalability goals stated in PyPy. Maybe these statements need to be adjusted as well. Effectively, I'd like to target this as the final sprint before transforming results into RPython. As a wish, RPython implementations might me tackled as well. But my primary intent is to have a very complete, well-tested code-base for app-level stackless, including composability. I will give a very intense introductory talk about this in the very beginning. Everybody will conceive composability as a simple concept which is not really new, but becomes new by introducing coroutines as the underlying structure for everything. This is non-intrusive, it just needs to be understood. The sprint can be done in my house, with a limit of about five external people, free food and accommodation included. I think a period of three days would be appropriate. Please give me timing proposals and different opinions as you see fit. Sincerely -- chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

4 9

Re: [pypy-dev] pypy with sympy
by Adam Olsen 17 Sep '07

17 Sep '07

> 4) Myabe allow to speed my code up (JIT), or make Python as fast as > possible. But this is rather a long term goal. Generally, I don't see > why my code written in C and the same code written in Python, not > using much dynamic features, couldn't be the same fast. (Of course I > understand why now it is slower, Python is interpreted, etc. But I > mean in principle. Python can have as much information about my code > as the C compiler.) I use Python for devising new numerical > algorithms, and than I need to rewrite them by hand to fortran for > speed. It's a pain. > 6) Allow efficient threading? CPython has the GIL (for good > technological reasons). Generally pypy can improve how to do things in > parallel in python. Currently, one needs to use pympi, pypar, or > something. These two together are, in my mind, where PyPy can really shine. Performance on par with CPython is critical to be taken seriously, but CPython's difficulties with the GIL mean I can only make it scalable at about 60% to 65% of the traditional efficiency (on pystones at least). That gap gives PyPy enough room to surpass CPython on scalable performance. -- Adam Olsen, aka Rhamphoryncus

4 4

tephan.
by Christian Tismer 17 Sep '07

17 Sep '07

Hi friends, Something in this check-in has a huge impact on all of the stackless tests in pypy/lib/app-tests. I think that this was unforeseen by Armin, although it was not appropriate to blame me for this without any reflection. I felt exposed to something which I was unable to react on. Especially since I did not write these modules but Stephan. Thanks to cfbolz who reminded me how to track such things down. I would like to know if the resulting errors come from a mis-use on our side, or if we did all-right so far. cheers -- chris arigo(a)codespeak.net wrote: > Author: arigo > Date: Sat Sep 15 15:38:24 2007 > New Revision: 46638 > > Modified: > pypy/dist/pypy/tool/cache.py > Log: > Protect cache building in a multithreaded py.py. > > > Modified: pypy/dist/pypy/tool/cache.py > ============================================================================== > --- pypy/dist/pypy/tool/cache.py (original) > +++ pypy/dist/pypy/tool/cache.py Sat Sep 15 15:38:24 2007 > @@ -25,6 +25,9 @@ > # Be sure to call the parent __init__() if you override it. > # > > +from threading import RLock > +lock = RLock() # multithreading protection > + > > class Cache(object): > def __init__(self): > @@ -32,20 +35,24 @@ > self._building = {} > > def getorbuild(self, key): > + lock.acquire() > try: > - return self.content[key] > - except KeyError: > - if key in self._building: > - raise Exception, "%s recursive building of %r" % ( > - self, key) > - self._building[key] = True > try: > - result = self._build(key) > - self.content[key] = result > - finally: > - del self._building[key] > - self._ready(result) > - return result > + return self.content[key] > + except KeyError: > + if key in self._building: > + raise Exception, "%s recursive building of %r" % ( > + self, key) > + self._building[key] = True > + try: > + result = self._build(key) > + self.content[key] = result > + finally: > + del self._building[key] > + self._ready(result) > + return result > + finally: > + lock.release() > getorbuild._annspecialcase_ = "specialize:memo" > > def _ready(self, result): > _______________________________________________ > pypy-svn mailing list > pypy-svn(a)codespeak.net > http://codespeak.net/mailman/listinfo/pypy-svn -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

2 1

46638 made Stackless to crash.
by Christian Tismer 16 Sep '07

16 Sep '07

Hi Armin, today you pointed me to stackless failures. They were caused by your revision 46638. both MacGyver:~/pypy-dist/pypy/lib/app_test tismer$ ../../../py/bin/py.test -A test_stackless.py -v -s and MacGyver:~/pypy-dist/pypy/lib/app_test tismer$ ~/pypy-builds/pypy-c ../../../py/bin/py.test -A test_stackless.py -v -s worked until your check-in. It is a bit annoying to get blamed on pypy-dev by you, after *you* obviously did not run these tests. Anyway, I would really like to understand what you did, since this bug does not make much sense to me. ciao -- chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

2 3

Re: [pypy-dev] pypy cleanup sprint
by Laura Creighton 14 Sep '07

14 Sep '07

In a message of Fri, 14 Sep 2007 04:31:34 +0200, Christian Tismer writes: >Laura Creighton wrote: > >... > >Well, I'd like to come if I'm wellcomed. >Has there been a concrete timing proposal, already? >-- >Christian Tismer :^) <mailto:tismer@stackless.com> >tismerysoft GmbH : Have a break! Take a ride on Python's >Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ >14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ >work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 >PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 > whom do you want to sponsor today? http://www.stackless.com/ Of course you are most welcome. People are trying to find a time now. see: http://doodle.ch/5awzskkt5mian94a There is now a direct flight from Berlin to Gbg. http://www.goteborgcityairport.se/info/resenaerer/destinationer.asp from (Tegel) -- so not sure exactly where that is, via Air Berlin. Laura

1 0

Re: [pypy-dev] pypy cleanup sprint
by Laura Creighton 14 Sep '07

14 Sep '07

In a message of Thu, 13 Sep 2007 09:33:34 EDT, "Paul deGrandis" writes: >------=_Part_27723_30036397.1189690414424 >Content-Type: text/plain; charset=ISO-8859-1 >Content-Transfer-Encoding: 7bit >Content-Disposition: inline > >Philadelphia, PA int the US. > >Paul > >On 9/13/07, Carl Friedrich Bolz <cfbolz(a)gmx.de> wrote: >> >> Laura Creighton wrote: >> > Planes may actually be cheaper. Where are you Paul? >> >> but it is more stressful and you get to see less :-) >> >> Cheers, >> >> Carl Friedrich Well, taking the Train from Oslo, Copenhagen or Stockholm will definitely be pleasant. You can also take the overnight ferry from Kiel in Germany or Fredrikshavn in Denmark. For you, though, the bulk of the cost will be on the transatlantic flight. Last time I checked, flying KLM to Amsterdam and then Amsterdam to Göteborg was the cheapest, despite the fact that KLM is not a discount airline. http://www.lfv.se/templates/LFV_InfoSida_Bred____1582.aspx lists the airlines that run to Landvetter Airport. Discount airlines include: Malmö Aviation (sometimes), Göteborg City Airline (sometimes) Sterling, Welcome Air, Blue1, Widerøe Then there is Göteborg Ciry airport. They have nothing but discount airlines flying there. http://www.goteborgcityairport.se/info/resenaerer/destinationer.asp Let me know if you need help with anything. Laura

2 1

Re: [pypy-dev] pypy cleanup sprint
by Laura Creighton 13 Sep '07

13 Sep '07

In a message of Thu, 13 Sep 2007 15:29:23 +0200, Carl Friedrich Bolz writes: >Laura Creighton wrote: >> Planes may actually be cheaper. Where are you Paul? > >but it is more stressful and you get to see less :-) > >Cheers, > >Carl Friedrich Absolutely. By all means take the train for maximum fun. But given that the request was for cheap airfair, the fact that the plane is usually the cheaper option to get from Stockholm to Göteborg may be relevant. Laura

1 0