Hi all, Here is a summary of what we discussed today on #pypy-sync. We tried to list the tasks that now lie ahead of us. There are a number of relatively independent tasks, but there is also a knot of dependent tasks that we'll have to untie bit by bit. RCtypes and the extension compiler are somewhat of a blocker for other tasks. A controversial solution is to drop them. This means: * To support calling external libraries, we would instead shift the focus on supporting the full ctypes at app-level. This means that we could support out of the box the same ctypes-based pure Python modules as CPython. There are issues here (e.g. is the regular ctypes really suitable to wrap large libraries?) but they are not worse than with rctypes. * Of course, we keep the general support for compiling stand-alone RPython programs, even if rctypes dies. * In RPython, fijal worked on a more basic replacement for rctypes, called rtti (almost finished already). It's a PyPy-specific interface to call external functions in RPython. Calls written in this interface can be compiled to good C code, but testing is a bit less convenient than with rctypes because rtti calls cannot be directly executed on top of CPython. * The CPy Object Space is going away initially too because it is based on rctypes. It should not be too hard to port it to rtti in order to continue supporting compiling extension modules for CPython. We need to think a bit more here - how can we compile ext modules for Jython and IronPython? - but it's likely not a high priority at the moment. Any comments or objections? Are there people that already make heavy use of rctypes, or the extension compiler? The graph of dependent tasks ("A --- B" means that B depends on A): merge rtti branch ---- app-level ctypes ---- kill rctypes and extcompiler \ / \ \-- port our modules --- kill keepalives \ \ -- pyrepl -- moving GCs / _______________________________/ / / malloc cleanup ----- gc transform polishing / \ fix weakrefs -----/ \ \ -- c/llvm backend sharing Additionally, some of the independent tasks mentioned: * move some app-level parts of the Python interpreter to interp-level * complete the support for Python 2.5 * switch to LLVM 2.0 * port the stackless transform to ootypesystem * JIT work A bientot, Armin.
Armin Rigo wrote:
This is actually rffi, not rtti (like Rpython Foreign Function Interface). There are probably few other, unmentioned issues: * What we do with ootypesystem backends and external functions? Right now this is implemented by backends which tends to be a bit ugly implementation. My idea would be to have backend-sensitive implementations which access backend-specific RPython functions for accessing underlaying platform classes/functions/whatever. * Platform specifics - how much do we care about portability of a generated C code? I've got several approaches in mind: - Have special kind of if which renders to #ifdef in the C code (hard) - There is a caching mechanism for basic types (size_t, int, etc.) which renders sizes of types per-platform. This works for multiple platforms, but renders generated C code non-portable. unsure if this is a problem. Cheers, fijal
Maciek Fijalkowski wrote:
I agree. Supporting external functions directly in the backend is easier, but now that gencli can call .NET code we don't need those functions to be external anymore, we can just provide a real implementation for them. The problem is that at the moment genjvm can't call java functions, so this solution would not work for it. ciao Anto
Armin Rigo wrote:
Hi all,
Hi, sorry, I completely forgot the meeting :-(. I was even at home and online at that time, but I missed it. At least I've read the logs :-). Here are some notes from my side. My personal short-term plan is to work a bit to make gencli more or less usable as a general .NET compiler, in particular for generating DLLs to be imported by others .NET programs. I personally would like not to work on rpython too much, but I have to convince my professor first :-). After having convinced him, I would like to work on the JIT, in particular to port it to ootype (and then add gencli support).
Armin, I read that you asked how hard would be to generate extension modules for Jython; the same question apply to IronPython. I think that the easiest way in the short term is to generate plain Java/.NET modules that can be imported by any Java/C# program, and rely on Jython/IronPython's mechanism to import it (thus, not relying on the CPy objectspace at all). I've already started in this direction, regarding gencli: the name of the front-end is SilveRPython, though I think I will rename it to avoid confusion with microsoft silverlight. Currently it mostly works, but I have to convince the rtyper not to mangle the name of the classes and the methods, because they need to be publicly available from the outside.
Additionally, some of the independent tasks mentioned:
* port the stackless transform to ootypesystem
I also would have spotted this point if I were at the meeting :-). I think it could be a nice topic for sprint, couldn't it? Finally, a note which is not related with the work plan but to a question that appeared in the logs; someone asked whether we want to "sell" RPython as a stand-alone product. Last sunday I gave a PyPy talk at Pycon italy and, surprisingly enough, RPython was perceived as the only (or the most relevant) usable product of the pypy project. I tried hardly to explain that rpython is basically only an accident and not very usable, but people are still impressed by the 300000% speedup. It would be interesting to know if this perception is world-wide or only limited to the attendants of my talk (maybe because I didn't stress enough the pros of the other pypy goals or the cons of rpython). ciao Anto
Antonio Cuni wrote: [snip]
I certainly don't know if it's universal. I do know that the extcompiler with a time sensitive module I ported to rpython is the only real practical use for pypy that I have had. I don't have the interest or time to offer to maintain the extcompiler (only used it for a personal hobby project which is on the back burner for the moment anyway) and after getting longs to work in the cpy object space I do appreciate the pain it may be to maintain. Janzert P.S. There is a certain irony that ShedSkin is growing a rudimentary extension compiler at the moment. :}
Hi Anto, all, On Fri, Jun 15, 2007 at 01:01 +0200, Antonio Cuni wrote:
FYI: Last week i gave a talk at the german Zope annual meeting (extradoc/talk/dzug2007/dzug2007.txt). To briefly summarize: I talked about the .NET backend, python interpreter prototypes/speed/status and the general usability of PyPy and its JIT for implementing other languages. I also noted that the extension module area is currently in flux. Questions at the talk then were mostly about the Python Interpreter, e.g. how to use persistence/Transparent proxy features e.g. with ZODB. Jim Fulton also noted that zope only needs few extension modules and often has pure python versions of its C-coded modules. There wasn't a strong focus on RPython or the ext-compiler but there was e.g. interest (prior to the talk) to port the Zope TAL interpreter to RPython in order to use it with CPython. Here is a thought: if our Python Interpreter becomes more usable and can be used as a replacement for CPython in more cases, people *can* write RPython modules for speed reasons if they want to. Practically, i guess, it'd then be important to untangle the compilation of such written-for-speed extension modules from the translation of the Python Interpreter itself. (this all apart from the question if a better JIT could obsolete the aim for writing RPython modules). best, holger
I would be very, very careful about what people talk about. Even if they talk about RPython and speed, they really don't know what they're talking about. There are some reasons why they use python and not ocaml or haskell for their purposes. One of the reasons is that it's easy to write well-working python program. With rpython it's not that easy, and we don't explain it in talks, how hard it is. Even giving enough effort to the translation toolchain error reporting, type inference problems are always hard to read and debug. That's why I think RPython will never become a language of choice of most of the people (despite few guys who know what they're doing and why). Also I (personally) think that good enough JIT would be a way better place to start writing speedy programs. You write it jit-friendly, you get speedup. From my POV maintaining all of parts required to use RPython as a general purpose language is a bit overkill, and people are really interested in parts which they yet don't know they're interested in. Cheers, fijal
On Fri, Jun 15, 2007 at 11:11 +0200, Maciek Fijalkowski wrote:
I would be very, very careful about what people talk about.
quick question: your mail appears in my Mutt as a reply to my previous mail ("Hi Anto, all") but your mail is not meant as relating to it specifically, right? holger
Maciek Fijalkowski wrote:
I understand that you need to be careful about not overselling RPython. I would also suggest you be careful about underestimating your audience. You just implied I don't know what I'm talking about because I'm interested in using RPython for speed. While my perspective on PyPy is definitely quite different than yours, I do think I have a reasonable picture of what it is about. [snip]
People will then rightfully ask you the question when you think they can benefit from a JITed interpreter in their Zope/Django/Pylons projects. RPython, immature as the toolchain might be, is at least somewhat usable today. You could tell me that I'd be more productive if I contributed to the JIT generator, but then I'd go away again and you'd lose a potential contributor. If I can speed up my templating language using RPython I might stick around. I realize that I personally am of small potential value to the project, but who knows who else you might draw in this way?
Two points: * in an open source project, others might be helping you maintain this toolchain, so the cost might be relatively little to you. * you will likely still have some maintenance cost. This could be an investment: maintaining something not part of your core goal may draw in sufficient new contributors to actually benefit the core goals as well. I realize I'm speaking from a quite different cultural perspective than many PyPy developers. I also realize I'm arguing from a self-interested perspective. I also genuinely believe that taking these other perspectives into account may help your project. So please cut me some slack here. :) I don't think you have to be worried on the short term about being flooded with clueless people taking up your time. Even if that happened, I'd consider it a luxury problem, as some percentage of new people would not be clueless and valuable to your project. Regards, Martijn
On 15 Jun 2007, at 13:04, Martijn Faassen wrote:
It seems like many are in this boat. Are there really such difficult stumbling blocks involved in making a (r)python lint and improving the current error messages while at the same time letting users of the system build a FAQ on the more common issues based in their experience? If not, why not branch off now? Alex --- Alexander Kellett PGP - 0x6BFA8EF3, FPR: DA65 D6DE 56A9 D715 EFB6 A948 B2EF 6622 6BFA 8EF3
On Fri, Jun 15, 2007 at 01:16:06PM +0200, Alexander Kellett wrote:
Some work has been done on this, using pylint which has an rpython checker. Contributed additional checks are welcome. http://www.logilab.org/projects/pylint -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science Reprise et maintenance de sites CPS: http://www.migration-cms.com/
Martijn Faassen wrote:
Sorry for underestimating you :) If I didn't add a disclaimer, I should at least - I meant *most* of the people are not that interested in speed, I kind of believe that you are the one who might be.
the real problem is that RPython is quite mature for building stand-alone applications, while it's usefulness as a CPython extension generator is somehow debatable. There are few steps needed for that, like improving our reference counting GC implementation, moving from rctypes to new interface and so on and so on. This would greatly improve performance and usability. These are not small and uninvolved tasks - it's very much in the core. I think we would welcome someone who will do that, but seems unlikely, as first someone from the core team would need to spend a lot of effort on that. It's not question about whether it's worth that - I think it is. The open question is who is willing to spend time on this - if there is someone, good, I'll not try to stop him.
I'm not saying this. I wouldn't be productive contributing to JIT.
It's not about cultural differencies I think. It's more about my own time and how I would like to use it. I would rather spend my own time on making extension modules or making CPython's extension modules work on top of PyPy than on extcompiler, which needs some more work.
Hey, I'm changing the title of this thread as this is another topic. It's one I care about as an observer of PyPy. Antonio Cuni wrote: [snip]
I think you should. I'll explain why below.
I got this impression back at the presentation last year at EuroPython. I therefore think this impression is more widespread than just your audience in Italy. This is not because you've communicated the state or future potential of PyPy wrongly, but because the audience likely has a different perspective than you do. I realize that PyPy is much more than RPython and that RPython is somewhat incidental to the project. For people working on production projects right now, these other features are less relevant at this point in time, however. Let's take PyPy's generated interpreter. If I'm currently using the CPython platform (as most Python programmers are), choosing the PyPy-C interpreter would mean that I have far less extension module support, and a slightly slower interpreter. I'd need some pretty unusual use cases currently for the benefits of PyPy-C to outweigh its drawbacks. The PyPy developers themselves of course *do* have unusual use cases. They want to experiment with interpreter generation, different language features, different platforms, and so on. I expect this to change over time. Once the PyPy (generated) JIT matures, PyPy will become a lot more attractive. If you could then somehow figure out a way to use CPython modules with PyPy's interpreter you'd get even more people. Right now, unless I really really need some unusual language feature in my production project, I don't have a motivation to use PyPy's interpreter. Maturing the PyPy interpreter will take quite a lot of time. You've come a long way and it's quite impressive, but at the same time I don't expect to be using a PyPy-generated interpreter to develop web applications any time soon. Now RPython. RPython generated modules are usable from CPython. This means I can write relatively plain Python code, generate a module, and start using this module in my web application today. The benefit is vastly increased speed for my module. Using RPython with CPython is attractive for the same reasons Pyrex is attractive, with the benefits that it's more well-engineered than Pyrex. Pyrex is currently more mature to use. Using RPython in this way is a bit of a struggle still in comparison. This is however tool support that I feel can improve relatively quickly. An open source projects needs mindshare. It needs users. PyPy is an open source project, and I'd recommend getting any users you can get. Granted, you'll get opportunists like me who are only there for production benefits. These people are less interested in language experimentation and new features - they're there for the speed primarily: the main thing that PyPy, through RPython, can offer to a Python programmer today. But if your aim is to provide a platform that is suitable for production use, you *need* opportunists like that. They can help mature the platform. Open source projects thrive by attracting developers that are in it for their own (enlightened) self-interest. Some of these opportunists might actually help out beyond just RPython. They might in fact be interested in such language experimentation. It might however not be a priority for them. For instance, I'm interested, but I need an excuse to spend more time on PyPy. Production benefits are a great excuse. Using RPython extensions with CPython is the shortest route to production benefits. Besides gaining new users, supporting RPython as a product will help PyPy in other ways as well. It will get you more module support. New users will come in and help you write more extension modules in RPython. They'll be doing this initially so they can use them in CPython, but it's a nice side effect. Currently I'm working with Godefroid Chapelle (well, he's been doing the most work so far) to try to make an RPython version of a web templating engine (Zope's page template engine). We hope we will be able to use this technology to speed up the generation of web templates. We'll be using this within a CPython server. We see the potential of PyPy. Godefroid made the observation that a web templating engine implements an interpreter. PyPy has technology to generate a JIT for arbitrary interpreters in RPython. This means that in the future we might be able to have a JIT-ed templating language, hopefully boosting performance even more. (besides the huge coolness factor) Just the potential for future JITting isn't motivating enough to experiment with PyPy right now though. The ability to use RPython with CPython is the critical bit. This is what gives us the hope we'll get a payoff in our production work. The JIT is a nice bonus that might become important later. So if people conclude that RPython is currently the most relevant product of the PyPy project for them, it doesn't mean that they haven't understood the potential of PyPy in the future. It's because they're thinking about how PyPy is relevant to their own work today. PyPy has always been a unique project: it's a research project and it's an open source project aiming for real-world use. It takes ideas from ivory towers and applies them to a pragmatic, real-world language like Python. I think this balance between pragmatic and research goals is one of this project's greatest strengths. I encourage PyPy developers to continue to embrace both goals. Regards, Martijn
On Thu, 14 Jun 2007 18:45:54 +0200 Armin Rigo <arigo@tunes.org> wrote:
Any comments or objections? Are there people that already make heavy use of rctypes, or the extension compiler?
Indeed. Mainly rctypes. As well as an ever expanding use of libc, we recently have interfaced to libSDL and cairo for some very funky rpython graphics apps. And we use libpython so that we can extract stuff from python modules. Also, we do byte shuffling with rctypes: reading binary data to/from buffers, etc. I hope we can do all this with rffi, but there are two really cool things about rctypes: 1) it runs on cpython (our main app takes 30minutes+ to compile) 2) it has an automatic code generator (ctypes can autogenerate python wrappers from C header files) Simon.
Simon Burton wrote:
Very valid points indeed. The 1st point sounds unlikely, also because you might use C macros with rffi, which somehow invalidates possible uses on top of ctypes. What I do is I test (by compiling) several parts (some small code snippets involving ext functions). That make debugging a bit harder (reading C code), but pretty straightforward - for example memory management is done by hand in this part, which makes you responsible of tracking C structures and how long they need to live. The second point is much more likely to happen and to be better than ctypes code generation. (Because rffi is way more Cish than ctypes are). Also it makes stuff a bit harder when you don't submit your bindings back. If you plan to come to the post-EP pypy sprint, we can sit a bit on the automatic code generator for rffi. Cheers, fijal
Hi Simon, (This is also a follow-up on concerns raised by Christian on #pypy today) On Fri, Jun 15, 2007 at 05:44:54PM -0700, Simon Burton wrote:
For 1), this would actually be possible with rffi too. The trick would be to have two implementations of the lltypesystem objects (array, structures, etc.). The existing implementation would only be used when ctypes is not available; if ctypes can be imported, then the other implementation could systematically be used. With a bit of effort it could have a fully compatible interface, but be internally implemented as ctypes objects (e.g. lltype.malloc(S) would build a ctypes structure type corresponding to S, and internally instantiate that). Then we could really call rffi functions, by turning it internally into a ctypes function and calling it with the ctypes objects underlying the passed-in arguments. This might even not be completely insane - in my opinion, less than implementing and maintaining rctypes itself. It's also something that can be done by someone not deeply into the code base, as long as he knows the lltype interface and how to use ctypes. A bientot, Armin.
Hi Richard, On Mon, Jun 25, 2007 at 01:01:35PM -0700, Richard Emslie wrote:
Anton (which I add to the CC's of this mail) proposed a solution for calling C APIs from llvm bytecode in a portable way, which would work for functions whose argument types are not exactly specified by the standards as well as for some kind of macros. The idea is to generate stub "helper" functions in C and compile them with llvm-gcc. The llvm bytecode that genllvm produces would then call the helper functions instead of calling directly the external C API. Shouldn't be a performance problem as llvm will inline the helpers agressively. Of course, it points again to the question of whether it's really worthwhile to have an llvm backend in PyPy or if just producing C and using llvm-gcc on everything wouldn't work just as well. On the other hand, it's interesting to note that there are other reasons for which exactly the same kind of helper functions would be useful; for example, to wrap a C library full of macros and unspecified types and constants, ctypes alone is not enough. A bientot, Armin.
Hi Armin, On 16.06.2007, at 19:10, Armin Rigo wrote:
Well, reacting a bit to it, here... The reason why this stuff kept me busy thinking since the meeting was the fact that we are about to trash exactly the little stuff people are actually using. We do have very little users, and most of them are involved right here. So if we are going to trasg stuff, then we should go and offer help to those few users, to not getting distracted from PyPy. <please assume that I could insert all of Martijn's text, too> So my thoughts were around these ideas: - Try to extend CTypes to support primitive-enough concepts that do translate to RPython or - offer support to our users, to make transition to rffi smooth and easy. I think both ways are possible, I also guess the latter woulod be supported even if we asked for help or some funding for this necessary work. Whatever way we choose, I think it would be best for PyPy if we try to be as user-friendly as possible, saying like """we believe we must change this and that immediately, but we will not leave you in the rain, but actively help porting your stuff.""" This also relates a bit to my former thoughts about things which we have donated to the open source. We can, physically, just trash anything, of course. But from a point of view of users and our growing community, this is a matter of belief and trust that we should be careful about. I'm not saying that we don't care, these are just my thoughts and feelings about what I want to be not questionable.
I think the idea is not bad, but unless I'm missing something, don't we exactly loose the testing feature by this, by having to have two different implementations, the benefit of simple CPython testing for lltypesystem would be gone?
Not sure that I immediately get it. You would use CTypes to emulate rffi functions if we are in Python, and use them directly when you are in RPython? How would the memory ownership problem be handled? Ah, the problem is circumvented by doing all this stuff on app-level, dropping high efficiency constraints? No I am confused.
By 'it', you mean the CTypes implementation, at app-level, based upon rffi? I hope I'm not completely lost, here :-) ciao - chris
Hey, Christian Tismer wrote:
Thanks Christian for making these points and having these concerns. I think that the project is wise to consider these issues once every while. I only understand a little of the technical discussion. I understand that RCTypes has problems, and that people anticipate having to break it and reimplement it in a different way in order to solve these problems. I understand that one feature that RCTypes offers, namely the ability to test your RPython code on CPython, might also go away. There are currently, as far as I can see it, two reasons to use RPython for CPython extensions: a) benefit from performance gains by rewriting your CPython code into RPython. This is what we want to do with the page template interpreter. In case of language implementations, this might lead to the ability to use other PyPy technology later (such as JIT generation and so on). Some others might purely be interested in performance though and aren't implementing a language. b) better ways to write extensions that use C libraries in CPython. The typical Pyrex use case, though it can do the former as well. This would use RCTypes. Use case a) probably has less interested people than use case b). Eventually JIT technologies will help here, which will make use case a) less important. I happen to be interested in use case a) though. It is quite exciting to me that we could have a faster template language interpreter implementation that we can use in CPython today, with the potential benefits of PyPy for interpreter translation in the future as well. We are basically language implementors, one of your core target audiences. We just feel more comfortable doing work if we know we have payoffs in the near future, i.e. use in CPython. If we were implementing this template language in isolation and had no hope we could deploy it in production any time soon, we would be less interested. I am actually quite interested in use case b) as well. While developing lxml, I've done quite a bit of work with Pyrex and I can see how RPython and RCTypes might potentially help projects like that. I've been taking a wait and see attitude here though, as the technology needed more maturing. As I understand it, the proposed changes wouldn't affect use case a). Or is the ability to generate CPython extensions also going to be broken due to this work? Please let us know, as if CPython support is going to be broken, we would likely stop our efforts for the time being. Use case b) would certainly be broken due to this work. What's more, I understand that testing your extension code in CPython first wouldn't work anymore (at least if it uses CTypes). This is a very nice feature of RPython that Pyrex doesn't have, by the way. You mentioned that there are potential ways to reimplement this feature again, but I'm not sure whether you're committing to such implementation. If I were writing C extension modules with PyPy I'd like to know things like: * what are the reasons this change is going to be made? * would this benefit just core PyPy or would it benefit extension module writers too? * when do you expect the new way to be finalized? * will you make a real commitment to make the "Test in CPython first" feature work again? Or is did you just sketched out possible ways to implement this in the new system and you hope some contributor is going to do the work? * will you make a commitment to make this new way useful and supported for extension module writers, or should they invest their time in some other technology? I.e. what are the changes a future shift will make me change things again? * How well documented will this be for CPython extension developers? Will you market this feature to developers? Things like this indicate your commitment in another way. :) Please note that I'm not actually *asking* for anything but information. I'm not telling the PyPy developers they should make commitments and do various kinds of work. That's up to you. I'm just trying to guess what other developers might want to know. Of course, the quality of your communication and the answers you actually give will make people decide whether to continue to use PyPy or move on. This will affect the way your open source community will grow and develop. I would urge the PyPy developers to consider investing in the open source community by supporting features like this and promoting them. This would mean serving admittedly self-interested developers with short term goals. The promise of immediate payoff is a great way to attract people to an open source project. If there is the hope of continuing payoff (and PyPy has tremendous potential here), some of those new people can be expected to become valuable contributors to the project as a whole. And yes, I'm self-interested in saying all of this - that's my point. Regards, Martijn
Hi Martijn, On Mon, Jun 18, 2007 at 04:36:10PM +0200, Martijn Faassen wrote:
That's a very good list of questions. Indeed, we don't have clear answers for them yet, but just vague thoughts. I think it's fair to say that we'll discuss this at EuroPython and make some commitments then; for now we're only tossing ideas around. A bientot, Armin.
Armin Rigo wrote:
This is actually rffi, not rtti (like Rpython Foreign Function Interface). There are probably few other, unmentioned issues: * What we do with ootypesystem backends and external functions? Right now this is implemented by backends which tends to be a bit ugly implementation. My idea would be to have backend-sensitive implementations which access backend-specific RPython functions for accessing underlaying platform classes/functions/whatever. * Platform specifics - how much do we care about portability of a generated C code? I've got several approaches in mind: - Have special kind of if which renders to #ifdef in the C code (hard) - There is a caching mechanism for basic types (size_t, int, etc.) which renders sizes of types per-platform. This works for multiple platforms, but renders generated C code non-portable. unsure if this is a problem. Cheers, fijal
Maciek Fijalkowski wrote:
I agree. Supporting external functions directly in the backend is easier, but now that gencli can call .NET code we don't need those functions to be external anymore, we can just provide a real implementation for them. The problem is that at the moment genjvm can't call java functions, so this solution would not work for it. ciao Anto
Armin Rigo wrote:
Hi all,
Hi, sorry, I completely forgot the meeting :-(. I was even at home and online at that time, but I missed it. At least I've read the logs :-). Here are some notes from my side. My personal short-term plan is to work a bit to make gencli more or less usable as a general .NET compiler, in particular for generating DLLs to be imported by others .NET programs. I personally would like not to work on rpython too much, but I have to convince my professor first :-). After having convinced him, I would like to work on the JIT, in particular to port it to ootype (and then add gencli support).
Armin, I read that you asked how hard would be to generate extension modules for Jython; the same question apply to IronPython. I think that the easiest way in the short term is to generate plain Java/.NET modules that can be imported by any Java/C# program, and rely on Jython/IronPython's mechanism to import it (thus, not relying on the CPy objectspace at all). I've already started in this direction, regarding gencli: the name of the front-end is SilveRPython, though I think I will rename it to avoid confusion with microsoft silverlight. Currently it mostly works, but I have to convince the rtyper not to mangle the name of the classes and the methods, because they need to be publicly available from the outside.
Additionally, some of the independent tasks mentioned:
* port the stackless transform to ootypesystem
I also would have spotted this point if I were at the meeting :-). I think it could be a nice topic for sprint, couldn't it? Finally, a note which is not related with the work plan but to a question that appeared in the logs; someone asked whether we want to "sell" RPython as a stand-alone product. Last sunday I gave a PyPy talk at Pycon italy and, surprisingly enough, RPython was perceived as the only (or the most relevant) usable product of the pypy project. I tried hardly to explain that rpython is basically only an accident and not very usable, but people are still impressed by the 300000% speedup. It would be interesting to know if this perception is world-wide or only limited to the attendants of my talk (maybe because I didn't stress enough the pros of the other pypy goals or the cons of rpython). ciao Anto
Antonio Cuni wrote: [snip]
I certainly don't know if it's universal. I do know that the extcompiler with a time sensitive module I ported to rpython is the only real practical use for pypy that I have had. I don't have the interest or time to offer to maintain the extcompiler (only used it for a personal hobby project which is on the back burner for the moment anyway) and after getting longs to work in the cpy object space I do appreciate the pain it may be to maintain. Janzert P.S. There is a certain irony that ShedSkin is growing a rudimentary extension compiler at the moment. :}
Hi Anto, all, On Fri, Jun 15, 2007 at 01:01 +0200, Antonio Cuni wrote:
FYI: Last week i gave a talk at the german Zope annual meeting (extradoc/talk/dzug2007/dzug2007.txt). To briefly summarize: I talked about the .NET backend, python interpreter prototypes/speed/status and the general usability of PyPy and its JIT for implementing other languages. I also noted that the extension module area is currently in flux. Questions at the talk then were mostly about the Python Interpreter, e.g. how to use persistence/Transparent proxy features e.g. with ZODB. Jim Fulton also noted that zope only needs few extension modules and often has pure python versions of its C-coded modules. There wasn't a strong focus on RPython or the ext-compiler but there was e.g. interest (prior to the talk) to port the Zope TAL interpreter to RPython in order to use it with CPython. Here is a thought: if our Python Interpreter becomes more usable and can be used as a replacement for CPython in more cases, people *can* write RPython modules for speed reasons if they want to. Practically, i guess, it'd then be important to untangle the compilation of such written-for-speed extension modules from the translation of the Python Interpreter itself. (this all apart from the question if a better JIT could obsolete the aim for writing RPython modules). best, holger
I would be very, very careful about what people talk about. Even if they talk about RPython and speed, they really don't know what they're talking about. There are some reasons why they use python and not ocaml or haskell for their purposes. One of the reasons is that it's easy to write well-working python program. With rpython it's not that easy, and we don't explain it in talks, how hard it is. Even giving enough effort to the translation toolchain error reporting, type inference problems are always hard to read and debug. That's why I think RPython will never become a language of choice of most of the people (despite few guys who know what they're doing and why). Also I (personally) think that good enough JIT would be a way better place to start writing speedy programs. You write it jit-friendly, you get speedup. From my POV maintaining all of parts required to use RPython as a general purpose language is a bit overkill, and people are really interested in parts which they yet don't know they're interested in. Cheers, fijal
On Fri, Jun 15, 2007 at 11:11 +0200, Maciek Fijalkowski wrote:
I would be very, very careful about what people talk about.
quick question: your mail appears in my Mutt as a reply to my previous mail ("Hi Anto, all") but your mail is not meant as relating to it specifically, right? holger
Maciek Fijalkowski wrote:
I understand that you need to be careful about not overselling RPython. I would also suggest you be careful about underestimating your audience. You just implied I don't know what I'm talking about because I'm interested in using RPython for speed. While my perspective on PyPy is definitely quite different than yours, I do think I have a reasonable picture of what it is about. [snip]
People will then rightfully ask you the question when you think they can benefit from a JITed interpreter in their Zope/Django/Pylons projects. RPython, immature as the toolchain might be, is at least somewhat usable today. You could tell me that I'd be more productive if I contributed to the JIT generator, but then I'd go away again and you'd lose a potential contributor. If I can speed up my templating language using RPython I might stick around. I realize that I personally am of small potential value to the project, but who knows who else you might draw in this way?
Two points: * in an open source project, others might be helping you maintain this toolchain, so the cost might be relatively little to you. * you will likely still have some maintenance cost. This could be an investment: maintaining something not part of your core goal may draw in sufficient new contributors to actually benefit the core goals as well. I realize I'm speaking from a quite different cultural perspective than many PyPy developers. I also realize I'm arguing from a self-interested perspective. I also genuinely believe that taking these other perspectives into account may help your project. So please cut me some slack here. :) I don't think you have to be worried on the short term about being flooded with clueless people taking up your time. Even if that happened, I'd consider it a luxury problem, as some percentage of new people would not be clueless and valuable to your project. Regards, Martijn
On 15 Jun 2007, at 13:04, Martijn Faassen wrote:
It seems like many are in this boat. Are there really such difficult stumbling blocks involved in making a (r)python lint and improving the current error messages while at the same time letting users of the system build a FAQ on the more common issues based in their experience? If not, why not branch off now? Alex --- Alexander Kellett PGP - 0x6BFA8EF3, FPR: DA65 D6DE 56A9 D715 EFB6 A948 B2EF 6622 6BFA 8EF3
On Fri, Jun 15, 2007 at 01:16:06PM +0200, Alexander Kellett wrote:
Some work has been done on this, using pylint which has an rpython checker. Contributed additional checks are welcome. http://www.logilab.org/projects/pylint -- Alexandre Fayolle LOGILAB, Paris (France) Formations Python, Zope, Plone, Debian: http://www.logilab.fr/formations Développement logiciel sur mesure: http://www.logilab.fr/services Informatique scientifique: http://www.logilab.fr/science Reprise et maintenance de sites CPS: http://www.migration-cms.com/
Martijn Faassen wrote:
Sorry for underestimating you :) If I didn't add a disclaimer, I should at least - I meant *most* of the people are not that interested in speed, I kind of believe that you are the one who might be.
the real problem is that RPython is quite mature for building stand-alone applications, while it's usefulness as a CPython extension generator is somehow debatable. There are few steps needed for that, like improving our reference counting GC implementation, moving from rctypes to new interface and so on and so on. This would greatly improve performance and usability. These are not small and uninvolved tasks - it's very much in the core. I think we would welcome someone who will do that, but seems unlikely, as first someone from the core team would need to spend a lot of effort on that. It's not question about whether it's worth that - I think it is. The open question is who is willing to spend time on this - if there is someone, good, I'll not try to stop him.
I'm not saying this. I wouldn't be productive contributing to JIT.
It's not about cultural differencies I think. It's more about my own time and how I would like to use it. I would rather spend my own time on making extension modules or making CPython's extension modules work on top of PyPy than on extcompiler, which needs some more work.
Hey, I'm changing the title of this thread as this is another topic. It's one I care about as an observer of PyPy. Antonio Cuni wrote: [snip]
I think you should. I'll explain why below.
I got this impression back at the presentation last year at EuroPython. I therefore think this impression is more widespread than just your audience in Italy. This is not because you've communicated the state or future potential of PyPy wrongly, but because the audience likely has a different perspective than you do. I realize that PyPy is much more than RPython and that RPython is somewhat incidental to the project. For people working on production projects right now, these other features are less relevant at this point in time, however. Let's take PyPy's generated interpreter. If I'm currently using the CPython platform (as most Python programmers are), choosing the PyPy-C interpreter would mean that I have far less extension module support, and a slightly slower interpreter. I'd need some pretty unusual use cases currently for the benefits of PyPy-C to outweigh its drawbacks. The PyPy developers themselves of course *do* have unusual use cases. They want to experiment with interpreter generation, different language features, different platforms, and so on. I expect this to change over time. Once the PyPy (generated) JIT matures, PyPy will become a lot more attractive. If you could then somehow figure out a way to use CPython modules with PyPy's interpreter you'd get even more people. Right now, unless I really really need some unusual language feature in my production project, I don't have a motivation to use PyPy's interpreter. Maturing the PyPy interpreter will take quite a lot of time. You've come a long way and it's quite impressive, but at the same time I don't expect to be using a PyPy-generated interpreter to develop web applications any time soon. Now RPython. RPython generated modules are usable from CPython. This means I can write relatively plain Python code, generate a module, and start using this module in my web application today. The benefit is vastly increased speed for my module. Using RPython with CPython is attractive for the same reasons Pyrex is attractive, with the benefits that it's more well-engineered than Pyrex. Pyrex is currently more mature to use. Using RPython in this way is a bit of a struggle still in comparison. This is however tool support that I feel can improve relatively quickly. An open source projects needs mindshare. It needs users. PyPy is an open source project, and I'd recommend getting any users you can get. Granted, you'll get opportunists like me who are only there for production benefits. These people are less interested in language experimentation and new features - they're there for the speed primarily: the main thing that PyPy, through RPython, can offer to a Python programmer today. But if your aim is to provide a platform that is suitable for production use, you *need* opportunists like that. They can help mature the platform. Open source projects thrive by attracting developers that are in it for their own (enlightened) self-interest. Some of these opportunists might actually help out beyond just RPython. They might in fact be interested in such language experimentation. It might however not be a priority for them. For instance, I'm interested, but I need an excuse to spend more time on PyPy. Production benefits are a great excuse. Using RPython extensions with CPython is the shortest route to production benefits. Besides gaining new users, supporting RPython as a product will help PyPy in other ways as well. It will get you more module support. New users will come in and help you write more extension modules in RPython. They'll be doing this initially so they can use them in CPython, but it's a nice side effect. Currently I'm working with Godefroid Chapelle (well, he's been doing the most work so far) to try to make an RPython version of a web templating engine (Zope's page template engine). We hope we will be able to use this technology to speed up the generation of web templates. We'll be using this within a CPython server. We see the potential of PyPy. Godefroid made the observation that a web templating engine implements an interpreter. PyPy has technology to generate a JIT for arbitrary interpreters in RPython. This means that in the future we might be able to have a JIT-ed templating language, hopefully boosting performance even more. (besides the huge coolness factor) Just the potential for future JITting isn't motivating enough to experiment with PyPy right now though. The ability to use RPython with CPython is the critical bit. This is what gives us the hope we'll get a payoff in our production work. The JIT is a nice bonus that might become important later. So if people conclude that RPython is currently the most relevant product of the PyPy project for them, it doesn't mean that they haven't understood the potential of PyPy in the future. It's because they're thinking about how PyPy is relevant to their own work today. PyPy has always been a unique project: it's a research project and it's an open source project aiming for real-world use. It takes ideas from ivory towers and applies them to a pragmatic, real-world language like Python. I think this balance between pragmatic and research goals is one of this project's greatest strengths. I encourage PyPy developers to continue to embrace both goals. Regards, Martijn
On Thu, 14 Jun 2007 18:45:54 +0200 Armin Rigo <arigo@tunes.org> wrote:
Any comments or objections? Are there people that already make heavy use of rctypes, or the extension compiler?
Indeed. Mainly rctypes. As well as an ever expanding use of libc, we recently have interfaced to libSDL and cairo for some very funky rpython graphics apps. And we use libpython so that we can extract stuff from python modules. Also, we do byte shuffling with rctypes: reading binary data to/from buffers, etc. I hope we can do all this with rffi, but there are two really cool things about rctypes: 1) it runs on cpython (our main app takes 30minutes+ to compile) 2) it has an automatic code generator (ctypes can autogenerate python wrappers from C header files) Simon.
Simon Burton wrote:
Very valid points indeed. The 1st point sounds unlikely, also because you might use C macros with rffi, which somehow invalidates possible uses on top of ctypes. What I do is I test (by compiling) several parts (some small code snippets involving ext functions). That make debugging a bit harder (reading C code), but pretty straightforward - for example memory management is done by hand in this part, which makes you responsible of tracking C structures and how long they need to live. The second point is much more likely to happen and to be better than ctypes code generation. (Because rffi is way more Cish than ctypes are). Also it makes stuff a bit harder when you don't submit your bindings back. If you plan to come to the post-EP pypy sprint, we can sit a bit on the automatic code generator for rffi. Cheers, fijal
Hi Simon, (This is also a follow-up on concerns raised by Christian on #pypy today) On Fri, Jun 15, 2007 at 05:44:54PM -0700, Simon Burton wrote:
For 1), this would actually be possible with rffi too. The trick would be to have two implementations of the lltypesystem objects (array, structures, etc.). The existing implementation would only be used when ctypes is not available; if ctypes can be imported, then the other implementation could systematically be used. With a bit of effort it could have a fully compatible interface, but be internally implemented as ctypes objects (e.g. lltype.malloc(S) would build a ctypes structure type corresponding to S, and internally instantiate that). Then we could really call rffi functions, by turning it internally into a ctypes function and calling it with the ctypes objects underlying the passed-in arguments. This might even not be completely insane - in my opinion, less than implementing and maintaining rctypes itself. It's also something that can be done by someone not deeply into the code base, as long as he knows the lltype interface and how to use ctypes. A bientot, Armin.
Hi Richard, On Mon, Jun 25, 2007 at 01:01:35PM -0700, Richard Emslie wrote:
Anton (which I add to the CC's of this mail) proposed a solution for calling C APIs from llvm bytecode in a portable way, which would work for functions whose argument types are not exactly specified by the standards as well as for some kind of macros. The idea is to generate stub "helper" functions in C and compile them with llvm-gcc. The llvm bytecode that genllvm produces would then call the helper functions instead of calling directly the external C API. Shouldn't be a performance problem as llvm will inline the helpers agressively. Of course, it points again to the question of whether it's really worthwhile to have an llvm backend in PyPy or if just producing C and using llvm-gcc on everything wouldn't work just as well. On the other hand, it's interesting to note that there are other reasons for which exactly the same kind of helper functions would be useful; for example, to wrap a C library full of macros and unspecified types and constants, ctypes alone is not enough. A bientot, Armin.
Hi Armin, On 16.06.2007, at 19:10, Armin Rigo wrote:
Well, reacting a bit to it, here... The reason why this stuff kept me busy thinking since the meeting was the fact that we are about to trash exactly the little stuff people are actually using. We do have very little users, and most of them are involved right here. So if we are going to trasg stuff, then we should go and offer help to those few users, to not getting distracted from PyPy. <please assume that I could insert all of Martijn's text, too> So my thoughts were around these ideas: - Try to extend CTypes to support primitive-enough concepts that do translate to RPython or - offer support to our users, to make transition to rffi smooth and easy. I think both ways are possible, I also guess the latter woulod be supported even if we asked for help or some funding for this necessary work. Whatever way we choose, I think it would be best for PyPy if we try to be as user-friendly as possible, saying like """we believe we must change this and that immediately, but we will not leave you in the rain, but actively help porting your stuff.""" This also relates a bit to my former thoughts about things which we have donated to the open source. We can, physically, just trash anything, of course. But from a point of view of users and our growing community, this is a matter of belief and trust that we should be careful about. I'm not saying that we don't care, these are just my thoughts and feelings about what I want to be not questionable.
I think the idea is not bad, but unless I'm missing something, don't we exactly loose the testing feature by this, by having to have two different implementations, the benefit of simple CPython testing for lltypesystem would be gone?
Not sure that I immediately get it. You would use CTypes to emulate rffi functions if we are in Python, and use them directly when you are in RPython? How would the memory ownership problem be handled? Ah, the problem is circumvented by doing all this stuff on app-level, dropping high efficiency constraints? No I am confused.
By 'it', you mean the CTypes implementation, at app-level, based upon rffi? I hope I'm not completely lost, here :-) ciao - chris
Hey, Christian Tismer wrote:
Thanks Christian for making these points and having these concerns. I think that the project is wise to consider these issues once every while. I only understand a little of the technical discussion. I understand that RCTypes has problems, and that people anticipate having to break it and reimplement it in a different way in order to solve these problems. I understand that one feature that RCTypes offers, namely the ability to test your RPython code on CPython, might also go away. There are currently, as far as I can see it, two reasons to use RPython for CPython extensions: a) benefit from performance gains by rewriting your CPython code into RPython. This is what we want to do with the page template interpreter. In case of language implementations, this might lead to the ability to use other PyPy technology later (such as JIT generation and so on). Some others might purely be interested in performance though and aren't implementing a language. b) better ways to write extensions that use C libraries in CPython. The typical Pyrex use case, though it can do the former as well. This would use RCTypes. Use case a) probably has less interested people than use case b). Eventually JIT technologies will help here, which will make use case a) less important. I happen to be interested in use case a) though. It is quite exciting to me that we could have a faster template language interpreter implementation that we can use in CPython today, with the potential benefits of PyPy for interpreter translation in the future as well. We are basically language implementors, one of your core target audiences. We just feel more comfortable doing work if we know we have payoffs in the near future, i.e. use in CPython. If we were implementing this template language in isolation and had no hope we could deploy it in production any time soon, we would be less interested. I am actually quite interested in use case b) as well. While developing lxml, I've done quite a bit of work with Pyrex and I can see how RPython and RCTypes might potentially help projects like that. I've been taking a wait and see attitude here though, as the technology needed more maturing. As I understand it, the proposed changes wouldn't affect use case a). Or is the ability to generate CPython extensions also going to be broken due to this work? Please let us know, as if CPython support is going to be broken, we would likely stop our efforts for the time being. Use case b) would certainly be broken due to this work. What's more, I understand that testing your extension code in CPython first wouldn't work anymore (at least if it uses CTypes). This is a very nice feature of RPython that Pyrex doesn't have, by the way. You mentioned that there are potential ways to reimplement this feature again, but I'm not sure whether you're committing to such implementation. If I were writing C extension modules with PyPy I'd like to know things like: * what are the reasons this change is going to be made? * would this benefit just core PyPy or would it benefit extension module writers too? * when do you expect the new way to be finalized? * will you make a real commitment to make the "Test in CPython first" feature work again? Or is did you just sketched out possible ways to implement this in the new system and you hope some contributor is going to do the work? * will you make a commitment to make this new way useful and supported for extension module writers, or should they invest their time in some other technology? I.e. what are the changes a future shift will make me change things again? * How well documented will this be for CPython extension developers? Will you market this feature to developers? Things like this indicate your commitment in another way. :) Please note that I'm not actually *asking* for anything but information. I'm not telling the PyPy developers they should make commitments and do various kinds of work. That's up to you. I'm just trying to guess what other developers might want to know. Of course, the quality of your communication and the answers you actually give will make people decide whether to continue to use PyPy or move on. This will affect the way your open source community will grow and develop. I would urge the PyPy developers to consider investing in the open source community by supporting features like this and promoting them. This would mean serving admittedly self-interested developers with short term goals. The promise of immediate payoff is a great way to attract people to an open source project. If there is the hope of continuing payoff (and PyPy has tremendous potential here), some of those new people can be expected to become valuable contributors to the project as a whole. And yes, I'm self-interested in saying all of this - that's my point. Regards, Martijn
Hi Martijn, On Mon, Jun 18, 2007 at 04:36:10PM +0200, Martijn Faassen wrote:
That's a very good list of questions. Indeed, we don't have clear answers for them yet, but just vague thoughts. I think it's fair to say that we'll discuss this at EuroPython and make some commitments then; for now we're only tossing ideas around. A bientot, Armin.
participants (11)
-
Alexander Kellett -
Alexandre Fayolle -
Antonio Cuni -
Armin Rigo -
Christian Tismer -
holger krekel -
Janzert -
Maciek Fijalkowski -
Martijn Faassen -
Richard Emslie -
Simon Burton