Hi all! Just to inform you a bit: Several PyPy developers have sat together and worked on a very rough roadmap for what tasks need to be done to make PyPy a realistic replacement for CPython. The work is going on here: https://codespeak.net/svn/pypy/extradoc/planning/roadmap/ We plan to work on this some more in the next days. As soon as it is in a more finished state we will ask write a mail to pypy-dev again and ask for general feedback. Cheers, Carl Friedrich
In a broader context this also mean that we would like to produce production-ready pypy interpreter as our goal :) On Dec 14, 2007 7:57 PM, Carl Friedrich Bolz <cfbolz@gmx.de> wrote:
Hi all!
Just to inform you a bit: Several PyPy developers have sat together and worked on a very rough roadmap for what tasks need to be done to make PyPy a realistic replacement for CPython. The work is going on here:
https://codespeak.net/svn/pypy/extradoc/planning/roadmap/
We plan to work on this some more in the next days. As soon as it is in a more finished state we will ask write a mail to pypy-dev again and ask for general feedback.
Cheers,
Carl Friedrich _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
"Carl Friedrich Bolz" <cfbolz@gmx.de> wrote in message news:4762D21A.3090903@gmx.de... | Hi all! | | Just to inform you a bit: Several PyPy developers have sat together and | worked on a very rough roadmap for what tasks need to be done to make | PyPy a realistic replacement for CPython. The work is going on here: | | https://codespeak.net/svn/pypy/extradoc/planning/roadmap/ Looks like a good start. | We plan to work on this some more in the next days. As soon as it is in | a more finished state we will ask write a mail to pypy-dev again and ask | for general feedback. An idea for your consideration: - Target Python 3.0 for 'production' PyPy. Rationale: It looks to me that the road to production PyPy will take several months -- about the time until 2.6 and possibly 3.0 are released. You are currently 'behind' at 2.4, working on catching up to 2.5. If you do release a 2.5 version, you will soon if not immediately be behind again. Leapfrogging would put you up to date and probably ahead of the other ports. 2.6 will have all the new 3.0 stuff that will fit without breaking anything -- but without any of the 3.0 deletions that balance the additions. It will be a relatively bloated and confusing release that newcomers should probably skip over. Parsing 3.0 should be easier with fewer alternatives, including no __future__s. Some of the new 3.0 stuff, such as most of the new IO system (io.py) is written in Python. So you get that much for free. If there are any showstoppers in 3.0 with respect to alternate implementations, now is the best time to find out and let Guido, etc, know. While there will be a 2to3.py for at least partly converting Python code, there is not, as far as I know, anything similar for C code using the C API. I expect (but could of course be wrong) that some of the people maintaining wrappers of C libraries might consider a 3.0 version to be a good time to switch using ctypes if they have not already. I am assuming that a complete cytpes module will be part of a production release, so that it will have relatively easy access to compiled shared libraries. Terry Jan Reedy
Carl Friedrich Bolz wrote:
Just to inform you a bit: Several PyPy developers have sat together and worked on a very rough roadmap for what tasks need to be done to make PyPy a realistic replacement for CPython. The work is going on here:
https://codespeak.net/svn/pypy/extradoc/planning/roadmap/
We plan to work on this some more in the next days. As soon as it is in a more finished state we will ask write a mail to pypy-dev again and ask for general feedback.
Hey, I subscribed to the mailing list again. I'm very glad that this planning is taking place. Some feedback: If I read it correctly, the list seems more or less to be about an end-result to the process, a more or less "ideal" production release. Not quite idea perhaps -- library support may still be limited, but yet quite a lot of things are in there. That's good for a goal; one should set the sights high, and I'm sure the project is capable of accomplishing all this technically, eventually. I think it's also very important to space out how you get there: have nearer-term goals with releases that aim to be useful. I think some of the most important thinking that still that needs to go into this roadmap is to figure out how to space this out over multiple releases, where you have multiple releases of hopefully increasing usefulness. This way you set up a positive feedback cycle where people test things and hopefully people will be motivated in contributing in useful places (libraries and such, and reporting bugs). At least, waiting until all of this is completed sounds like it would take quite a while as there are many tasks involved. To get to a list of projected releases, you could restrict your ambitions along various dimensions. For instance, you could focus on only supporting one or a limited amount of backends for a first release. Or you could defer erformance optimizations. Alternatively you could restrict the amount of libraries you are going to implement (this is already in the roadmap, but on the other hand the support for a GUI toolkit is in there too). I'm sure other dimensions exist that I can't even think of. Regards, Martijn
Thanks for the feedback Martijn, some comments below onsdagen den 19 december 2007 skrev Martijn Faassen:
Carl Friedrich Bolz wrote:
Just to inform you a bit: Several PyPy developers have sat together and worked on a very rough roadmap for what tasks need to be done to make PyPy a realistic replacement for CPython. The work is going on here:
https://codespeak.net/svn/pypy/extradoc/planning/roadmap/
We plan to work on this some more in the next days. As soon as it is in a more finished state we will ask write a mail to pypy-dev again and ask for general feedback.
Hey,
I subscribed to the mailing list again. I'm very glad that this planning is taking place.
Some feedback:
If I read it correctly, the list seems more or less to be about an end-result to the process, a more or less "ideal" production release. Not quite idea perhaps -- library support may still be limited, but yet quite a lot of things are in there. That's good for a goal; one should set the sights high, and I'm sure the project is capable of accomplishing all this technically, eventually.
I think it's also very important to space out how you get there: have nearer-term goals with releases that aim to be useful. I think some of the most important thinking that still that needs to go into this roadmap is to figure out how to space this out over multiple releases, where you have multiple releases of hopefully increasing usefulness. This way you set up a positive feedback cycle where people test things and hopefully people will be motivated in contributing in useful places (libraries and such, and reporting bugs). At least, waiting until all of this is completed sounds like it would take quite a while as there are many tasks involved.
The roadmap is part of a multi step process. After we get acceptance from people in the project on where we want to go, the next step is outlining how to get there. I hope that people will conclude that the best way to go about things is to make a series of releases, but we will see. Releases take a lot of effort and there is a balance to strike between producing and releasing.
To get to a list of projected releases, you could restrict your ambitions along various dimensions. For instance, you could focus on only supporting one or a limited amount of backends for a first release. Or you could defer erformance optimizations. Alternatively you could restrict the amount of libraries you are going to implement (this is already in the roadmap, but on the other hand the support for a GUI toolkit is in there too). I'm sure other dimensions exist that I can't even think of.
I think the roadmap is a set of restrictions. To be a viable replacement for CPython, there is a single backend that needs to be supported. We are not focusing on the JVM or .NET in this roadmap and we are not talking about support for other languages. For libraries, I think that is something that is absolutely necessary in order to get traction among Python users. While it would be nice if someone comes along and ports a GUI library, I don't think we can count on that happening. It is in any case a task that belongs near the end of a timeline with releases. So, the roadmap reflects our current understanding of what we have to do to have a system that can attain critical mass and become a major Python implementation, hopefully supplanting CPython one day. I am certain we will have reason to revise the roadmap along the way, dropping some tasks and introducing others. Unfortunately, there are a large number of things that have to get done, but that is to a large extent the consequences of Python being a fairly old and well established programming language. Jacob
Hi Jacob, Martijn, all, thanks for your advises and input! IMHO it makes sense to wait a bit with discussing release plans or strategies until after the tasks and their interaction are more settled and somewhat commonly agreed. And until after we know a bit more on everyone's commitments and availabilities. Maybe Carl Friedrich or Armin could kick this off once they think it makes sense to discuss release plans or strategies? best & happy days to you all! holger
Thanks for the feedback Martijn, some comments below
onsdagen den 19 december 2007 skrev Martijn Faassen:
Carl Friedrich Bolz wrote:
Just to inform you a bit: Several PyPy developers have sat together and worked on a very rough roadmap for what tasks need to be done to make PyPy a realistic replacement for CPython. The work is going on here:
https://codespeak.net/svn/pypy/extradoc/planning/roadmap/
We plan to work on this some more in the next days. As soon as it is in a more finished state we will ask write a mail to pypy-dev again and ask for general feedback.
Hey,
I subscribed to the mailing list again. I'm very glad that this planning is taking place.
Some feedback:
If I read it correctly, the list seems more or less to be about an end-result to the process, a more or less "ideal" production release. Not quite idea perhaps -- library support may still be limited, but yet quite a lot of things are in there. That's good for a goal; one should set the sights high, and I'm sure the project is capable of accomplishing all this technically, eventually.
I think it's also very important to space out how you get there: have nearer-term goals with releases that aim to be useful. I think some of the most important thinking that still that needs to go into this roadmap is to figure out how to space this out over multiple releases, where you have multiple releases of hopefully increasing usefulness. This way you set up a positive feedback cycle where people test things and hopefully people will be motivated in contributing in useful places (libraries and such, and reporting bugs). At least, waiting until all of this is completed sounds like it would take quite a while as there are many tasks involved.
The roadmap is part of a multi step process. After we get acceptance from people in the project on where we want to go, the next step is outlining how to get there. I hope that people will conclude that the best way to go about things is to make a series of releases, but we will see. Releases take a lot of effort and there is a balance to strike between producing and releasing.
To get to a list of projected releases, you could restrict your ambitions along various dimensions. For instance, you could focus on only supporting one or a limited amount of backends for a first release. Or you could defer erformance optimizations. Alternatively you could restrict the amount of libraries you are going to implement (this is already in the roadmap, but on the other hand the support for a GUI toolkit is in there too). I'm sure other dimensions exist that I can't even think of.
I think the roadmap is a set of restrictions. To be a viable replacement for CPython, there is a single backend that needs to be supported. We are not focusing on the JVM or .NET in this roadmap and we are not talking about support for other languages. For libraries, I think that is something that is absolutely necessary in order to get traction among Python users. While it would be nice if someone comes along and ports a GUI library, I don't think we can count on that happening. It is in any case a task that belongs near the end of a timeline with releases.
So, the roadmap reflects our current understanding of what we have to do to have a system that can attain critical mass and become a major Python implementation, hopefully supplanting CPython one day. I am certain we will have reason to revise the roadmap along the way, dropping some tasks and introducing others. Unfortunately, there are a large number of things that have to get done, but that is to a large extent the consequences of Python being a fairly old and well established programming language.
Jacob _______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
-- merlinux GmbH Steinbergstr. 42 31139 Hildesheim http://merlinux.de tel +49 5121 20800 75 (fax 77)
On Dec 19, 2007 11:34 PM, holger krekel <holger@merlinux.de> wrote:
Hi Jacob, Martijn, all,
thanks for your advises and input! IMHO it makes sense to wait a bit with discussing release plans or strategies until after the tasks and their interaction are more settled and somewhat commonly agreed. And until after we know a bit more on everyone's commitments and availabilities.
Maybe Carl Friedrich or Armin could kick this off once they think it makes sense to discuss release plans or strategies?
Yes, imho Carl or Armin (or someone else from the core pypy team) should lead this effort. Ondrej
Jacob Hallén wrote:
Thanks for the feedback Martijn, some comments below [snip] The roadmap is part of a multi step process. After we get acceptance from people in the project on where we want to go, the next step is outlining how to get there. I hope that people will conclude that the best way to go about things is to make a series of releases, but we will see. Releases take a lot of effort and there is a balance to strike between producing and releasing.
Yes, I'm talking about a roadmap for the roadmap. :) Just making clear that the hard part is not so much in deciding what you would like to produce in the end, but in coming up with a way to get there. This will also be an iterative approach with adjustment along the road.
To get to a list of projected releases, you could restrict your ambitions along various dimensions. For instance, you could focus on only supporting one or a limited amount of backends for a first release. Or you could defer erformance optimizations. Alternatively you could restrict the amount of libraries you are going to implement (this is already in the roadmap, but on the other hand the support for a GUI toolkit is in there too). I'm sure other dimensions exist that I can't even think of.
I think the roadmap is a set of restrictions. To be a viable replacement for CPython, there is a single backend that needs to be supported. We are not focusing on the JVM or .NET in this roadmap and we are not talking about support for other languages.
Support for other languages is indeed not in there - this is a roadmap for CPython replacement, after all. Ah, wait, I see two end-goals in here, something I missed previously: cpython replacement and jython/ironpython replacement. I had at first thought everything in this directory was part of a single roadmap, but apparently it is not?
For libraries, I think that is something that is absolutely necessary in order to get traction among Python users. While it would be nice if someone comes along and ports a GUI library, I don't think we can count on that happening. It is in any case a task that belongs near the end of a timeline with releases.
Yes, my point was that you don't have to wait until you cover a good portion of libraries yet. Before that's completed you can already do quite a few releases and get people contributing.
So, the roadmap reflects our current understanding of what we have to do to have a system that can attain critical mass and become a major Python implementation, hopefully supplanting CPython one day. I am certain we will have reason to revise the roadmap along the way, dropping some tasks and introducing others. Unfortunately, there are a large number of things that have to get done, but that is to a large extent the consequences of Python being a fairly old and well established programming language.
I never doubted there was still a large amount of things to do. It's good that these things are being explicitly identified now. Regards, Martijn
Hey, The most important point I want to make is this one. The current discussion is about a roadmap towards CPython replacement (or other interpreter replacements). This is a *different* goal from having PyPy being useful in *some* production settings. The latter is a more modest goal. It also shifts the focus: instead of aiming for parity with CPython, you aim to be *better* than CPython at *some* things, while not having parity in other areas. So, you could have an interpreter that is not yet a full replacement for CPython being very useful for some tasks. This is where my primarily interest lies right now: it'll take a while before I can use a PyPy-derived interpreter instead of CPython for many tasks, but I'm interested in using it for at least *some* tasks. Being better at some things than existing interpreters in production will validate the project in the minds of many, and should increase the mindshare of PyPy tremendously and attract enough contributors to reach the final goal of being better at *everything*. I don't want to detract from that goal of parity with existing interpreters (and beyond). I don't want to take away from that. I just want to frame the discussion in a different light, and highlight other goals. You don't need parity in all respects in order to go beyond existing interpreters in some. I'll first note one way in which PyPy is already better than existing interpreters: flexibility. Unfortunately this benefit by itself is not going to be very useful in production itself. The idea is to exploit this flexibility. Flexibility is a goal of the project, but it's only a means, not a goal, of a production-interpreter project. As to the goal of being a CPython replacement, one important realisation is that you might in fact reach replacement of CPython through a completely other route than the one sketched out. Who is to say that a CPython replacement has to be run through a C compiler, after all? The only requirement is for it to run Python code. If that Python code runs on top of .NET or JVM, so be it, as long as it runs, right? Of course as an *end* goal you might want third-party library support, but you could get quite far as a replacement of CPython in other ways without covering those. So, what intermediate goals could you set that would make PyPy better than other interpreters in production settings? These are the main routes that I can see right now: You could aim for compliance with CPython (and library support) - but then actually replacing CPython is a goal you can't compete on soon! Your compliance will shine the most on other backends, like .NET or JVM, and the feature would be access to libraries on those platforms, so integration features would be the most important. Any performance or JIT-related task is less important on the near term, while integration tasks are most important. You could make compliance less important and aim for performance. In this case you could complete with CPython right away, even if your compliance and library support isn't perfect yet. You might only support Python 2.4 features and implement a limited set of libraries, and limited extension facilities, and still attract people. Performance and JIT related tasks are the most important. You could also target one specialist platform (some embedded architecture, or some dedicated VM architecture, like the Google Android platform). You could make PyPy work way better than any competition on that. (Multi)platform support beyond Linux, OS/X and Windows becomes way more important then. What your initial focus is would influence which tasks you focus on first a lot. I think this is important to keep in mind during the roadmap discussions. I'd of course also be curious to find out which ones you're most interested in as initial goals. Saying "all of them, right now" is not good enough. :) Regards, Martijn
Dear Martijn, all, I am aware that you have initiated and participated in various successful open source communities and i know that you have experience and advises to share. But the roadmap files are currently not about release planning or strategies and ASFAIK no active developer has asked for a public discussion about such topics. I ask you and everybody to respect that. If neccessary please only write brief mails. FWIW, i think i understand and share your main point that next-release goals do not neccessarily need to focus around a PyPy's CPython-replacing interpreter. Btw, did you see that Anto has added his drafts of a "goal_jython_ironpython_replacement.txt"? best & cheers, holger On Thu, Dec 20, 2007 at 10:04 +0100, Martijn Faassen wrote:
Hey,
The most important point I want to make is this one. The current discussion is about a roadmap towards CPython replacement (or other interpreter replacements). This is a *different* goal from having PyPy being useful in *some* production settings. The latter is a more modest goal. It also shifts the focus: instead of aiming for parity with CPython, you aim to be *better* than CPython at *some* things, while not having parity in other areas.
So, you could have an interpreter that is not yet a full replacement for CPython being very useful for some tasks. This is where my primarily interest lies right now: it'll take a while before I can use a PyPy-derived interpreter instead of CPython for many tasks, but I'm interested in using it for at least *some* tasks.
Being better at some things than existing interpreters in production will validate the project in the minds of many, and should increase the mindshare of PyPy tremendously and attract enough contributors to reach the final goal of being better at *everything*. I don't want to detract from that goal of parity with existing interpreters (and beyond). I don't want to take away from that. I just want to frame the discussion in a different light, and highlight other goals. You don't need parity in all respects in order to go beyond existing interpreters in some.
I'll first note one way in which PyPy is already better than existing interpreters: flexibility. Unfortunately this benefit by itself is not going to be very useful in production itself. The idea is to exploit this flexibility. Flexibility is a goal of the project, but it's only a means, not a goal, of a production-interpreter project.
As to the goal of being a CPython replacement, one important realisation is that you might in fact reach replacement of CPython through a completely other route than the one sketched out. Who is to say that a CPython replacement has to be run through a C compiler, after all? The only requirement is for it to run Python code. If that Python code runs on top of .NET or JVM, so be it, as long as it runs, right? Of course as an *end* goal you might want third-party library support, but you could get quite far as a replacement of CPython in other ways without covering those.
So, what intermediate goals could you set that would make PyPy better than other interpreters in production settings? These are the main routes that I can see right now:
You could aim for compliance with CPython (and library support) - but then actually replacing CPython is a goal you can't compete on soon! Your compliance will shine the most on other backends, like .NET or JVM, and the feature would be access to libraries on those platforms, so integration features would be the most important. Any performance or JIT-related task is less important on the near term, while integration tasks are most important.
You could make compliance less important and aim for performance. In this case you could complete with CPython right away, even if your compliance and library support isn't perfect yet. You might only support Python 2.4 features and implement a limited set of libraries, and limited extension facilities, and still attract people. Performance and JIT related tasks are the most important.
You could also target one specialist platform (some embedded architecture, or some dedicated VM architecture, like the Google Android platform). You could make PyPy work way better than any competition on that. (Multi)platform support beyond Linux, OS/X and Windows becomes way more important then.
What your initial focus is would influence which tasks you focus on first a lot. I think this is important to keep in mind during the roadmap discussions. I'd of course also be curious to find out which ones you're most interested in as initial goals. Saying "all of them, right now" is not good enough. :)
Regards,
Martijn
_______________________________________________ pypy-dev@codespeak.net http://codespeak.net/mailman/listinfo/pypy-dev
-- merlinux GmbH Steinbergstr. 42 31139 Hildesheim http://merlinux.de tel +49 5121 20800 75 (fax 77)
Hi Carl Friedrich, all, On Fri, Dec 14, 2007 at 19:57 +0100, Carl Friedrich Bolz wrote:
Just to inform you a bit: Several PyPy developers have sat together and worked on a very rough roadmap for what tasks need to be done to make PyPy a realistic replacement for CPython. The work is going on here:
https://codespeak.net/svn/pypy/extradoc/planning/roadmap/
We plan to work on this some more in the next days. As soon as it is in a more finished state we will ask write a mail to pypy-dev again and ask for general feedback.
As i mentioned to you on chat, i see many of the files in the above directory as goals, coarse- or fine-grained and sometimes depending on other goals. They can be used to construct roadmaps out of it which then should also include release plans etc - as it stands the discussion can easily get confused there. IOW, I'd like to get to more clarity both for internal, project and outside communication purposes. best & cheers, holger
Hi Holger, 2007/12/20, holger krekel <holger@merlinux.de>:
Hi Carl Friedrich, all,
On Fri, Dec 14, 2007 at 19:57 +0100, Carl Friedrich Bolz wrote:
Just to inform you a bit: Several PyPy developers have sat together and worked on a very rough roadmap for what tasks need to be done to make PyPy a realistic replacement for CPython. The work is going on here:
https://codespeak.net/svn/pypy/extradoc/planning/roadmap/
We plan to work on this some more in the next days. As soon as it is in a more finished state we will ask write a mail to pypy-dev again and ask for general feedback.
As i mentioned to you on chat, i see many of the files in the above directory as goals, coarse- or fine-grained and sometimes depending on other goals. They can be used to construct roadmaps out of it which then should also include release plans etc - as it stands the discussion can easily get confused there. IOW, I'd like to get to more clarity both for internal, project and outside communication purposes.
Could you be a bit more specific what your point is, e.g. which specific files you mean? I agree that not everything is clear yet, but I don't think that things are completely confusing. To restate: I think a task is a concrete thing to do, including steps etc. while a goal is a largeish collection of tasks. Right now I think we should collect goals and tasks that we think we can manage in the mid-term. Exactly which of the tasks should be made into releases is an orthogonal issue, IMO. Maybe we could add a new kind of file release-XXX.txt that points to various tasks? On the other hand I am not sure it makes sense to plan more than one release ahead. To say a bit about the next release: My opinion is (and I think others agreed during the Gothenburg sprint, see http://morepypy.blogspot.com/2007/11/sprint-discussions-releases-testing.htm... ) that the next release should be really mostly a compliance and testing release. Which means that to make the release we should set up better test infrastructure and test "real-world-applications" on PyPy's Python interpreter, as well as write some mid-sized apps for our special features. In a sense that would be a rather boring release with mostly only many incremental features but on the other hand with a lot of stability improvements. How does this sound? Cheers, Carl Friedrich
Hi CF, On Thu, Dec 20, 2007 at 15:21 +0100, Carl Friedrich Bolz wrote:
2007/12/20, holger krekel <holger@merlinux.de>:
Hi Carl Friedrich, all,
On Fri, Dec 14, 2007 at 19:57 +0100, Carl Friedrich Bolz wrote:
Just to inform you a bit: Several PyPy developers have sat together and worked on a very rough roadmap for what tasks need to be done to make PyPy a realistic replacement for CPython. The work is going on here:
https://codespeak.net/svn/pypy/extradoc/planning/roadmap/
We plan to work on this some more in the next days. As soon as it is in a more finished state we will ask write a mail to pypy-dev again and ask for general feedback.
As i mentioned to you on chat, i see many of the files in the above directory as goals, coarse- or fine-grained and sometimes depending on other goals. They can be used to construct roadmaps out of it which then should also include release plans etc - as it stands the discussion can easily get confused there. IOW, I'd like to get to more clarity both for internal, project and outside communication purposes.
Could you be a bit more specific what your point is, e.g. which specific files you mean? I agree that not everything is clear yet, but I don't think that things are completely confusing.
I referred to the notion of a "roadmap" in that it usually contains releases, milestones and concrete plans for such. Currently, i think we are in the phase of (individually) collecting goals/tasks. Next is to complete, settle and somewhat agree on them if you ask me. Mentioning too much the "roadmap" word at this point IMO can contribute to raise the discussion to a wrong (and possibly confusing) level. i didn't mean to imply "completely confusing", btw.
To restate: I think a task is a concrete thing to do, including steps etc. while a goal is a largeish collection of tasks.
I am trying to talk about the concrete files/contents and their foreseeable future - not about possible definitions of "goal" and "tasks". There I see files following the "status/todo/expected outcome" structure to contain relatively concrete tasks in their "todo" sections, but representing goals itself, e.g. "pervastive testing" or "oo bytecode compiler". Others, like the current "have pypy provide the CPython API for existing C-Extensions" also sound like goals to me, not concrete steps or tasks. Anyway, maybe we can at this point just agree to disagree on the importance of having a categorization of "goals" and "tasks"?
Right now I think we should collect goals and tasks that we think we can manage in the mid-term. Exactly which of the tasks should be made into releases is an orthogonal issue, IMO.
fine with me. best & cheers, holger
participants (7)
-
Carl Friedrich Bolz -
holger krekel -
Jacob Hallén -
Maciej Fijalkowski -
Martijn Faassen -
Ondrej Certik -
Terry Reedy