http://code.google.com/p/unladen-swallow/wiki/ProjectPlan
Hi friends, please have a look at this. http://code.google.com/p/unladen-swallow/wiki/ProjectPlan is this YAIPAP ? Yet Another Ignorant Python Acceleration Project I see them mentioning Hölzle and Self, and I see a reference to Psyco where they want to steal from, but PyPy does not exist. IMHO, this is totally GAGA - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Mar 25, 2009, at 9:27 PM, Christian Tismer wrote:
I was reading it earlier, the simpler ideas, like making pickle faster and most of q1 deliverables seems nice, and could really help python right now, but those seems more like the things Need For Speed sprint was doing. Not the LLVM-JIT, new GC, eliminate the GIL seems unrealistic, even the pace of the project seems to be counting on tons of new developers joining the project after the Q1 milestone. -- Leonardo Santagada santagada at gmail.com
On Thu, Mar 26, 2009 at 02:42, Leonardo Santagada <santagada@gmail.com> wrote:
First, I suggest you have a look at the FAQ - when I did, I discovered the developers are full-time Google engineers. Having said that, the Google V8 group does not seem to be involved, and that's quite stupid. However, I studied with Lars Bak and the base ideas he had are the same. Missing knowledge of PyPy is also stupid obviously, but I wonder why nobody proposed "hey, let's tell them"; fighting another project. They'd sure benefit, for instance, from CALL_METHOD and CALL_LIKELY_BUILTIN ideas (they mention among their purposes fixing the performance problems addressed by CALL_LIKELY_BUILTIN, that's why I mention these). Also, they're already started with releasing. Have a look at their benchmarks: http://code.google.com/p/unladen-swallow/wiki/Releases Did you look at that because declaring it GAGA?
I don't know about Need For Sprint, but my first guess is that this maybe happens because, as pointed out various ways, CPython has lots of "obvious" deficiencies.
Well, the milestones seem crazy but they did achieve something notable in their Q1 deliverable, but most of the ideas seem correct. "Eliminate the GIL" is not hard by itself, the problem (IMHO, no hard numbers for that) is that it's impossible with refcounting (because of the cost of atomic refcount manipulation). The author of the old "free threading" patch mentioned only another problem, that is locking around name lookups in mutable name dictionaries (for looking up object members, globals, etc.), which can also be approached (and I think refcount manipulation is a bigger issue). Regards -- Paolo Giarrusso
On 3/30/09 1:28 AM, Paolo Giarrusso wrote:
Interesting. Yes, I read the FAQ, but I don't know the people.
Well, we could have contacted them. But (I'm speaking for myself) if they have decided to totally ignore a project like PyPy, then I think it is for a reason. Not knowing about PyPy means to close more than two eyes. Therefore I see no point in approaching them.
My first perception of the project was a bit distorted. I saw this as an attempt to replace PyPy with something better, and it seemed GAGA to me to do that ignoring prior work. Now that I realize that the project has much smaller goals, it becomes more realistic. The Q1 goals are relatively doable without doubt. The current achievements speedwise remind me of the anxient Python2C project. It showed the typical acceleration by a factor of around 2, which is what you can expect when eliminating the interpreter loop.
Yes, Need for Speed did small, doable things.
Q1 is fine, but it does by no means give any proof that the next milestones can be achieved.
As I remember that patch, the overhead was around 40%, mostly because of reference counting. I guess nobody actually goes this path, because it is such a waste, compared to multiple processes, and doing it "right" (where I'm referring to some Java achievements I heard of) is pretty much of a total re-write of Python. I'm pretty much wondering if the latter makes sense, given the existence of PyPy. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Mon, Mar 30, 2009 at 13:48, Christian Tismer <tismer@stackless.com> wrote:
Well, there is a mention of this project on the PyPy blog (just noticed), and they talk in term of "friendly competition". Since they seem to know about PyPy, I wonder why it's not listed on the website. But it's anyway by far incomplete (lots of interesting questions weren't answered by those pages).
You're right, but I couldn't find someone stating that it was due to reference counting, on the MLs or on FAQs. The mention of dictionaries (which are probably also a problem) was from the author of the patch, Greg Stein: http://mail.python.org/pipermail/python-dev/2001-August/017099.html
Well, converting each and every C module from refcounting to garbage collection is _not_ a total rewrite of Python. I can understand why you claim it has the same costs. But eXtreme Programming on one side, and the daily activity of the Linux kernel on another side, show that given enough manpower such things can be done. I mean, why CPython doesn't allow having multiple interpreters in the same process? Because it would take time to move all statics into the thread state. Well, much bigger changes do take place in the Linux kernel much more than once per release. Having had experience working there, I see either a severe lack of manpower, or a lack of trust into what developers can do. And even if I say "enough manpower", it still doesn't amount at all to a complete rewrite. For a human programmer, the conversion is mostly mechanical: remove (or make conditional) refcount stuff, then in all functions, register all pointers with the GC (or, in C++, turn them into handles). If you don't do that over night, you'll get bugs only in very particular situations. It's obviously boring though, but programming is not always fun.
I'm pretty much wondering if the latter makes sense, given the existence of PyPy.
Because PyPy still needs lots of time, because the PyPy idea _is_ research, and because if they meet their deadlines (I'm not sure if it's conceivably possible) Google will save computing power much earlier. Especially because their C API will be more compatible with the current one. Well, they actually claim full compatibility, but this point _is_ IMHO crazy, and they'll notice when they'll have to remove refcounting. About the "PyPy is research" statement, just a quote: http://morepypy.blogspot.com/2008/10/sprint-discussions-jit-generator.html "Partial evaluation (the basis for our JIT generator) is a 30 years old technique that was always just promising and never really successful, so the fact that we think we can solve its problems in a few years is very much hubris anyway :-). On the positive side, we think that we now know these problems much better than ever before and that we have a plan that has a chance to succeed." I didn't read the post about tracing JITs yet, so I don't know if that still holds true (I mean, if this is still about partial evaluation). Anyway, at most it's research approaching completion and production status, but IMHO nobody imagines getting production stability by the end of 2009 (that's IMHO though, I might very well be wrong, and I hope so). Regards -- Paolo Giarrusso
Christian Tismer wrote:
A bit less than that, but this sounds about right. Last I tried (somewhere before the 0.10 release), Cython gave you a total of about 10-30% for (most of) pybench, some things (like loops) being really more in the order of a factor of 2 to 6. I'd expect it to be another bit more in 0.12.
... and Cython. The fact that Cython uses CPython's C-API doesn't mean that it's not in the same order as a Python implementation. We just happily reuse what's there already - and we happily use it to interface with what's there already. Stefan
On Mon, Apr 6, 2009 at 10:42 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
I'm completely not up to argue, but cython is effectively a compiler right? (which means you cannot just run it - you need to compile it first and wait). I expect jit to be a little more transparent than that. btw - does cython support all of the python language or just a subset (sorry for ignorance) Cheers, fijal
Maciej Fijalkowski wrote:
I'm completely not up to argue,
Sure, fine with me. I'm actually happy the PyPy project is there. It gives us both competition and inspiration (although I do think that there is some space left for broader discussions...).
but cython is effectively a compiler right?
Yes.
(which means you cannot just run it - you need to compile it first and wait).
Worse than that: you have to push your code through Cython, a C compiler, a linker, and then call into CPython to load the module. :-] BTW, the bottleneck in this pipeline is really the C compiler. Cython itself is pretty fast, especially when its parser is compiled to a C extension (Cython bootstraps parts of itself to C now).
I expect jit to be a little more transparent than that.
Luckily, we have a CPython import hook (pyximport) that does all of these things for you, so it's /almost/ like a JIT. It already works for a couple of stdlib modules, for example.
btw - does cython support all of the python language or just a subset
Almost. Complete Python language support is a 1.0 goal. (no clear Python version target, we support a wild mix of Py2.x and 3.0 features today) Currently, there is a bit of work left to finish up closures, so we still can't do inner functions/classes, generators and friends. That's more a question of manpower than a real technical issue, though. Many developers are more interested in generating fast C code and fast interfaces to external libraries (C/Fortran), than in supporting all of the Python language. Stefan
On Mar 25, 2009, at 9:27 PM, Christian Tismer wrote:
I was reading it earlier, the simpler ideas, like making pickle faster and most of q1 deliverables seems nice, and could really help python right now, but those seems more like the things Need For Speed sprint was doing. Not the LLVM-JIT, new GC, eliminate the GIL seems unrealistic, even the pace of the project seems to be counting on tons of new developers joining the project after the Q1 milestone. -- Leonardo Santagada santagada at gmail.com
On Thu, Mar 26, 2009 at 02:42, Leonardo Santagada <santagada@gmail.com> wrote:
First, I suggest you have a look at the FAQ - when I did, I discovered the developers are full-time Google engineers. Having said that, the Google V8 group does not seem to be involved, and that's quite stupid. However, I studied with Lars Bak and the base ideas he had are the same. Missing knowledge of PyPy is also stupid obviously, but I wonder why nobody proposed "hey, let's tell them"; fighting another project. They'd sure benefit, for instance, from CALL_METHOD and CALL_LIKELY_BUILTIN ideas (they mention among their purposes fixing the performance problems addressed by CALL_LIKELY_BUILTIN, that's why I mention these). Also, they're already started with releasing. Have a look at their benchmarks: http://code.google.com/p/unladen-swallow/wiki/Releases Did you look at that because declaring it GAGA?
I don't know about Need For Sprint, but my first guess is that this maybe happens because, as pointed out various ways, CPython has lots of "obvious" deficiencies.
Well, the milestones seem crazy but they did achieve something notable in their Q1 deliverable, but most of the ideas seem correct. "Eliminate the GIL" is not hard by itself, the problem (IMHO, no hard numbers for that) is that it's impossible with refcounting (because of the cost of atomic refcount manipulation). The author of the old "free threading" patch mentioned only another problem, that is locking around name lookups in mutable name dictionaries (for looking up object members, globals, etc.), which can also be approached (and I think refcount manipulation is a bigger issue). Regards -- Paolo Giarrusso
On 3/30/09 1:28 AM, Paolo Giarrusso wrote:
Interesting. Yes, I read the FAQ, but I don't know the people.
Well, we could have contacted them. But (I'm speaking for myself) if they have decided to totally ignore a project like PyPy, then I think it is for a reason. Not knowing about PyPy means to close more than two eyes. Therefore I see no point in approaching them.
My first perception of the project was a bit distorted. I saw this as an attempt to replace PyPy with something better, and it seemed GAGA to me to do that ignoring prior work. Now that I realize that the project has much smaller goals, it becomes more realistic. The Q1 goals are relatively doable without doubt. The current achievements speedwise remind me of the anxient Python2C project. It showed the typical acceleration by a factor of around 2, which is what you can expect when eliminating the interpreter loop.
Yes, Need for Speed did small, doable things.
Q1 is fine, but it does by no means give any proof that the next milestones can be achieved.
As I remember that patch, the overhead was around 40%, mostly because of reference counting. I guess nobody actually goes this path, because it is such a waste, compared to multiple processes, and doing it "right" (where I'm referring to some Java achievements I heard of) is pretty much of a total re-write of Python. I'm pretty much wondering if the latter makes sense, given the existence of PyPy. cheers - chris -- Christian Tismer :^) <mailto:tismer@stackless.com> tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/
On Mon, Mar 30, 2009 at 13:48, Christian Tismer <tismer@stackless.com> wrote:
Well, there is a mention of this project on the PyPy blog (just noticed), and they talk in term of "friendly competition". Since they seem to know about PyPy, I wonder why it's not listed on the website. But it's anyway by far incomplete (lots of interesting questions weren't answered by those pages).
You're right, but I couldn't find someone stating that it was due to reference counting, on the MLs or on FAQs. The mention of dictionaries (which are probably also a problem) was from the author of the patch, Greg Stein: http://mail.python.org/pipermail/python-dev/2001-August/017099.html
Well, converting each and every C module from refcounting to garbage collection is _not_ a total rewrite of Python. I can understand why you claim it has the same costs. But eXtreme Programming on one side, and the daily activity of the Linux kernel on another side, show that given enough manpower such things can be done. I mean, why CPython doesn't allow having multiple interpreters in the same process? Because it would take time to move all statics into the thread state. Well, much bigger changes do take place in the Linux kernel much more than once per release. Having had experience working there, I see either a severe lack of manpower, or a lack of trust into what developers can do. And even if I say "enough manpower", it still doesn't amount at all to a complete rewrite. For a human programmer, the conversion is mostly mechanical: remove (or make conditional) refcount stuff, then in all functions, register all pointers with the GC (or, in C++, turn them into handles). If you don't do that over night, you'll get bugs only in very particular situations. It's obviously boring though, but programming is not always fun.
I'm pretty much wondering if the latter makes sense, given the existence of PyPy.
Because PyPy still needs lots of time, because the PyPy idea _is_ research, and because if they meet their deadlines (I'm not sure if it's conceivably possible) Google will save computing power much earlier. Especially because their C API will be more compatible with the current one. Well, they actually claim full compatibility, but this point _is_ IMHO crazy, and they'll notice when they'll have to remove refcounting. About the "PyPy is research" statement, just a quote: http://morepypy.blogspot.com/2008/10/sprint-discussions-jit-generator.html "Partial evaluation (the basis for our JIT generator) is a 30 years old technique that was always just promising and never really successful, so the fact that we think we can solve its problems in a few years is very much hubris anyway :-). On the positive side, we think that we now know these problems much better than ever before and that we have a plan that has a chance to succeed." I didn't read the post about tracing JITs yet, so I don't know if that still holds true (I mean, if this is still about partial evaluation). Anyway, at most it's research approaching completion and production status, but IMHO nobody imagines getting production stability by the end of 2009 (that's IMHO though, I might very well be wrong, and I hope so). Regards -- Paolo Giarrusso
Christian Tismer wrote:
A bit less than that, but this sounds about right. Last I tried (somewhere before the 0.10 release), Cython gave you a total of about 10-30% for (most of) pybench, some things (like loops) being really more in the order of a factor of 2 to 6. I'd expect it to be another bit more in 0.12.
... and Cython. The fact that Cython uses CPython's C-API doesn't mean that it's not in the same order as a Python implementation. We just happily reuse what's there already - and we happily use it to interface with what's there already. Stefan
On Mon, Apr 6, 2009 at 10:42 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
I'm completely not up to argue, but cython is effectively a compiler right? (which means you cannot just run it - you need to compile it first and wait). I expect jit to be a little more transparent than that. btw - does cython support all of the python language or just a subset (sorry for ignorance) Cheers, fijal
Maciej Fijalkowski wrote:
I'm completely not up to argue,
Sure, fine with me. I'm actually happy the PyPy project is there. It gives us both competition and inspiration (although I do think that there is some space left for broader discussions...).
but cython is effectively a compiler right?
Yes.
(which means you cannot just run it - you need to compile it first and wait).
Worse than that: you have to push your code through Cython, a C compiler, a linker, and then call into CPython to load the module. :-] BTW, the bottleneck in this pipeline is really the C compiler. Cython itself is pretty fast, especially when its parser is compiled to a C extension (Cython bootstraps parts of itself to C now).
I expect jit to be a little more transparent than that.
Luckily, we have a CPython import hook (pyximport) that does all of these things for you, so it's /almost/ like a JIT. It already works for a couple of stdlib modules, for example.
btw - does cython support all of the python language or just a subset
Almost. Complete Python language support is a 1.0 goal. (no clear Python version target, we support a wild mix of Py2.x and 3.0 features today) Currently, there is a bit of work left to finish up closures, so we still can't do inner functions/classes, generators and friends. That's more a question of manpower than a real technical issue, though. Many developers are more interested in generating fast C code and fast interfaces to external libraries (C/Fortran), than in supporting all of the Python language. Stefan
participants (5)
-
Christian Tismer -
Leonardo Santagada -
Maciej Fijalkowski -
Paolo Giarrusso -
Stefan Behnel