Re: [Python-Dev] A new webpage promoting Compiler technology for CPython
Travis Oliphant wrote:
With Numba and Blaze we have been doing a lot of work on what essentially is compiler technology and realizing more and more that we are treading on ground that has been plowed before with many other projects. So, we wanted to create a web-site and perhaps even a mailing list or forum where people could coordinate and communicate about compiler projects, compiler tools, and ways to share efforts and ideas.
The website is: http://compilers.pydata.org/
This is a rather nice resource. Thank you for letting us know about it! There has been an attempt to record different Python implementations on the Python Wiki, and now that this is available again, I'd like to remind people about it: http://wiki.python.org/moin/PythonImplementations I know that this isn't quite the same thing as a page about compiler technology, but there is a substantial overlap.
This page is specifically for Compiler projects that either integrate with or work directly with the CPython run-time which is why PyPy is not presently listed. The PyPy project is a great project but we just felt that we wanted to explicitly create a collection of links to compilation projects that are accessible from CPython which are likely less well known.
I think that given the scope of the projects listed, it shouldn't preclude PyPy from being listed, really. After all, interoperability with CPython extensions is something of a focus area for PyPy: http://pypy.org/compat.html I don't have an agenda here - I don't use PyPy actively, my only involvement with Shedskin (which is referenced and which can produce CPython extension modules) is in packaging it for Debian, and although I do have a static analysis library I see no pressing need to promote it extensively - but I feel that when it comes to educational resources people should be a bit careful about drawing boundaries that exclude things that would benefit people substantially if they only knew about it. My reason for speaking up about this is that I've had to tell a room full of people who were told to use Cython, NumPy and even plain C to make their Python programs faster that PyPy existed. (Of course, one can justify ignoring the elephant in the room by claiming things like scientific users rely on native libraries or CPython extensions - since "science" is a very broad term, this obviously isn't universally true - but I think people should be entitled to make their own minds up, and I was not completely certain that everyone involved in the case in question was oblivious to PyPy's existence or status.)
But that is just where we started from. The website is intended to be a community website constructed from a github repository. So, we welcome pull requests from anyone who would like to see the website updated to reflect their related project. Jon Riehl (Mython, PyFront, ROFL, and many other interesting projects) and Stephen Diehl (Blaze) and I will be moderating the pull requests to begin with. But, we welcome others with similar interests to participate in that effort of moderation.
The github repository is here: https://github.com/pydata/compilers-webpage
This is intended to be a community website for information spreading, and so we welcome any and all contributions.
There is also the python-static-type-checking Google Group: https://groups.google.com/forum/?fromgroups#!forum/python-static-type-checki... If no-one beats me to it, I may post details of the site to that group because it may well be of interest to the members. Thanks once again for bringing such information together in one place! Paul
Paul Boddie, 15.02.2013 22:44:
Travis Oliphant wrote:
This page is specifically for Compiler projects that either integrate with or work directly with the CPython run-time which is why PyPy is not presently listed. The PyPy project is a great project but we just felt that we wanted to explicitly create a collection of links to compilation projects that are accessible from CPython which are likely less well known.
I think that given the scope of the projects listed, it shouldn't preclude PyPy from being listed, really. After all, interoperability with CPython extensions is something of a focus area for PyPy:
I don't have an agenda here - I don't use PyPy actively, my only involvement with Shedskin (which is referenced and which can produce CPython extension modules) is in packaging it for Debian, and although I do have a static analysis library I see no pressing need to promote it extensively - but I feel that when it comes to educational resources people should be a bit careful about drawing boundaries that exclude things that would benefit people substantially if they only knew about it.
My reason for speaking up about this is that I've had to tell a room full of people who were told to use Cython, NumPy and even plain C to make their Python programs faster that PyPy existed. (Of course, one can justify ignoring the elephant in the room by claiming things like scientific users rely on native libraries or CPython extensions - since "science" is a very broad term, this obviously isn't universally true - but I think people should be entitled to make their own minds up, and I was not completely certain that everyone involved in the case in question was oblivious to PyPy's existence or status.)
This is off-topic for this list, but the main problem with PyPy is that you'll quickly hit a lot of walls when you try to use it for anything serious in the area. It's true that there is a certain level of interoperability with CPython extensions, but calling it a "focus area" makes it sound bigger than it actually is in my ears. Even trying to get bugs fixed to at least make things work at all often means running against walls on their side. I can tell, trying to port Cython mostly consisted of bugging PyPy developers to fix stuff, which took anything from days to still-not-done, each time. And, by design, PyPy makes it very hard and time consuming to debug it and to try to fix bugs in their code base. So, while I agree that PyPy is worth a try in certain application areas, and can be helpful for some special needs, also in the field of scientific computing, it's lightyears away from a production-ready state in that area. It just doesn't integrate with the huge bulk of software that people use in their daily work. And once you rely on that software, which is hand tuned, well integrated and real-world proven in so many ways, over the time span of way more than a decade, the most visible advantage of PyPy to make Python code run faster becomes almost pointless. In that light, telling people to try PyPy and to drop (most of) their current ecosystem for it doesn't really sound helpful and clearly appears outside of the focus of the web site in question. Stefan
On Fri, Feb 15, 2013 at 2:36 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
This is off-topic for this list, but the main problem with PyPy is that you'll quickly hit a lot of walls when you try to use it for anything serious in the area. It's true that there is a certain level of interoperability with CPython extensions, but calling it a "focus area" makes it sound bigger than it actually is in my ears. Even trying to get bugs fixed to at least make things work at all often means running against walls on their side. I can tell, trying to port Cython mostly consisted of bugging PyPy developers to fix stuff, which took anything from days to still-not-done, each time. And, by design, PyPy makes it very hard and time consuming to debug it and to try to fix bugs in their code base.
I take issue with how you've described this, Stefan: I recall many on pypy-dev working with you quite a bit on the Cython port. There are some difficult problems involved and the port is not a main focus of the core PyPy team -- there's only so many free cycles. You should ping us (IRC is great) about any outstanding issues.
So, while I agree that PyPy is worth a try in certain application areas, and can be helpful for some special needs, also in the field of scientific computing, it's lightyears away from a production-ready state in that area. It just doesn't integrate with the huge bulk of software that people use in their daily work. And once you rely on that software, which is hand tuned, well integrated and real-world proven in so many ways, over the time span of way more than a decade, the most visible advantage of PyPy to make Python code run faster becomes almost pointless. In that light, telling people to try PyPy and to drop (most of) their current ecosystem for it doesn't really sound helpful and clearly appears outside of the focus of the web site in question.
I disagree that it's pointless. Numba disagrees too: it also attempts to make Python code faster. PyPy is indeed a work in progress in this area, but that doesn't necessarily preclude it from being included. -- Philip Jenvey
Philip Jenvey, 16.02.2013 01:01:
On Fri, Feb 15, 2013 at 2:36 PM, Stefan Behnel wrote:
So, while I agree that PyPy is worth a try in certain application areas, and can be helpful for some special needs, also in the field of scientific computing, it's lightyears away from a production-ready state in that area. It just doesn't integrate with the huge bulk of software that people use in their daily work. And once you rely on that software, which is hand tuned, well integrated and real-world proven in so many ways, over the time span of way more than a decade, the most visible advantage of PyPy to make Python code run faster becomes almost pointless. In that light, telling people to try PyPy and to drop (most of) their current ecosystem for it doesn't really sound helpful and clearly appears outside of the focus of the web site in question.
I disagree that it's pointless. Numba disagrees too: it also attempts to make Python code faster.
That's not even the main goal AFAIK, but in any case, it does it from the very inside of the existing ecosystem, like all of the compilers (and related software) that are listed on the website. That's the whole point.
PyPy is indeed a work in progress in this area, but that doesn't necessarily preclude it from being included.
That may be a matter of POV, but as long as PyPy fails to integrate (and you just called that "not a main focus"), I find it hard to defend its relevance. Stefan
On 16/02/13 16:41, Stefan Behnel wrote:
PyPy is indeed a work in progress in this area, but that doesn't necessarily preclude it from being included.
That may be a matter of POV, but as long as PyPy fails to integrate (and you just called that "not a main focus"), I find it hard to defend its relevance.
Stefan, you're talking about a website which lists projects such as a Forth- like language in Python, a bytecode transformer, a project "Peregine-Falcon" which appears to be a one-man experimental implementation of Python in C++, ByteRun, a "pure-Python implementation of a Python bytecode execution virtual machine", and SPARK, which has been stuck in version 0.7 pre-alpha for years. I am astonished that you think PyPy is too immature or niche for this list. http://compilers.pydata.org/ -- Steven
Steven D'Aprano, 16.02.2013 07:13:
On 16/02/13 16:41, Stefan Behnel wrote:
PyPy is indeed a work in progress in this area, but that doesn't necessarily preclude it from being included.
That may be a matter of POV, but as long as PyPy fails to integrate (and you just called that "not a main focus"), I find it hard to defend its relevance.
Stefan, you're talking about a website which lists projects such as a Forth- like language in Python, a bytecode transformer, a project "Peregine-Falcon" which appears to be a one-man experimental implementation of Python in C++, ByteRun, a "pure-Python implementation of a Python bytecode execution virtual machine", and SPARK, which has been stuck in version 0.7 pre-alpha for years. I am astonished that you think PyPy is too immature or niche for this list.
Hmm, I don't have the feeling that this discussion is leading anywhere (especially not on this list). After all, it's quite possible to fire up a PyPy runtime from a CPython runtime and have them talk to each other, letting each one do what it's good at. That doesn't make PyPy a "compiler for CPython", but at least it means that it doesn't fail completely to integrate with the rest of the world. There are arguments for both sides, which means in dubio pro reo, I guess. Stefan
On Sat, Feb 16, 2013 at 5:14 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Hmm, I don't have the feeling that this discussion is leading anywhere (especially not on this list). After all, it's quite possible to fire up a PyPy runtime from a CPython runtime and have them talk to each other, letting each one do what it's good at. That doesn't make PyPy a "compiler for CPython", but at least it means that it doesn't fail completely to integrate with the rest of the world.
There are arguments for both sides, which means in dubio pro reo, I guess.
If anyone is interested in fast Python code that integrates cleanly with external C libraries, then the combination of PyPy and cffi (http://cffi.readthedocs.org/en/latest/) should definitely be on their list of candidates to consider. Now, it may be excluded quickly because there's a CPython extension involved in the project that isn't a trivial wrapper around an existing non-Python library (and thus can't be easily replaced with cffi), but that's no excuse for not at least considering the combination in the cases where it makes sense. Yes, the PyPy team and scientific users of Python have a long history of talking past each other (and abusing each other for the mutual lack of understanding). However, that's no excuse for deliberately ignoring the advantages JIT compilation can bring, when cffi has been created specifically to give PyPy a typesafe (and JIT-transparent) way to interface with any library that provides C bindings. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Nick Coghlan, 16.02.2013 08:49:
Yes, the PyPy team and scientific users of Python have a long history of talking past each other (and abusing each other for the mutual lack of understanding). However, that's no excuse for deliberately ignoring the advantages JIT compilation can bring
No-one's ignoring that. Philip already mentioned Numba, for example. The only question is whether this needs to happen outside of the existing ecosystem, or if we can't just embrace the 95-to-5 rule and use the right tools just for the performance critical bits of our code, without sacrificing what we achieved in the last 20 years. Stefan
On Sat, Feb 16, 2013 at 10:03 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Nick Coghlan, 16.02.2013 08:49:
Yes, the PyPy team and scientific users of Python have a long history of talking past each other (and abusing each other for the mutual lack of understanding). However, that's no excuse for deliberately ignoring the advantages JIT compilation can bring
No-one's ignoring that. Philip already mentioned Numba, for example. The only question is whether this needs to happen outside of the existing ecosystem, or if we can't just embrace the 95-to-5 rule and use the right tools just for the performance critical bits of our code, without sacrificing what we achieved in the last 20 years.
Stefan
Hi Stefan. I'm sorry PyPy does not cater for your needs, by being not like CPython. This is a conscious decision that Armin made after the psyco experiment and this is the one you disagree with, but it also allows us to do cool stuff, whether you like it or not. I understand it might be frustrating and I understand you guys simply don't like us for that. I also understand that people would love to promote their own work, that's fine too. CPyext is hard not because PyPy is undebuggable, but because the problem to solve (be not like CPython, but look like CPython) is a hard one. Until we come up with an idea how to make it so most CPython C extensions work out of the box and don't need careful refcount fixes, the focus is indeed not that major. However, considering this, the discussion of a website listing random projects from useful to outright obscure does not belong to python-dev, please take it somewhere else. Cheers, fijal
participants (6)
-
Maciej Fijalkowski -
Nick Coghlan -
Paul Boddie -
Philip Jenvey -
Stefan Behnel -
Steven D'Aprano