To anyone who took notes at the language summit at PyCon today, even if you took them just for yourself, would you mind posting them here? It would be good to have some kind of (informal!) as soon as possible, before we collectively forget. You won't be held responsible for correctness. Here are some of my own recollections (I didn't take notes but I have a decent memory): - Packaging sucks, but we're improving, and we're actually doing better than other dynamic languages. - Kevin Modzelewski answered questions about Pyston, a new (very early stage) Python VM based on the new LLVM JIT engine (which is much different from what defeated Unladen Swallow). Alex Gaynor seemed unconcerned. :-) - Jukka Lehtosalo gave a talk and answered questions about mypy, his design and implementation of pragmatic type annotations (no new syntax required, uses Python 3 function annotations). See mypy-lang.org. In response, Greg P Smith pointed people to a similar project from Google, https://github.com/google/pytypedecl, which has annotations in a separate file (hence amenable to Python 2). Larry Hastings brought up that Argument Clinic (a new way of specifying signatures for C extensions), released as part of 3.4) encodes similar information in the docstring of C functions. - Maybe this is should be the year when we start getting agreement on a standard use of function annotations to specify argument and return types, now that we seem to have a somewhat critical mass of experience with annotations. - We should make an effort to publicize that we're NOT sunsetting Python 2.7 just yet; support will continue (hopefully with ample support from distro vendors), and someone should update PEP 373. (Unclear what the new EOL is but we should definitely rescind the currently published schedule.) - We (I) still don't want to do a 2.8 release, and I don't want to accelerate 3.5, but I do think we should make things better for people who have to straddle Python 2 and 3 in a single codebase, by developing more tools, and by security and possibly installer updates to 2.7 (PEP 466). - Some suggestions that were made: PSF financial support for tool development and/or porting, add more "-3" warnings to a future Python 2.7 release, additional 2to3 fixers to help convert Python-2-only code to Python-2-and-3-single-source code, a separate linter, a sumo 2.7 distribution that includes all known backported-from-Python-3-stdlib packages, adding ensure_pip to the 2.7.7 stdlib, and several more I forgot. IIRC Glyph and Alex Gaynor are going to compile a list of pain points for people. (I can't honestly say that I convinced Glyph and Alex and a few others not to pine for 2.8, but I also honestly don't believe it will have the effect that they expect. Nor do I believe any new feature we add to 3.5 can serve as a big enough carrot.) - The recommended and least painful way to develop for Python 2 and 3 is definitely to use a single source that runs under both without translation; we no longer recommend auto-generating Python 3 compatible source code using 2to3, for a variety of reasons. Several people attested that single-source has worked well for them; Mercurial is using the 2to3 approach but they're not too happy with it. - An argument for releasing something labeled 2.8 was made based on the unavailability (current or future?) of Visual Studio 2008; the uncomfortable alternative would be to switch to a newer compiler at some 2.7.x bugfix release, which would break extension modules compiled with for 2.7.0-2.7.6, and that would confuse and upset people. - Apparently no restaurant in downtown Montreal takes reservations for a group of 30 people to show up in one hour. -- --Guido van Rossum (python.org/~guido)
Here are my notes that I jotted down from the back row. Forgive me for any mistakes. (As I shared in the intro, I am trying to get back and keep up. :)) Python Release Process: * Larry Hastings goes for vote for shortend release process. But Guido does not seem to be excited about it. Would go for go for email based voting. PyPy: * Alex Gaynor mentioned about the 7th iteration of STM - Software Transactional Memory that is being worked on. * PyPy is targetting go from 2.7.3 to 2.7.6 and Brett teases about STM enabled CPython interpreter? Iron Python: * Dino gave an updated on IronPython, 2.7.5 is under development, 3.x is under development (/will be under development soon) * Contributions from community has grown recently. MaL encouraged them to submit for a funding proposal. Jython: * Support for Buffer Protocol, For Python 3 support. cffi backend for Python is coming up. Discussion about splitting the standard library: * IronPython, Pypy say that it is not a priority request for them. Packaging: * Nick Coghlan shares his experience on how difficult is get the packing right. Every agrees and kind of recognize that recent efforts are in the right direction. Mentioned about https://pypi-preview.a.ssl.fastly.net/ and wheels packaging format. * Well maintained docs at http://packaging.python.org/en/latest/ - Python packaging user guide. * The focus/goal is not get new users easy to understand python ecosystem and use python packages. Pyston: * Kevin shared his ideas and updates on Pyston. Folks suggested about using the speed.pypy.org benchmarks to measure the effort. MyPy: * The optional static typing using functional annotations demonstrated by MyPy interested a number of developers. Few felt that it would be a nice to have feature in Python 3.5 It basically means identify what's lacking in current function annotations and work on enhancing it. * Thomos Wouters suggested to Larry Hastings that Argument clinic could be enhanced to support such a feature. * Interested parties should get together on a python type checking mailing list. Features we care about 3.5: ☐ bytes formatting redux ☐ Binary mode cleanup ☐ Type Annotations. ☐ Improved tooling AI based tooling to convert 2.x code to 3.x and provide better error messages. (Sounds exciting!) On Wed, Apr 9, 2014 at 9:08 PM, Guido van Rossum <guido@python.org> wrote:
To anyone who took notes at the language summit at PyCon today, even if you took them just for yourself, would you mind posting them here? It would be good to have some kind of (informal!) as soon as possible, before we collectively forget. You won't be held responsible for correctness.
Here are some of my own recollections (I didn't take notes but I have a decent memory):
- Packaging sucks, but we're improving, and we're actually doing better than other dynamic languages.
- Kevin Modzelewski answered questions about Pyston, a new (very early stage) Python VM based on the new LLVM JIT engine (which is much different from what defeated Unladen Swallow). Alex Gaynor seemed unconcerned. :-)
- Jukka Lehtosalo gave a talk and answered questions about mypy, his design and implementation of pragmatic type annotations (no new syntax required, uses Python 3 function annotations). See mypy-lang.org. In response, Greg P Smith pointed people to a similar project from Google, https://github.com/google/pytypedecl, which has annotations in a separate file (hence amenable to Python 2). Larry Hastings brought up that Argument Clinic (a new way of specifying signatures for C extensions), released as part of 3.4) encodes similar information in the docstring of C functions.
- Maybe this is should be the year when we start getting agreement on a standard use of function annotations to specify argument and return types, now that we seem to have a somewhat critical mass of experience with annotations.
- We should make an effort to publicize that we're NOT sunsetting Python 2.7 just yet; support will continue (hopefully with ample support from distro vendors), and someone should update PEP 373. (Unclear what the new EOL is but we should definitely rescind the currently published schedule.)
- We (I) still don't want to do a 2.8 release, and I don't want to accelerate 3.5, but I do think we should make things better for people who have to straddle Python 2 and 3 in a single codebase, by developing more tools, and by security and possibly installer updates to 2.7 (PEP 466).
- Some suggestions that were made: PSF financial support for tool development and/or porting, add more "-3" warnings to a future Python 2.7 release, additional 2to3 fixers to help convert Python-2-only code to Python-2-and-3-single-source code, a separate linter, a sumo 2.7 distribution that includes all known backported-from-Python-3-stdlib packages, adding ensure_pip to the 2.7.7 stdlib, and several more I forgot. IIRC Glyph and Alex Gaynor are going to compile a list of pain points for people. (I can't honestly say that I convinced Glyph and Alex and a few others not to pine for 2.8, but I also honestly don't believe it will have the effect that they expect. Nor do I believe any new feature we add to 3.5 can serve as a big enough carrot.)
- The recommended and least painful way to develop for Python 2 and 3 is definitely to use a single source that runs under both without translation; we no longer recommend auto-generating Python 3 compatible source code using 2to3, for a variety of reasons. Several people attested that single-source has worked well for them; Mercurial is using the 2to3 approach but they're not too happy with it.
- An argument for releasing something labeled 2.8 was made based on the unavailability (current or future?) of Visual Studio 2008; the uncomfortable alternative would be to switch to a newer compiler at some 2.7.x bugfix release, which would break extension modules compiled with for 2.7.0-2.7.6, and that would confuse and upset people.
- Apparently no restaurant in downtown Montreal takes reservations for a group of 30 people to show up in one hour.
-- --Guido van Rossum (python.org/~guido)
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/senthil%40uthcode.com
On Apr 9, 2014, at 10:30 PM, Senthil Kumaran <senthil@uthcode.com> wrote:
Mentioned about https://pypi-preview.a.ssl.fastly.net/
For what it’s worth, https://warehouse.python.org/ is a somewhat easier to remember demo url for that :] ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On Thu, Apr 10, 2014 at 6:38 AM, Guido van Rossum <guido@python.org> wrote:
To anyone who took notes at the language summit at PyCon today, even if you took them just for yourself, would you mind posting them here? It would be good to have some kind of (informal!) as soon as possible, before we collectively forget. You won't be held responsible for correctness.
From here the talks suddenly moved into mythical Python 2.8 which we will not have, nope, sorry :) Guido wants a feature list from the
The day started with introductions. Guido introduced himself as its all his fault. Release management discussion ============================== Larry Hastings started the day with discussion on 3.5 release. 3.4 release was actually in 16 months. He wanted a feedback on the next release, if we want it in a smaller release cycle than the usual 18 months. Guido mentioned to stay with the 18 month cycle. Larry also asked about opinions on state of the SCM after release candidate 1, should we create 3.5 branch and if yes then should we allow people to commit there or not? Default should point to 3.5.1 or 3.6 at that time? There can be another scenario where we do not create the 3.5 branch and keep the default as 3.5 release itself. The discussion will continue in the mailing list. Next topic in the agenda was reports from different implementations. PyPy ===== Alex Gaynor gave us the current status of `PyPy <http://pypy.org>`_ project. There will be a second fund raiser on STM. The next release is targeting 2.7.6, there were a million downloads. While discussing about Python 3 branch he explained that it it only 3 bugs away from shipping and it is based on 3.2. There was a small discussion about state of CFFI for standard library inclusion. Alex and David Beazley are supposed to work on cleaning PLY for the same. General opinion was that it will be hidden as a private part of the standard lib and to be used by the language only. Ironpython =========== Dino Viehland talked about the status of `Ironpython <http://ironpython.net>`_ project. Development is going on both 2.7 and 3.x series. 2.7.4 was released last year. Many new contributors came into the project which is a good news. Jython ======= The developers sent a detailed report to Micheal Foord and he will forward it to the python-dev list. The takeaways from the mail are * Small number of contributors is a big problem. * 2.7.beta2 is tagged which used Java7. * Buffer protocol work is done (foundation to Python3 support). * They are also working on PyPi tooling. * There is also hope for releasing CFFI backend for Jython during Europycon sprints. No standard library as module ============================== When it was asked that if the other implementations want the standard library as a separate module to be resused, all agreed as 'No'. Packaging =========== It was the longest discussion which made hungry developers really hungry :) Jokes aside, Nick Coghlan gave a detailed report on the advancement of the packaging world. Most of the development/design discussions are now happening on the distutils sig and in pypi mailing lists. He managed to put the use cases a very broader audience now, so we can except better feedbacks. On the development side, Warehouse is now implementing all old API(s), you may want to try it out at `https://warehouse.python.org/ <https://warehouse.python.org/>`_. 3.4 has pip included, one usecase was to help people who downloads binary installers from our site. They can now install Django or other projects in wheel format. Everyone also agreed that having the buildsystem inside the language is a bad idea. The buildsystem should be able to do cross-version builds. Nick also pointed us to `http://packaging.python.org/ <http://packaging.python.org/en/latest/>`_ which is the documentation for the whole echosystem. We all agreed that the Python echosystem is bigger than the core interpreter. Glyph wants a PSF fund to a usability study on Python. There were a few other suggestion on PSF support for tooling development. Pyston ======= Kevin Modzelewski explained how they are rebuilding a complete vm which is targeted to Python, this also means too much work but one can customize. It is targeting Python2.7 as Dropbox runs on it. At this time of discussion Nick pointed us to `http://speed.python.org/ <http://speed.python.org/>`_, he asked if any of the implementations wants to maintain it. We need more volunteers for that, target is to have a common set of tests to benchmark different implementations. Mypy project ============= Jukka Lehtosalo gave a talk on his `mypy project <http://mypy-lang.org>`_ which uses Python3 function annotations. Greg P Smith pointed us to a similar kind of Google project, `https://github.com/google/pytypedecl <https://github.com/google/pytypedecl>`_. Notes from teaching and outreach ================================= Selena Deckelmann talked about few pain points from teaching and outreach. * Website is confusing. (Should I go for Python2 or Python3?) * Packaging and installer problem * So many different bug tracking system is also confusing * OPW program for Cpython, this is the first year we are participating. * Jessica McKellar will write "brand new coder tutorials". Mercurial =========== Matt Mackall talked about Mercurial's painpoints for Python3. It currently works for 2.4-2.7, though he might drop 2.4 support in near future. It will be on 2.7 till RHEL7 is not EOL. He also said startup time is concern for him. Only big positive point he can see in Python3 is SNI. That feature allows you to do HTTPS to non ip based virtual hosts. Porting whole Mercurial to Python 3 is still a very big work. They had two gsoc students in last two years. people who are asking for 2.8 to understand better. We also want to help developers to make a single source for Python 2 and Python 3 release less painful. Python 2.7 is alive and in good health and support will continue on the same. Few points were talked about from 3.5, like byte formatting, unicode surrogate, binary mode cleans for bytes etc. Kushal
Le 10/04/2014 13:24, Kushal Das a écrit :
At this time of discussion Nick pointed us to `http://speed.python.org/ <http://speed.python.org/>`_, he asked if any of the implementations wants to maintain it. We need more volunteers for that, target is to have a common set of tests to benchmark different implementations.
I feel a bit tired to point out that there *is* a common set of cross-implementation benchmarks at http://hg.python.org/benchmarks It is maintained and there is also a mailing-list to discuss it at https://mail.python.org/mailman/listinfo/speed Regards Antoine.
Maybe we don't need a volunteer to maintain it, but we sure need a volunteer to coordinate and spread the knowledge! :-) On Thu, Apr 10, 2014 at 10:20 AM, Antoine Pitrou <solipsis@pitrou.net>wrote:
Le 10/04/2014 13:24, Kushal Das a écrit :
At this time of discussion Nick pointed us to `http://speed.python.org/ <http://speed.python.org/>`_, he asked if any of the implementations wants to maintain it. We need more volunteers for that, target is to have a common set of tests to benchmark different implementations.
I feel a bit tired to point out that there *is* a common set of cross-implementation benchmarks at http://hg.python.org/benchmarks
It is maintained and there is also a mailing-list to discuss it at https://mail.python.org/mailman/listinfo/speed
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Thu, Apr 10, 2014 at 7:50 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
I feel a bit tired to point out that there *is* a common set of cross-implementation benchmarks at http://hg.python.org/benchmarks
It is maintained and there is also a mailing-list to discuss it at https://mail.python.org/mailman/listinfo/speed
I think I had the comma in the wrong place for that sentence. Thanks for information, I am adding them in the blog post. Kushal -- http://fedoraproject.org http://kushaldas.in
On 10 Apr 2014 10:23, "Antoine Pitrou" <solipsis@pitrou.net> wrote:
Le 10/04/2014 13:24, Kushal Das a écrit :
At this time of discussion Nick pointed us to `http://speed.python.org/ <http://speed.python.org/>`_, he asked if any of the implementations wants to maintain it. We need more volunteers for that, target is to have a common set of tests to benchmark different implementations.
I feel a bit tired to point out that there *is* a common set of
cross-implementation benchmarks at http://hg.python.org/benchmarks Yes, I pointed that out - the issue is that they're not run regularly and the results posted as they currently for PyPy. That is, the discussion was about making speed.python.org a going concern, not the benchmarks themselves. Cheers, Nick.
p It is maintained and there is also a mailing-list to discuss it at https://mail.python.org/mailman/listinfo/speed
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On Thu, Apr 10, 2014 at 4:24 AM, Kushal Das <kushaldas@gmail.com> wrote:
On Thu, Apr 10, 2014 at 6:38 AM, Guido van Rossum <guido@python.org> wrote:
To anyone who took notes at the language summit at PyCon today, even if you took them just for yourself, would you mind posting them here? It would be good to have some kind of (informal!) as soon as possible, before we collectively forget. You won't be held responsible for correctness.
The day started with introductions. Guido introduced himself as its all his fault.
Release management discussion ==============================
Larry Hastings started the day with discussion on 3.5 release. 3.4 release was actually in 16 months. He wanted a feedback on the next release, if we want it in a smaller release cycle than the usual 18 months. Guido mentioned to stay with the 18 month cycle.
Larry also asked about opinions on state of the SCM after release candidate 1, should we create 3.5 branch and if yes then should we allow people to commit there or not? Default should point to 3.5.1 or 3.6 at that time? There can be another scenario where we do not create the 3.5 branch and keep the default as 3.5 release itself. The discussion will continue in the mailing list.
Next topic in the agenda was reports from different implementations.
PyPy =====
Alex Gaynor gave us the current status of `PyPy <http://pypy.org>`_ project. There will be a second fund raiser on STM. The next release is targeting 2.7.6, there were a million downloads. While discussing about Python 3 branch he explained that it it only 3 bugs away from shipping and it is based on 3.2.
There was a small discussion about state of CFFI for standard library inclusion. Alex and David Beazley are supposed to work on cleaning PLY for the same. General opinion was that it will be hidden as a private part of the standard lib and to be used by the language only.
No, the opinion was that it _shouldn't_ be hidden as a private part of the standard library :) But some cleanup needs to happen before it can be added to the stdlib.
Ironpython ===========
Dino Viehland talked about the status of `Ironpython <http://ironpython.net>`_ project. Development is going on both 2.7 and 3.x series. 2.7.4 was released last year. Many new contributors came into the project which is a good news.
Jython =======
The developers sent a detailed report to Micheal Foord and he will forward it to the python-dev list. The takeaways from the mail are
* Small number of contributors is a big problem. * 2.7.beta2 is tagged which used Java7. * Buffer protocol work is done (foundation to Python3 support). * They are also working on PyPi tooling. * There is also hope for releasing CFFI backend for Jython during Europycon sprints.
No standard library as module ==============================
When it was asked that if the other implementations want the standard library as a separate module to be resused, all agreed as 'No'.
Their answer was mostly "don't care". It has some minor benefits, in particular when they move to Python 3 and track active development more closely, but no important ones.
Packaging ===========
It was the longest discussion which made hungry developers really hungry :) Jokes aside, Nick Coghlan gave a detailed report on the advancement of the packaging world. Most of the development/design discussions are now happening on the distutils sig and in pypi mailing lists. He managed to put the use cases a very broader audience now, so we can except better feedbacks. On the development side, Warehouse is now implementing all old API(s), you may want to try it out at `https://warehouse.python.org/ <https://warehouse.python.org/>`_.
3.4 has pip included, one usecase was to help people who downloads binary installers from our site. They can now install Django or other projects in wheel format.
Everyone also agreed that having the buildsystem inside the language is a bad idea. The buildsystem should be able to do cross-version builds.
Nick also pointed us to `http://packaging.python.org/ <http://packaging.python.org/en/latest/>`_ which is the documentation for the whole echosystem. We all agreed that the Python echosystem is bigger than the core interpreter.
Glyph wants a PSF fund to a usability study on Python. There were a few other suggestion on PSF support for tooling development.
Pyston =======
Kevin Modzelewski explained how they are rebuilding a complete vm which is targeted to Python, this also means too much work but one can customize. It is targeting Python2.7 as Dropbox runs on it.
At this time of discussion Nick pointed us to `http://speed.python.org/ <http://speed.python.org/>`_, he asked if any of the implementations wants to maintain it. We need more volunteers for that, target is to have a common set of tests to benchmark different implementations.
Mypy project =============
Jukka Lehtosalo gave a talk on his `mypy project <http://mypy-lang.org>`_ which uses Python3 function annotations. Greg P Smith pointed us to a similar kind of Google project, `https://github.com/google/pytypedecl <https://github.com/google/pytypedecl>`_.
Notes from teaching and outreach =================================
Selena Deckelmann talked about few pain points from teaching and outreach.
* Website is confusing. (Should I go for Python2 or Python3?) * Packaging and installer problem * So many different bug tracking system is also confusing * OPW program for Cpython, this is the first year we are participating. * Jessica McKellar will write "brand new coder tutorials".
I believe this was mostly about collecting new coder resources that already exist, but are hard to find (and to qualitatively judge.)
Mercurial ===========
Matt Mackall talked about Mercurial's painpoints for Python3. It currently works for 2.4-2.7, though he might drop 2.4 support in near future. It will be on 2.7 till RHEL7 is not EOL. He also said startup time is concern for him. Only big positive point he can see in Python3 is SNI. That feature allows you to do HTTPS to non ip based virtual hosts. Porting whole Mercurial to Python 3 is still a very big work. They had two gsoc students in last two years.
From here the talks suddenly moved into mythical Python 2.8 which we will not have, nope, sorry :) Guido wants a feature list from the people who are asking for 2.8 to understand better. We also want to help developers to make a single source for Python 2 and Python 3 release less painful.
Python 2.7 is alive and in good health and support will continue on the same.
Few points were talked about from 3.5, like byte formatting, unicode surrogate, binary mode cleans for bytes etc.
Kushal _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/thomas%40python.org
-- Thomas Wouters <thomas@python.org> Hi! I'm an email virus! Think twice before sending your email to help me spread!
On Thu, Apr 10, 2014 at 11:41 AM, Thomas Wouters <thomas@python.org> wrote:
On Thu, Apr 10, 2014 at 4:24 AM, Kushal Das <kushaldas@gmail.com> wrote:
There was a small discussion about state of CFFI for standard library
inclusion. Alex and David Beazley are supposed to work on cleaning PLY for the same. General opinion was that it will be hidden as a private part of the standard lib and to be used by the language only.
No, the opinion was that it _shouldn't_ be hidden as a private part of the standard library :) But some cleanup needs to happen before it can be added to the stdlib.
Huh, I totally missed this (and I just gave Kushal a confused answer when he asked me about it in person). Can someone please post here what the plan is exactly? I don't want to press for a PEP, but I would at least like to understand the plan for CFFI and PLY before it is executed, since I have never had to use either one, and it feels like each of these will require some commitment to maintenance once they are in, in addition to cleanup before they go in. And no, waving hands and saying "there's already a blog post about CFFI somewhere" is not good enough. I want the full description of the plan written up here in python-dev. Blog links might serve to clarify the motivation though. -- --Guido van Rossum (python.org/~guido)
On Thu, Apr 10, 2014 at 12:58 PM, Guido van Rossum <guido@python.org> wrote:
On Thu, Apr 10, 2014 at 11:41 AM, Thomas Wouters <thomas@python.org> wrote:
On Thu, Apr 10, 2014 at 4:24 AM, Kushal Das <kushaldas@gmail.com> wrote:
There was a small discussion about state of CFFI for standard library inclusion. Alex and David Beazley are supposed to work on cleaning PLY for the same. General opinion was that it will be hidden as a private part of the standard lib and to be used by the language only.
No, the opinion was that it _shouldn't_ be hidden as a private part of the standard library :) But some cleanup needs to happen before it can be added to the stdlib.
Huh, I totally missed this (and I just gave Kushal a confused answer when he asked me about it in person). Can someone please post here what the plan is exactly? I don't want to press for a PEP, but I would at least like to understand the plan for CFFI and PLY before it is executed, since I have never had to use either one, and it feels like each of these will require some commitment to maintenance once they are in, in addition to cleanup before they go in. And no, waving hands and saying "there's already a blog post about CFFI somewhere" is not good enough. I want the full description of the plan written up here in python-dev. Blog links might serve to clarify the motivation though.
The discussion happened leading up to the language summit in 2013. ply: https://mail.python.org/pipermail/python-dev/2013-February/124389.html pycparser: https://mail.python.org/pipermail/python-dev/2013-February/124341.html (earlier part of same discussion) My recollection was that we'd add a cleaned-up ply to the stdlib and leave pycparser as a private implementation detail of cffi. -eric
Le 10/04/2014 20:58, Guido van Rossum a écrit :
Huh, I totally missed this (and I just gave Kushal a confused answer when he asked me about it in person). Can someone please post here what the plan is exactly? I don't want to press for a PEP, but I would at least like to understand the plan for CFFI and PLY before it is executed, since I have never had to use either one, and it feels like each of these will require some commitment to maintenance once they are in, in addition to cleanup before they go in.
FWIW, I do hope there would be a PEP before including CFFI... Actually I don't understand what would justify an exemption. Regards Antoine.
On Thu, Apr 10, 2014 at 12:30 PM, Antoine Pitrou <solipsis@pitrou.net>wrote:
Le 10/04/2014 20:58, Guido van Rossum a écrit :
Huh, I totally missed this (and I just gave Kushal a confused answer when he asked me about it in person). Can someone please post here what the plan is exactly? I don't want to press for a PEP, but I would at least like to understand the plan for CFFI and PLY before it is executed, since I have never had to use either one, and it feels like each of these will require some commitment to maintenance once they are in, in addition to cleanup before they go in.
FWIW, I do hope there would be a PEP before including CFFI... Actually I don't understand what would justify an exemption
There's absolutely no reason to exempt CFFI, IMHO. On the contrary -- the dependence on other 3rd party modules (PLY and pycparesr), and the related dilemma of whether to expose each/both as stdlib modules or hide as internal implementation details -- makes a PEP even more important here. Eli
Well, I was going to put off requesting a PEP until I'd judged the plan, but clearly (a) there isn't actually a plan (just some vague description of an end result that some feel desirable) and (b) it's controversial. So, yes, it definitely needs a PEP. Also, even though this came up a year ago, nobody actually cared enough to write a PEP; which makes me think that there are probably more than a few problems with the whole idea. (One of which may be that while PLY is stable, it also sounds completely unmaintainable; another may be that CFFI seems to require too much of a development environment to be available to count as an alternative to ctypes.) On Thu, Apr 10, 2014 at 3:34 PM, Eli Bendersky <eliben@gmail.com> wrote:
On Thu, Apr 10, 2014 at 12:30 PM, Antoine Pitrou <solipsis@pitrou.net>wrote:
Le 10/04/2014 20:58, Guido van Rossum a écrit :
Huh, I totally missed this (and I just gave Kushal a confused answer when he asked me about it in person). Can someone please post here what the plan is exactly? I don't want to press for a PEP, but I would at least like to understand the plan for CFFI and PLY before it is executed, since I have never had to use either one, and it feels like each of these will require some commitment to maintenance once they are in, in addition to cleanup before they go in.
FWIW, I do hope there would be a PEP before including CFFI... Actually I don't understand what would justify an exemption
There's absolutely no reason to exempt CFFI, IMHO. On the contrary -- the dependence on other 3rd party modules (PLY and pycparesr), and the related dilemma of whether to expose each/both as stdlib modules or hide as internal implementation details -- makes a PEP even more important here.
Eli
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org
-- --Guido van Rossum (python.org/~guido)
On Apr 10, 2014, at 12:34 PM, Eli Bendersky wrote:
There's absolutely no reason to exempt CFFI, IMHO. On the contrary -- the dependence on other 3rd party modules (PLY and pycparesr), and the related dilemma of whether to expose each/both as stdlib modules or hide as internal implementation details -- makes a PEP even more important here.
Agreed. We've already started down the slippery slope of bundling third party packages (e.g. the ensurepip dependencies) and this is causing headaches for some of us distro guys. Either they're in the stdlib or their not. -Barry
On 10 April 2014 20:30, Antoine Pitrou <solipsis@pitrou.net> wrote:
FWIW, I do hope there would be a PEP before including CFFI... Actually I don't understand what would justify an exemption.
I agree. I'd like to see a clear explanation of what advantages (and disadvantages!) CFFI gives over ctypes, as well as the plan for inclusion and how the inevitable confusion over whether to use ctypes or cffi will be handled. The fact that cffi requires bringing in ply and a vendored-but-not-public copy of pycparser, seems to imply to me that there's a lot of cost and I'd like to understand the gains. That's not to say that adding ply to the standard library mightn't be worth it in its own right, but there are a lot of other parsers out there, and I'd rather we blessed one as "best of breed" rather than "because cffi uses it". In particular, my understanding is that in order to get the key benefits of cffi (API compatibility rather than ABI) you need to include some sort of complex "generate and compile C" step in your project's build. That implies that using cffi requires building separate wheels for each Python version (as with any C extension) and having a C compiler to do the build. There are a lot of projects that I know of (particularly wrappers for Windows APIs like Colorama and pyreadline) migrating to ctypes precisely because they get a pure Python solution that doesn't need binary builds or version-dependent distributions. Those projects won't presumably be migrating to cffi. Also, a key area where ctypes is used seems to be the Windows API - and there, ABI compatibility for Windows APIs is the norm so the advantage of only needing API compatibility is minimal at best. At the moment, I see no real reason to add cffi to the stdlib - it certainly isn't an "obvious" decision to do so. This seems like clear PEP territory. Paul
On Thu, Apr 10, 2014 at 3:12 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 10 April 2014 20:30, Antoine Pitrou <solipsis@pitrou.net> wrote:
FWIW, I do hope there would be a PEP before including CFFI... Actually I don't understand what would justify an exemption.
I agree. I'd like to see a clear explanation of what advantages (and disadvantages!) CFFI gives over ctypes, as well as the plan for inclusion and how the inevitable confusion over whether to use ctypes or cffi will be handled. The fact that cffi requires bringing in ply and a vendored-but-not-public copy of pycparser, seems to imply to me that there's a lot of cost and I'd like to understand the gains. That's not to say that adding ply to the standard library mightn't be worth it in its own right, but there are a lot of other parsers out there, and I'd rather we blessed one as "best of breed" rather than "because cffi uses it".
In particular, my understanding is that in order to get the key benefits of cffi (API compatibility rather than ABI) you need to include some sort of complex "generate and compile C" step in your project's build. That implies that using cffi requires building separate wheels for each Python version (as with any C extension) and having a C compiler to do the build. There are a lot of projects that I know of (particularly wrappers for Windows APIs like Colorama and pyreadline) migrating to ctypes precisely because they get a pure Python solution that doesn't need binary builds or version-dependent distributions. Those projects won't presumably be migrating to cffi. Also, a key area where ctypes is used seems to be the Windows API - and there, ABI compatibility for Windows APIs is the norm so the advantage of only needing API compatibility is minimal at best.
Nope. CFFI supports both C-built extensions and ctypes-style shared library loading. There are pure-Python bindings for modules that use CFFI and don't require a C compiler(https://github.com/kirbyfan64/clamav-python).
At the moment, I see no real reason to add cffi to the stdlib - it certainly isn't an "obvious" decision to do so. This seems like clear PEP territory.
It's somewhat faster but has a longer warm-up time. Useful if the same module is going to be reused several times.
Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com
-- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated."
Hi, On 10 April 2014 22:12, Paul Moore <p.f.moore@gmail.com> wrote:
I agree. I'd like to see a clear explanation of what advantages (and disadvantages!) CFFI gives over ctypes, as well as the plan for inclusion and how the inevitable confusion over whether to use ctypes or cffi will be handled. (...)
I can't judge exactly what was told in the Language Summit, but here is my own position about CFFI. Code-wise, we're in precisely the same spot as last year. The usage of CFFI seems to be growing a lot. However, it's not in any stdlib-ready state right now. Why not? Because we have a plan to go forward and fix the main issues people seem to be having: when used in "API mode" there is some building-C-sources-and-compiling-them going on under your feet; however, "explicit is better than implicit" seems to apply here too. Thus, it seems that the basic model might be changed toward a variant in which you put your C declarations into some separate file that you need to execute once, in order to build and compile a regular C extension module. This would be superficial, but change the perception of CFFI to be "a preprocessor that produces C extension modules". This affects the "API mode" but not the "ABI mode" (which is basically the same as ctypes, modulo the syntax). A bientôt, Armin.
On 11 Apr 2014 05:39, "Armin Rigo" <arigo@tunes.org> wrote:
Hi,
On 10 April 2014 22:12, Paul Moore <p.f.moore@gmail.com> wrote:
I agree. I'd like to see a clear explanation of what advantages (and disadvantages!) CFFI gives over ctypes, as well as the plan for inclusion and how the inevitable confusion over whether to use ctypes or cffi will be handled. (...)
I can't judge exactly what was told in the Language Summit, but here is my own position about CFFI. Code-wise, we're in precisely the same spot as last year. The usage of CFFI seems to be growing a lot. However, it's not in any stdlib-ready state right now.
Why not? Because we have a plan to go forward and fix the main issues people seem to be having: when used in "API mode" there is some building-C-sources-and-compiling-them going on under your feet; however, "explicit is better than implicit" seems to apply here too. Thus, it seems that the basic model might be changed toward a variant in which you put your C declarations into some separate file that you need to execute once, in order to build and compile a regular C extension module.
This would be superficial, but change the perception of CFFI to be "a preprocessor that produces C extension modules".
This affects the "API mode" but not the "ABI mode" (which is basically the same as ctypes, modulo the syntax).
This is along the lines of what Alex said at the summit (i.e. it wasn't proposed for 3.4 because you didn't consider it stdlib ready yet). The clearer separation of the build and import steps definitely sounds promising, as the current implicit approach poses challenges getting cffi extensions to play nice with wheel based distribution. Cheers, Nick.
A bientôt,
Armin. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
On 11 April 2014 10:36, Armin Rigo <arigo@tunes.org> wrote:
This would be superficial, but change the perception of CFFI to be "a preprocessor that produces C extension modules".
Thanks, that clarification helps a lot. Does this mean that "API-mode" CFFI is competing with things like swig (which is not used much these days, as far as I know) and Cython (which is used a lot in the numeric community)? ("ABI-mode" CFFI is obviously directly competing with ctypes). Paul
On Fri, Apr 11, 2014 at 2:22 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 11 April 2014 10:36, Armin Rigo <arigo@tunes.org> wrote:
This would be superficial, but change the perception of CFFI to be "a preprocessor that produces C extension modules".
Thanks, that clarification helps a lot. Does this mean that "API-mode" CFFI is competing with things like swig (which is not used much these days, as far as I know) and Cython (which is used a lot in the numeric community)? ("ABI-mode" CFFI is obviously directly competing with ctypes).
Yes.
Hi, On 11 April 2014 19:55, Maciej Fijalkowski <fijall@gmail.com> wrote:
Thanks, that clarification helps a lot. Does this mean that "API-mode" CFFI is competing with things like swig (which is not used much these days, as far as I know) and Cython (which is used a lot in the numeric community)? ("ABI-mode" CFFI is obviously directly competing with ctypes).
Yes.
Funny how a high-level shift --- not really changing anything under the hood --- changes how we look at a project :-) A bientôt, Armin.
On Wed, Apr 09, 2014 at 09:08:04PM -0400, Guido van Rossum wrote:
To anyone who took notes at the language summit at PyCon today, even if you took them just for yourself, would you mind posting them here? It would be good to have some kind of (informal!) as soon as possible, before we collectively forget. You won't be held responsible for correctness.
From a 30-second check, the recordings I made on my laptop are listenable (though the audio levels in the morning were too low). We probably don't want to post them publicly, just because they haven't been listened to. I provided them to Kushal Das, who was taking notes; if other attendees want them for reference, please let me know.
--amk
Guido van Rossum, 10.04.2014 03:08:
- Jukka Lehtosalo gave a talk and answered questions about mypy, his design and implementation of pragmatic type annotations (no new syntax required, uses Python 3 function annotations).
FWIW, signature type annotations aren't enough for a static compiler like Cython, which also benefits from local and global variable declarations, static functions, etc. However, we initially discussed this feature in the project some five years ago or so and never actually implemented it, so I finally decided to add support for it to Cython. There already was a way to provide static Cython/C type declarations in pure Python code, also for function arguments, but it's nice to have a way that is also naturally runtime inspectable in the signature. It essentially looks like this now: def func(plain_python_type: dict, named_python_type: 'list', explicit_python_type: {'type': dict}, explicit_named_python_type: {'type': 'tuple'}, explicit_c_type: {'ctype': 'int'}): ... Pretty straight forward, I think. Stefan
Stefan Behnel, 12.04.2014 19:11:
Guido van Rossum, 10.04.2014 03:08:
- Jukka Lehtosalo gave a talk and answered questions about mypy, his design and implementation of pragmatic type annotations (no new syntax required, uses Python 3 function annotations).
FWIW, signature type annotations aren't enough for a static compiler like Cython, which also benefits from local and global variable declarations, static functions, etc. However, we initially discussed this feature in the project some five years ago or so and never actually implemented it, so I finally decided to add support for it to Cython. There already was a way to provide static Cython/C type declarations in pure Python code, also for function arguments, but it's nice to have a way that is also naturally runtime inspectable in the signature.
It essentially looks like this now:
def func(plain_python_type: dict, named_python_type: 'list', explicit_python_type: {'type': dict}, explicit_named_python_type: {'type': 'tuple'}, explicit_c_type: {'ctype': 'int'}): ...
It may not be obvious to everyone, so I guess I should add a comment on why this wasn't considered important enough during the last five years to implement it before (and that didn't change much now). One thing I learned in the Cython project is that it's often a bad idea to statically type input arguments at all. The reason is that it violates the principle of being liberal with input (but strict with output). There are cases where this is ok, e.g. using the C type "double" is perfectly ok most of the time, because it's not (really) range restricting and all numeric-ish Python types can happily coerce to it. Using integer types is ok only if their restricted range really is enough, but can lead to the Py2.5 problem of having too small integer types if things grow larger. Similarly, using self-defined (extension) types to type input arguments is ok because at the native level, the code will almost certainly depend on their internal implementation details, so accepting anything else here would be useless and wrong (CPython does the same for "self" type checking, for example). However, what I've often seen people do is to write something like the above example, i.e. they use explicit Python types for input arguments. This leads to an overly narrow API that rejects lots of otherwise acceptable input types. What people usually want as input type is something like "iterable", or "mapping". What many users end up writing instead is "list" or "dict". The reason is that in Python code, input really *is* a list or a dict most of the time, and at the C level (to which Cython translates), list and dict have (limited) performance advantages, whereas "iterable" and "mapping" do not. So using something like the ABC types Iterable or Mapping for static typing is almost completely useless, except for documentation purposes (and maybe a bit earlier type checking, in some cases). The point where this is extremely visible is string/bytes input. I've seen many people actually use "str" or "bytes" to type their input. In Python 2, "str" is a very bad idea, because it excludes "unicode". In Py3, that's less of a problem, because all reasonable string input really is "str" (or, more rarely, a subtype). In both Py2 and Py3, however, statically typing input as "bytes" is a *very* bad idea, because it excludes everything bytes-ish: bytearray, memoryview, buffers. Many users don't see this need at first. While using buffers instead is easy enough in Cython, getting it right for a specific use case and properly using it to interface with external native libraries is less straight forward and requires more thought than just writing "I want bytes". So, what I've learned from seven years of Cython is that static typing in signatures is actually less interesting than you might think at first sight. It might be ok for documentation purposes, although its verboseness makes that also a bit questionable. But for actual input checking, there's substantially more than just the (generic) type, even at the C level, so users who properly understand the problem don't use static argument typing in many cases and instead write their own input validation, conversion, normalisation code. Or just let the code raise an exception if the input doesn't work the way it's being used. Errors don't always have to be raised at signature evaluation time. Stefan
On Apr 13, 2014, at 12:48 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Stefan Behnel, 12.04.2014 19:11:
So, what I've learned from seven years of Cython is that static typing in signatures is actually less interesting than you might think at first sight. It might be ok for documentation purposes, although its verboseness makes that also a bit questionable.
You raise a valid point that type hinting a dict instead of a Mapping is likely to create an overly limiting API. This sort of error is however quite easy to fix forward. To counter, we’ve had multiple data points during the summit suggesting that duck typing is not really that often used in production code. In other words, after initial prototyping and testing, an API of a callable gets settled and is later used with a very limited number of types. Of course, fuzzy terms like “often”, “seldom”, and “limited” suggest that we don’t have hard data here. That being said, there is a clear need to improve our ability to lint code, refactor code easier, and document the contracts better. This has been voiced by various conference attendees and the consensus is tremendous. Moreover, this need has been proven by owners of large codebases like Google, Facebook, and Microsoft by creating optionally typed variants of known dynamic languages (namely Dart, Hack and TypeScript). For Python specifically the existence of prototypes like Mypy, obiwan, or pytypedecl shows that there is growing interest in standardizing this functionality. Therefore, I'm drafting a PEP for opt-in type hints with optional checks at runtime and by static analysis. Will post the initial form in the upcoming days, and then we can go paint that bike shed. I'm aware that great care here is needed for the outcome to feel Pythonic and not overwhelming. Don't worry before you get your hands on the draft, though. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev
On 4/13/2014 4:11 AM, Łukasz Langa wrote:
On Apr 13, 2014, at 12:48 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Stefan Behnel, 12.04.2014 19:11:
So, what I've learned from seven years of Cython is that static typing in signatures is actually less interesting than you might think at first sight. It might be ok for documentation purposes, although its verboseness makes that also a bit questionable.
You raise a valid point that type hinting a dict instead of a Mapping is likely to create an overly limiting API. This sort of error is however quite easy to fix forward.
To counter, we’ve had multiple data points during the summit suggesting that duck typing is not really that often used in production code. In other words, after initial prototyping and testing, an API of a callable gets settled and is later used with a very limited number of types.
For public library code, where the use case is not known, apis should usually be as generic as sensible. For private library code, I can imagine that apis are and possibly even should be limited to classes actually used. There naturally is a bit of a bias here for public code.
Of course, fuzzy terms like “often”, “seldom”, and “limited” suggest that we don’t have hard data here. That being said, there is a clear need to improve our ability to lint code, refactor code easier, and document the contracts better. This has been voiced by various conference attendees and the consensus is tremendous.
This need applies to public as well as private code. Private code is a good place for experiments.
For Python specifically the existence of prototypes like Mypy, obiwan, or pytypedecl shows that there is growing interest in standardizing this functionality. Therefore, I'm drafting a PEP for opt-in type hints with optional checks at runtime and by static analysis. Will post the initial form in the upcoming days, and then we can go paint that bike shed.
I look forward to it.
I'm aware that great care here is needed for the outcome to feel Pythonic and not overwhelming. Don't worry before you get your hands on the draft, though.
I'm not. PEPs routinely get revised, and ofter improved, no matter the author. -- Terry Jan Reedy
On Sun, 13 Apr 2014 15:59:36 -0400, Terry Reedy <tjreedy@udel.edu> wrote:
On 4/13/2014 4:11 AM, Łukasz Langa wrote:
On Apr 13, 2014, at 12:48 AM, Stefan Behnel <stefan_ml@behnel.de> wrote:
Stefan Behnel, 12.04.2014 19:11:
So, what I've learned from seven years of Cython is that static typing in signatures is actually less interesting than you might think at first sight. It might be ok for documentation purposes, although its verboseness makes that also a bit questionable.
You raise a valid point that type hinting a dict instead of a Mapping is likely to create an overly limiting API. This sort of error is however quite easy to fix forward.
To counter, we’ve had multiple data points during the summit suggesting that duck typing is not really that often used in production code. In other words, after initial prototyping and testing, an API of a callable gets settled and is later used with a very limited number of types.
For public library code, where the use case is not known, apis should usually be as generic as sensible. For private library code, I can imagine that apis are and possibly even should be limited to classes actually used. There naturally is a bit of a bias here for public code.
The way *I* heard it was that types of the input arguments did not, in general, change *during the running of an application*. That doesn't mean that the types wouldn't be different in a different application, which means that in a library that duck types, duck typing is indeed used in production, it's just that the types don't *change* for a production application. So, yeah, pretty much what Terry said about library code versus application code. That is, after all, what duck typing is about, and there is a *reason* we use it. --David
On 04/13/2014 04:58 PM, R. David Murray wrote:
On Sun, 13 Apr 2014 15:59:36 -0400, Terry Reedy wrote:
On 4/13/2014 4:11 AM, Łukasz Langa wrote:
On Apr 13, 2014, at 12:48 AM, Stefan Behnel wrote:
So, what I've learned from seven years of Cython is that static typing in signatures is actually less interesting than you might think at first sight. It might be ok for documentation purposes, although its verboseness makes that also a bit questionable.
You raise a valid point that type hinting a dict instead of a Mapping is likely to create an overly limiting API. This sort of error is however quite easy to fix forward.
To counter, we’ve had multiple data points during the summit suggesting that duck typing is not really that often used in production code. In other words, after initial prototyping and testing, an API of a callable gets settled and is later used with a very limited number of types.
For public library code, where the use case is not known, apis should usually be as generic as sensible. For private library code, I can imagine that apis are and possibly even should be limited to classes actually used. There naturally is a bit of a bias here for public code.
The way *I* heard it was that types of the input arguments did not, in general, change *during the running of an application*. That doesn't mean that the types wouldn't be different in a different application, which means that in a library that duck types, duck typing is indeed used in production, it's just that the types don't *change* for a production application. So, yeah, pretty much what Terry said about library code versus application code. That is, after all, what duck typing is about, and there is a *reason* we use it.
Personal experience: I have my own copy of paramiko because it type checks for strings, and I routinely use a str-subclass. -- ~Ethan~
On Apr 19, 2014, at 11:05 AM, Ethan Furman <ethan@stoneleaf.us> wrote:
Personal experience: I have my own copy of paramiko because it type checks for strings, and I routinely use a str-subclass.
I have had that kind of problem myself in the past. Most of the time the core issue wasn’t type checking, it was how (badly) it was implemented. `assert obj.__class__ == list` being my favorite one (especially when any iterable would work just fine). -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev
Here, a week later, are some of my thoughts from the summit, for the record: 2.8: The issue of a hyptothetical 2.8 never fails to entertain. However, I noticed that there seem to be at least two distinct missions of such a thing. 1. An aid in the conversion from 2.x series to 3.x series. Enabling a bunch of warnings and such by default. Perhaps allowing 3.x syntax in some places without fuss. The problem with this idea is that it is pointless. Why would anyone want to upgrade from 2.7 to 2.8 if all they get is some new warnings for 3.x? If people are willing to make a version upgrade just to get new warnings (i.e. no immediate feature benefit) they might as well go directly to 3.x and be done with it. 2. Feature enhancement to 2.8. Take a robust and popular version of python and add some of the language goodies that have been added to 3.x and that don’t have an inherent 3.x aspect. Yield from. New exception model. Stdlib enhancements such as futures. The argument goes like this: We have a very popular platform out there with lots of momentum. People want incremental enhancements to it. Why not give them what they want? Bread and games and all that? A Rockband cannot stay cooped up in a studio producing experimental concept albums all the time. That is death. Sometimes it needs to go on tour and play old hits for the fans! 3.5 features When asked what should we aim for in 3.5, there were mostly some very minor incremental changes suggested, IIRC. In my opinion, the reason 3.x has not caught on is that there is no real carrot there. There is no new vision, no killer feature. Nothing that a programmer sees and makes him say “Yeah! I want to program my next project using this feature, it will be super!”. In my opinion we should be thinking more boldly. Either for 3.x or for a version 4. We should be taking the language to a new level. Thinking about evolving the language. New paradigms. Look at what C# is doing, with each language revision. Look at Go. I’m no CS but here are some ideas on stuff we could visit: 1. Code blocks as a core language construct. Re-implement context managers as block executors. We shouldn’t let details such as syntax questions distract us. That’s like saying that we can’t eat spaghetti because our Italian is so poor. Proper code blocks would open up new avenues for exploration of expressability and paradigms. 2. Concurrency primitives built into the language. Again, see C# with its “async” keyword (a feature we’ve experimented with in stacklesslib, see e.g. stacklesslib.async in https://bitbucket.org/stackless-dev/stacklesslib ). Look at Go with its channels and more importantly, the select feature. ( see goless, http://goless.readthedocs.org/en/latest/index.html a 2014 sprint project). Don’t get distracted by the GIL. Concurrency is as much about orchestration of operations as it is about parallel execution of code. Let’s learn from the success of stackless, gevent, go, and build on top of it by absorbing tried and tested research from more than 30 years of CS. These are the immediate ideas rolling off the top of my head. Notice how I don’t mention “removing the GIL” here since that is not a “language feature” as such, not something inspiring new thinking and invention. Of course a non-GIL implementation is also desirable, even if it would involve completely rethinking the C API. For a version 4 of python. But I think we thinking beyond that, even. Let’s keep on truckin’! K From: Python-Dev [mailto:python-dev-bounces+kristjan=ccpgames.com@python.org] On Behalf Of Guido van Rossum Sent: 10. apríl 2014 01:08 To: Python-Dev Subject: [Python-Dev] Language Summit notes To anyone who took notes at the language summit at PyCon today, even if you took them just for yourself, would you mind posting them here? It would be good to have some kind of (informal!) as soon as possible, before we collectively forget. You won't be held responsible for correctness. )
On 18 April 2014 08:04, Kristján Valur Jónsson <kristjan@ccpgames.com> wrote:
Here, a week later, are some of my thoughts from the summit, for the record: 1. An aid in the conversion from 2.x series to 3.x series. Enabling a bunch of warnings and such by default. Perhaps allowing 3.x syntax in some places without fuss. The problem with this idea is that it is pointless. Why would anyone want to upgrade from 2.7 to 2.8 if all they get is some new warnings for 3.x? If people are willing to make a version upgrade just to get new warnings (i.e. no immediate feature benefit) they might as well go directly to 3.x and be done with it.
Right, this is much better handled through tools like linters, import hooks, custom interpreters, additional support modules, etc. I think http://python-future.org/ represents the current state of the art, but there are still a lot of opportunities for improvements here.
2. Feature enhancement to 2.8. Take a robust and popular version of python and add some of the language goodies that have been added to 3.x and that don’t have an inherent 3.x aspect. Yield from. New exception model. Stdlib enhancements such as futures. The argument goes like this: We have a very popular platform out there with lots of momentum. People want incremental enhancements to it. Why not give them what they want? Bread and games and all that? A Rockband cannot stay cooped up in a studio producing experimental concept albums all the time. That is death. Sometimes it needs to go on tour and play old hits for the fans!
Do you know how much work a new Python 2.x release creates for people? All the redistributors have to update, books get outdated, a new wrinkle gets added to the compatibility matrix for everyone. A new Python 2.x release is simple untenable at this point in the transition - it's a *massively* expensive way to achieve things that can be achieved more cheaply in other ways. Take yield from, for example. Hy is able to compile *LISP* syntax to Python AST structures. PEP 380 includes a semantic expansion of yield from in terms of yield. Is it *really* impossible to get "yield from" based code running in Python 2.6? Or have people just assumed it's not possible and never even tried, because the idea of using import hooks to backport syntax to earlier feature releases is too novel? (importlib includes tools to make this kind of thing relatively straightforward, and there's nothing inherently impossible about using importlib to write import hooks in Python 2 - it's just a matter of having to do the backport first, and Eric Snow is now seriously considering doing exactly that. Hit us up on import-sig if you think that might be interesting to you)
3.5 features
When asked what should we aim for in 3.5, there were mostly some very minor incremental changes suggested, IIRC. In my opinion, the reason 3.x has not caught on is that there is no real carrot there. There is no new vision, no killer feature. Nothing that a programmer sees and makes him say “Yeah! I want to program my next project using this feature, it will be super!”.
I *really* wish folks from North America, Europe and other regions where 8-bit encodings can handle their native language and where Anglicisation of terms to fit them into the ASCII identifier restriction poses no barrier to communication would stop trotting out this "no killer feature in Python 3" canard. While it is *possible* to write internationalised and localised applications in it, Python 2's Unicode support is so broken that some people can't even run the interpreter from their home directory because it can't cope with their username. Python 3 makes Unicode support in the interpreter pervasive (just as it is in the JVM and CLR), and we've spent the last 5+ years dealing with the latent Unicode handling defects this uncovered in the standard library, fixing mistakes we made in the initial transition, and figuring out how to get the new system to play nice with the POSIX "text is just bytes in a particular encoding" model, as well as figuring out what we really lost in the transition when it comes to binary data manipulation. If anyone is *ever* tempted to utter the words "Python 3 has no killer feature" without immediately following it up with the "for me" qualifier, please go read this post about the creation of a Portuguese version of Stack Overflow: http://blog.stackoverflow.com/2014/02/cant-we-all-be-reasonable-and-speak-en... Late last year, Alex Gaynor made the insightful observation that we write software either for the users we have or for the users we want. Python 3 is the software we're writing for the users we want. Python 2.7 long term maintenance, PyPy, standard library backports, transition libraries and tools, packaging ecosystem improvements, Python 3 features designed primarily to easy migration from Python 2, PyPI modules that run on both Python 2 & 3 - that's the software we and many other people are writing for the users we have. Sure, there's not yet a killer Python 3 feature for many developers that *were already happy with Python 2.7* (although matrix multiplication syntax in 3.5 will likely reach that bar for a subset of users, and chained exceptions are already pretty amazing for debugging obscure failures that then run into broken error handlers). That's fine - that's why Python 2.7 is now being supported upstream out to 2020. It's also why lots of things are already available as backports on PyPI (if any of us as core devs identify a Python 3 library that *we* think is a killer feature, we'll often maintain a backport for it, since many of us are still maintaining Python 2 systems in our day jobs), and we're going to work on ensuring that those are easier to access for new users. Two of the best projects people can actually work on that front are actually https://github.com/python/pythondotorg and http://packaging.python.org/, to help enable them to become effective gateways to the entire Python *ecosystem*, rather than having people downloading CPython and assuming the reference interpreter and the standard library are all they have available to them.
In my opinion we should be thinking more boldly. Either for 3.x or for a version 4. We should be taking the language to a new level. Thinking about evolving the language. New paradigms. Look at what C# is doing, with each language revision. Look at Go. I’m no CS but here are some ideas on stuff we could visit:
1. Code blocks as a core language construct. Re-implement context managers as block executors. We shouldn’t let details such as syntax questions distract us. That’s like saying that we can’t eat spaghetti because our Italian is so poor.
When it comes to blocks, the syntax isn't a distraction - it's the fundamental framing of what the construct is and how it should be used. How does a block get used to define a sorting key function? How does it get broadcast over a NumPy array? How does it get distributed to multiple threads, processes or greenlets, or passed to an IO scheduler? How does it differ from an ordinary closure? How does it relate to loops and other constructs? There is a large existing constellation of concepts that blocks need to be fitted into, and any successful proposal for adding blocks to the language will require giving that process the respect and attention it deserves.
Proper code blocks would open up new avenues for exploration of expressability and paradigms.
This won't happen without a full usability study, but I've been speaking to Fernando Perez and Greg Wilson about some vague concepts for doing exactly that. I hope to be able to explore that possibility further at SciPy in July - Greg laments the lack of "evidence based language design" and I think the strong scientific presence in the Python community provides us with a genuine opportunity to move in that direction when it comes to controversial changes like code blocks.
2. Concurrency primitives built into the language. Again, see C# with its “async” keyword (a feature we’ve experimented with in stacklesslib, see e.g. stacklesslib.async in https://bitbucket.org/stackless-dev/stacklesslib ). Look at Go with its channels and more importantly, the select feature. ( see goless, http://goless.readthedocs.org/en/latest/index.html a 2014 sprint project). Don’t get distracted by the GIL. Concurrency is as much about orchestration of operations as it is about parallel execution of code. Let’s learn from the success of stackless, gevent, go, and build on top of it by absorbing tried and tested research from more than 30 years of CS.
I'm personally letting Armin Rigo spearhead that effort (via his STM work), as I believe PyPy provides a better foundation for that exploration. It also doesn't make sense to me to do this at a language level until we resolve the blocks question, since that opens up many new possibilities for parallelisation that won't even need new syntax at that point.
These are the immediate ideas rolling off the top of my head. Notice how I don’t mention “removing the GIL” here since that is not a “language feature” as such, not something inspiring new thinking and invention. Of course a non-GIL implementation is also desirable, even if it would involve completely rethinking the C API. For a version 4 of python. But I think we thinking beyond that, even.
If people are genuinely interested in building and supporting the Python *community*, stop thinking about tinkering with the *language*. Tinkering with the language is *fun* (that's why I still do it), but the most interesting innovations in the Python world with the largest near term impact *aren't going to happen in the reference interpreter*. Our cycle times are too long, especially once commercial redistributors are taken into account, and the ongoing popularity of Python 2.7 shows that the language itself reached "good enough" status for most use cases years ago. As technologists we like to tinker with out toys, but it's important to remember that *Excel spreadsheets* are far more dominant as a programming paradigm than any open source programming language. So folks looking to core development as the engine room of Python's growth are *looking in the wrong place*. We laid the foundation (and will continue to sustain and evolve it), but the real growth drivers have now moved further out to things like the work the PyPA is doing to make upstream packaging easier to consume, the work Travis Oliphant and others are doing to make the scientific Python stack easier to consume, the work Fernando Perez is coordinating around IPython and IPython notebook, the work Titus Brown et al are doing around reproducable science, the work Greg Wilson and Mozilla Science Labs are doing around Software Carpentry, the work Van Lindberg and others are putting into opening up the PSF, the work Jesse Noller has been driving around opening up python.org to more contributors, the work Jessica McKellar and more are doing around getting Python into high school curricula in the US, the work the Raspberry Pi Foundation are doing in the UK, the work around the core mentorship program, GSoC and the Gnome Outreach Program for Women, the work to build the regional network of Python conferences (I'm told we're currently running at a rate of something like 2 new PyCons per week!), the work folks like me, Guido, Steve Dower, Jesse and more are doing to better engage with corporate users and inviting them to more directly support the sustainability of the upstream Python community (on *our* terms), the work PyLadies is doing around education and outreach, the work Steve Holden is now doing around improving the ready availability of Python training courses, and on, and on, and on. We are *far* past the point where any specific language feature will be the defining aspect of language adoption - it's all about the ecosystem, and has been for years. Discoverability and usability and building relationships with external groups - that is how you make the leap across the chasm from early adopters to mass market popularity. Not tinkering with syntactic details - do the latter for fun and to help people years down the track, but do other things (like helping with packaging or the core development workflow tools or communications projects like Mailman 3 and the HyperKitty archiver) if the long game doesn't excite you and you would like to make a more immediate impact on the world. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
-----Original Message----- From: Nick Coghlan [mailto:ncoghlan@gmail.com]
2. Feature enhancement to 2.8. Take a robust and popular version of python and add some of the language goodies that have been added to 3.x and that don’t have an inherent 3.x aspect. Yield from. New exception model. Stdlib enhancements such as futures. The argument goes like this: We have a very popular platform out there with lots of momentum. People want incremental enhancements to it. Why not give them what they want? Bread and games and all that? A Rockband cannot stay cooped up in a studio producing experimental concept albums all the time. That is death. Sometimes it needs to go on tour and play old hits for the fans!
Do you know how much work a new Python 2.x release creates for people? All the redistributors have to update, books get outdated, a new wrinkle gets added to the compatibility matrix for everyone. A new Python 2.x release is simple untenable at this point in the transition The key word here, "transition". I’m not sure that everyone wants to transit. This may be the core of the issue. I know that this has been hashed to death before. Never the less the pressure is there, I think and it would, I predict, be a crowd-pleaser.
- it's a *massively* expensive way to achieve things that can be achieved more cheaply in other ways. More cheaply from "our" point of view, perhaps.
Take yield from, for example. Hy is able to compile *LISP* syntax to Python AST structures. PEP 380 includes a semantic expansion of yield from in terms of yield. Is it *really* impossible to get "yield from" based code running in Python 2.6? Or have people just assumed it's not possible and never even tried, because the idea of using import hooks to backport syntax to earlier feature releases is too novel?
Some things are better done as language features than as complicated reverse hacks. You keep saying that this and that can be done with add-on modules. I think you underestimate the practical and psychological barrier towards those things. Every new dependency on a package In some package directory is a new complication in a project. It is something new you have t get, something requiing yet another virtualenv directory, something unblessed. Another factor is the simply sheer size of Pypi by now. How do you find things? How do you even guess that things like "yield from" might be available as a package to pip install? I know that this can be improved and tat there is work in process in doing that but the PyPi is still not core python and there is a gap that must be bridged for a user to start looking for solutions there.
3.5 features
When asked what should we aim for in 3.5, there were mostly some very minor incremental changes suggested, IIRC. In my opinion, the reason 3.x has not caught on is that there is no real carrot there. There is no new vision, no killer feature. Nothing that a programmer sees and makes him say “Yeah! I want to program my next project using this feature, it will be super!”.
I *really* wish folks from North America, Europe and other regions where 8- bit encodings can handle their native language and where Anglicisation of terms to fit them into the ASCII identifier restriction poses no barrier to communication would stop trotting out this "no killer feature in Python 3" canard. I intentionally didn't mention this because it is like the GIL. It is a technical feature of the language, a refinement if you will. But not a new thing in terms of language evolution. Look, I have no disregard for the importance of this, myself coming from a non-ascii language. I also worked on the internationalization of EVE, many years ago, and am familiar with the annoyance of implicit Unicode conversions. I work in a company that uses unicode in all its products, new and old. The other day a new stand alone service was developed. I suggested to the developer that he might want to use Python 3 because it is unicode through and through. He just shrugged, said it wasn't an issue. He'll store the relevant database tables as unicode, get unicode out of the stuff, encode unicode in json and everything will just work. While I'm not saying that the new model is not better (I think it is) it does come with some baggage, particularly in how it has been more cumbersome to work with bytes. But anyway, this is why I didn"t mention unicode and why I don't count it as a killer feature.
While it is *possible* to write internationalised and localised
applications in it, Python 2's Unicode support is so broken that some people can't even run the interpreter from their home directory because it can't cope with their username. Years ago we implemented fixes to that for python 2.5. core dev wasn't interested :).
If anyone is *ever* tempted to utter the words "Python 3 has no killer feature" without immediately following it up with the "for me" qualifier, please go read this post about the creation of a Portuguese version of Stack Overflow: http://blog.stackoverflow.com/2014/02/cant-we-all-be-reasonable-and- speak-english/ Look, unicode is important. Just as having an automatic choke in a car is important. Do you remember manual choke levers in cars? I’m sure the Ford Falcon had one. Removing the choke is a refinement. But not a quantum leap.
Late last year, Alex Gaynor made the insightful observation that we write software either for the users we have or for the users we want. Python 3 is the software we're writing for the users we want. Python 2.7 long term maintenance, PyPy, standard library backports, transition libraries and tools, packaging ecosystem improvements, Python 3 features designed primarily to easy migration from Python 2, PyPI modules that run on both Python 2 & 3 - that's the software we and many other people are writing for the users we have.
1. Code blocks as a core language construct. Re-implement context managers as block executors. We shouldn’t let details such as syntax questions distract us. That’s like saying that we can’t eat spaghetti because our Italian is so poor.
When it comes to blocks, the syntax isn't a distraction - it's the fundamental framing of what the construct is and how it should be used. How does a block get used to define a sorting key function? How does it get broadcast over a NumPy array? How does it get distributed to multiple threads, processes or greenlets, or passed to an IO scheduler? How does it differ from an ordinary closure? How does it relate to loops and other constructs?
There is a large existing constellation of concepts that blocks need to be fitted into, and any successful proposal for adding blocks to the language will require giving that process the respect and attention it deserves.
You're getting bogged down in implementation details, letting side issues distract you from the core goal. We could for example start with a vertical slice. Enable code blocks for context managers. See how we solve the syntax (already done) and explore the possibilities that it gives us. Then look at sorting functions. One step at a time. I'm sure we'll find a way, unless we are already convinced that it is impossible. But anyway, I mentioned code blocks because it is such an obvious thing to explore and has been an ongoing chestnut on python-dev for eons.
This won't happen without a full usability study, but I've been speaking to Fernando Perez and Greg Wilson about some vague concepts for doing exactly that. I hope to be able to explore that possibility further at SciPy in July - Greg laments the lack of "evidence based language design" and I think the strong scientific presence in the Python community provides us with a genuine opportunity to move in that direction when it comes to controversial changes like code blocks. The literature is full of them and there are languages out there that use them. This isn't even pioneering CS :(
If people are genuinely interested in building and supporting the Python *community*, stop thinking about tinkering with the *language*.
This is python-dev. It is about the *language*. CPython is the reference implementation. It is where most of that are passionate about the language operate. I'm trying to instill a passion for boldness here, to not be happy with letting the evolution of python die some beurocratic death. I'm sure we can come up with ways of making Python continue to innovate, while still maintaining all this *community*.
So folks looking to core development as the engine room of Python's growth are *looking in the wrong place*. We laid the foundation (and will continue to sustain and evolve it), but the real growth drivers have now moved further out to things like the work the PyPA is doing to make upstream packaging easier to consume, the work Travis Oliphant and others are doing to make the scientific Python stack easier to consume, the work Fernando Perez is coordinating around IPython and IPython notebook, the work Titus Brown et al are doing around reproducable science, the work Greg Wilson and Mozilla Science Labs are doing around Software Carpentry, the work Van Lindberg and others are putting into opening up the PSF, the work Jesse Noller has been driving around opening up python.org to more contributors, the work Jessica McKellar and more are doing around getting Python into high school curricula in the US, the work the Raspberry Pi Foundation are doing in the UK, the work around the core mentorship program, GSoC and the Gnome Outreach Program for Women, the work to build the regional network of Python conferences (I'm told we're currently running at a rate of something like 2 new PyCons per week!), the work folks like me, Guido, Steve Dower, Jesse and more are doing to better engage with corporate users and inviting them to more directly support the sustainability of the upstream Python community (on *our* terms), the work PyLadies is doing around education and outreach, the work Steve Holden is now doing around improving the ready availability of Python training courses, and on, and on, and on. This is all super work, and laudable. But I don't understand your point. Because all this python related work is being done, does that mean that we, as core python developers, should not continue to innovate on the language? People have different interests and aspirations and everyone should be able to find a slot in our great community where he feels he can contribute at his best. I have already mentioned C# above. This is a language probably as big as Python. Yet with every revision of it, it seems to adopt previously experimental language features into the language core, making it even more powerful. At the same time, it is also being used by lots of people. These need not be mutually exclusive goals, mass marked popularity and language innovation.
It's like Country and Western: We can have both kinds. Let's do it. Let's have fun! K
On Fri, 18 Apr 2014 12:04:10 +0000 Kristján Valur Jónsson <kristjan@ccpgames.com> wrote:
2. Feature enhancement to 2.8. Take a robust and popular version of python and add some of the language goodies that have been added to 3.x and that don’t have an inherent 3.x aspect. Yield from. New exception model. Stdlib enhancements such as futures. The argument goes like this: We have a very popular platform out there with lots of momentum. People want incremental enhancements to it. Why not give them what they want? Bread and games and all that? A Rockband cannot stay cooped up in a studio producing experimental concept albums all the time. That is death. Sometimes it needs to go on tour and play old hits for the fans!
I don't think we have recent download numbers since the Website overhaul (do we?), but Python 3 isn't an "experimental concept language" anymore (it hasn't been since 3.3 or 3.2, I'd say). Regards Antoine.
On Apr 18, 2014, at 2:31 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Fri, 18 Apr 2014 12:04:10 +0000 Kristján Valur Jónsson <kristjan@ccpgames.com> wrote:
2. Feature enhancement to 2.8. Take a robust and popular version of python and add some of the language goodies that have been added to 3.x and that don’t have an inherent 3.x aspect. Yield from. New exception model. Stdlib enhancements such as futures. The argument goes like this: We have a very popular platform out there with lots of momentum. People want incremental enhancements to it. Why not give them what they want? Bread and games and all that? A Rockband cannot stay cooped up in a studio producing experimental concept albums all the time. That is death. Sometimes it needs to go on tour and play old hits for the fans!
I don't think we have recent download numbers since the Website overhaul (do we?), but Python 3 isn't an "experimental concept language" anymore (it hasn't been since 3.3 or 3.2, I'd say).
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
Fastly logs are stored in Dreamhost so we could make those numbers if they aren’t already available. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
participants (22)
-
A.M. Kuchling -
Antoine Pitrou -
Armin Rigo -
Barry Warsaw -
Donald Stufft -
Eli Bendersky -
Eric Snow -
Ethan Furman -
Guido van Rossum -
Kristján Valur Jónsson -
Kushal Das -
Maciej Fijalkowski -
Nick Coghlan -
Paul Moore -
R. David Murray -
Ryan Gonzalez -
Senthil Kumaran -
Stefan Behnel -
Stephen J. Turnbull -
Terry Reedy -
Thomas Wouters -
Łukasz Langa