Mailman 3 More "ensure*" packages - Python-ideas

More "ensure*" packages

Steve Dower

13 Aug 2015 13 Aug '15

5:06 p.m.

I'd like to propose expanding the list of 3rd-party packages we bundle and install by default. (Obviously this does not apply to platforms that repackage Python and can do whatever they want, but on Windows and Mac we are fully responsible for these.) Currently, we bundle pip (and some of its dependencies - let's avoid that particular discussion right now please, it's on python-dev) and install it by default in a way that lets users easily update to the latest version. Including pip in the standard library would lock users into a specific version for the lifetime of that Python version, which would be a bad thing. From my point-of-view, this has been very successful in Python 2.7, 3.4 and will also be successful in 3.5. For Python 3.6, I'd like to do a similar thing with: * requests * tkinter (including tcl/tk, IDLE, and other dependencies) Given the language summit discussion at PyCon this year, I think requests is easy to justify. (Quick summary for those who weren't there: we'd love to include requests in the stdlib, but it's too important and needs much more frequent updates.) Preinstalling a given version in a way that allows updates (and maybe attempting an update on installation) sounds great to me. tkinter is worth more discussion :) For the remainder of this email, I'll use "tkinter" as shorthand to refer to Tcl, Tk, Tix, _tkinter, tkinter, idlelib/IDLE, PyDoc, turtledemo and any other dependencies or dependents that I missed. In my experience, few Python scripts depend on or assume tkinter is available. tkinter is already an optional item in the Windows installer (maybe Mac too? I don't know) and there are certainly installations of Python out there that don't have it. From this side, nothing would actually change by installing tkinter into site-packages rather than Lib. (One impact may be the start menu shortcuts for IDLE and PyDoc, but provided the entry points into those tools are kept stable we can continue adding shortcuts from the installer. People who omit tkinter and then install it later would not get shortcuts. But since they omitted it from the installer, they probably don't want them - they likely just got a package that has tkinter as a dependency.) IDLE is already allowed to make enhancements in maintenance branches (https://www.python.org/dev/peps/pep-0434/), and we have recently received patches that are to be applied to *four* branches. The freedom to enhance IDLE is greatly improved by making it a PyPI installable package and disconnecting it from the stdlib's schedule. How this would actually be structured is up for discussion. I believe the change can be made without sacrificing anything, and the resulting flexibility will be worth it. Thoughts? Cheers, Steve

Show replies by date

Joonas Liik

13 Aug 13 Aug

5:31 p.m.

...

Currently, we bundle pip (and some of its dependencies - let's avoid that particular discussion right now please, it's on python-dev) and install it by default in a way that lets users easily update to the latest version. Including pip in the standard library would lock users into a specific version for the lifetime of that Python version, which would be a bad thing.

pip install --upgrade pip has worked well every time i have tried it.. I would like to mention that dealing with any package that has a c-extension is utter pain under windows tho, ..especially if you want somebody else to be able to run your code, how is not a dev.

Steve Dower

5:46 p.m.

On 13Aug2015 1031, Joonas Liik wrote:

...

...
Currently, we bundle pip (and some of its dependencies - let's avoid that particular discussion right now please, it's on python-dev) and install it by default in a way that lets users easily update to the latest version. Including pip in the standard library would lock users into a specific version for the lifetime of that Python version, which would be a bad thing.

pip install --upgrade pip has worked well every time i have tried it..

I would like to mention that dealing with any package that has a c-extension is utter pain under windows tho,

..especially if you want somebody else to be able to run your code, how is not a dev.

We are well aware of those issues, but it is completely off-topic for this thread. Cheers, Steve

Donald Stufft

5:35 p.m.

On August 13, 2015 at 1:08:11 PM, Steve Dower (steve.dower@python.org) wrote:

...

I'd like to propose expanding the list of 3rd-party packages we bundle and install by default. (Obviously this does not apply to platforms that repackage Python and can do whatever they want, but on Windows and Mac we are fully responsible for these.)

Currently, we bundle pip (and some of its dependencies - let's avoid that particular discussion right now please, it's on python-dev) and install it by default in a way that lets users easily update to the latest version. Including pip in the standard library would lock users into a specific version for the lifetime of that Python version, which would be a bad thing.

From my point-of-view, this has been very successful in Python 2.7, 3.4 and will also be successful in 3.5. For Python 3.6, I'd like to do a similar thing with:

* requests * tkinter (including tcl/tk, IDLE, and other dependencies)

Given the language summit discussion at PyCon this year, I think requests is easy to justify. (Quick summary for those who weren't there: we'd love to include requests in the stdlib, but it's too important and needs much more frequent updates.) Preinstalling a given version in a way that allows updates (and maybe attempting an update on installation) sounds great to me.

tkinter is worth more discussion :) For the remainder of this email, I'll use "tkinter" as shorthand to refer to Tcl, Tk, Tix, _tkinter, tkinter, idlelib/IDLE, PyDoc, turtledemo and any other dependencies or dependents that I missed.

In my experience, few Python scripts depend on or assume tkinter is available. tkinter is already an optional item in the Windows installer (maybe Mac too? I don't know) and there are certainly installations of Python out there that don't have it. From this side, nothing would actually change by installing tkinter into site-packages rather than Lib.

(One impact may be the start menu shortcuts for IDLE and PyDoc, but provided the entry points into those tools are kept stable we can continue adding shortcuts from the installer. People who omit tkinter and then install it later would not get shortcuts. But since they omitted it from the installer, they probably don't want them - they likely just got a package that has tkinter as a dependency.)

IDLE is already allowed to make enhancements in maintenance branches (https://www.python.org/dev/peps/pep-0434/), and we have recently received patches that are to be applied to *four* branches. The freedom to enhance IDLE is greatly improved by making it a PyPI installable package and disconnecting it from the stdlib's schedule.

How this would actually be structured is up for discussion. I believe the change can be made without sacrificing anything, and the resulting flexibility will be worth it.

Thoughts?

One possible thing to look at for prior art, is what Haskell does. They don’t have a bunch of ensure* modules or anything like it, instead they have their compiler (which is like “Haskell Core” and then on top of that they layer a bunch of libraries (Called “Haskell Platform”). This platform releases every ~6 months and just includes something like 40 different libraries with it that represent common development tools and widely used libraries [1]. So I guess my question is, instead of continuing down a path where we add more ensure* style modules to the standard library, why not do something similar and have “Python the Language” and “The Python Platform”, and the platform would be the Python language + N “important” or “popular” packages. This could release on a quicker release schedule than Python itself (since it would really be more like a meta package than anything that itself got developed) and would give the ability to ship things like this without the problems that we’ve had with ensurepip. From a downstream perspective they would just package all of this stuff as normal and it would just be available as normal. We could even publish a metapackage on PyPI that had no code of it’s own, but existed simply to list all of the platform packages as dependencies (with ==) and then people could easily depend on the Python “platform” in their own code. This would essentially involve someone(s) needing to be the gatekeeper of which libraries become part of the Python platform, some small packaging shims to handle the metapackage on PyPI, and then the installer stuff for OSX and Windows (probably nothing for other OSs? Or maybe a tarball? I don’t know). [1] https://www.haskell.org/platform/contents.html ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Alex Grönholm

6:49 p.m.

13.08.2015, 20:35, Donald Stufft kirjoitti:

...

On August 13, 2015 at 1:08:11 PM, Steve Dower (steve.dower@python.org) wrote:

...
I'd like to propose expanding the list of 3rd-party packages we bundle and install by default. (Obviously this does not apply to platforms that repackage Python and can do whatever they want, but on Windows and Mac we are fully responsible for these.)

Currently, we bundle pip (and some of its dependencies - let's avoid that particular discussion right now please, it's on python-dev) and install it by default in a way that lets users easily update to the latest version. Including pip in the standard library would lock users into a specific version for the lifetime of that Python version, which would be a bad thing.

From my point-of-view, this has been very successful in Python 2.7, 3.4 and will also be successful in 3.5. For Python 3.6, I'd like to do a similar thing with:

* requests * tkinter (including tcl/tk, IDLE, and other dependencies)

Given the language summit discussion at PyCon this year, I think requests is easy to justify. (Quick summary for those who weren't there: we'd love to include requests in the stdlib, but it's too important and needs much more frequent updates.) Preinstalling a given version in a way that allows updates (and maybe attempting an update on installation) sounds great to me.

tkinter is worth more discussion :) For the remainder of this email, I'll use "tkinter" as shorthand to refer to Tcl, Tk, Tix, _tkinter, tkinter, idlelib/IDLE, PyDoc, turtledemo and any other dependencies or dependents that I missed.

In my experience, few Python scripts depend on or assume tkinter is available. tkinter is already an optional item in the Windows installer (maybe Mac too? I don't know) and there are certainly installations of Python out there that don't have it. From this side, nothing would actually change by installing tkinter into site-packages rather than Lib.

(One impact may be the start menu shortcuts for IDLE and PyDoc, but provided the entry points into those tools are kept stable we can continue adding shortcuts from the installer. People who omit tkinter and then install it later would not get shortcuts. But since they omitted it from the installer, they probably don't want them - they likely just got a package that has tkinter as a dependency.)

IDLE is already allowed to make enhancements in maintenance branches (https://www.python.org/dev/peps/pep-0434/), and we have recently received patches that are to be applied to *four* branches. The freedom to enhance IDLE is greatly improved by making it a PyPI installable package and disconnecting it from the stdlib's schedule.

How this would actually be structured is up for discussion. I believe the change can be made without sacrificing anything, and the resulting flexibility will be worth it.

Thoughts?

One possible thing to look at for prior art, is what Haskell does. They don’t have a bunch of ensure* modules or anything like it, instead they have their compiler (which is like “Haskell Core” and then on top of that they layer a bunch of libraries (Called “Haskell Platform”). This platform releases every ~6 months and just includes something like 40 different libraries with it that represent common development tools and widely used libraries [1].

So I guess my question is, instead of continuing down a path where we add more ensure* style modules to the standard library, why not do something similar and have “Python the Language” and “The Python Platform”, and the platform would be the Python language + N “important” or “popular” packages. This could release on a quicker release schedule than Python itself (since it would really be more like a meta package than anything that itself got developed) and would give the ability to ship things like this without the problems that we’ve had with ensurepip. From a downstream perspective they would just package all of this stuff as normal and it would just be available as normal. We could even publish a metapackage on PyPI that had no code of it’s own, but existed simply to list all of the platform packages as dependencies (with ==) and then people could easily depend on the Python “platform” in their own code.

This would essentially involve someone(s) needing to be the gatekeeper of which libraries become part of the Python platform, some small packaging shims to handle the metapackage on PyPI, and then the installer stuff for OSX and Windows (probably nothing for other OSs? Or maybe a tarball? I don’t know).

Amen to this. This is EXACTLY where I've hoped Python would go :)

...

[1] https://www.haskell.org/platform/contents.html

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Steve Dower

8:28 p.m.

Didn't see Donald's email, so I'm replying via Alex's reply. 13.08.2015, 20:35, Donald Stufft kirjoitti:

...

So I guess my question is, instead of continuing down a path where we add more ensure* style modules to the standard library, why not do something similar and have “Python the Language” and “The Python Platform”, and the platform would be the Python language + N “important” or “popular” packages. This could release on a quicker release schedule than Python itself (since it would really be more like a meta package than anything that itself got developed) and would give the ability to ship things like this without the problems that we’ve had with ensurepip. From a downstream perspective they would just package all of this stuff as normal and it would just be available as normal. We could even publish a metapackage on PyPI that had no code of it’s own, but existed simply to list all of the platform packages as dependencies (with ==) and then people could easily depend on the Python “platform” in their own code.

This would essentially involve someone(s) needing to be the gatekeeper of which libraries become part of the Python platform, some small packaging shims to handle the metapackage on PyPI, and then the installer stuff for OSX and Windows (probably nothing for other OSs? Or maybe a tarball? I don’t know).

So basically we could add a requirements.txt into the core CPython repo and installers could trigger it (with user permission) on install? We could call it "python-platform.txt", and even add an "ensurepythonplatform" module to run the install (all names are only suggestions). I'd still like to bundle wheels with the latest available versions at build so that non-networked installs can get the packages, if not necessarily the latest. This seems like a better long term approach than one-ensure-module-per-package. I feel like moving tkinter&co onto PyPI is the more controversial suggestion :) Cheers, Steve

Donald Stufft

8:48 p.m.

On August 13, 2015 at 4:29:11 PM, Steve Dower (steve.dower@python.org) wrote:

...

Didn't see Donald's email, so I'm replying via Alex's reply.

Hopefully you get this, I got some error about SPF last time I sent the email :/

...

So basically we could add a requirements.txt into the core CPython repo and installers could trigger it (with user permission) on install? We could call it "python-platform.txt", and even add an "ensurepythonplatform" module to run the install (all names are only suggestions).

I'd still like to bundle wheels with the latest available versions at build so that non-networked installs can get the packages, if not necessarily the latest. This seems like a better long term approach than one-ensure-module-per-package.

Well, I’d probably make the “Python Platform” sort of a super-installer that has inside of it the Python installer and then a requirements.txt or whatever along with whatever bundled wheels it needs. Leave the old Python installers alone and continue to generate them (as a sort of “Minimal” installation option). The benefit to having a super installer over top of the core installers, is that you can update and version this super installer independently of the Python installers, so we could do releases more often, maybe every 3-6 months or so. This would essentially just be pulling in the latest Python version, and the latest versions of each of the bundled libraries. To be clear, the bundled libraries would be bundled into the super installer itself, not into Python. The idea being, that this is something that gets layered overtop of the “Python Runtime” (aka Python the language and standard library) and doesn’t require anything special or any changes to Python itself. I *think* this will make it easier for downstream redistributors because they could implement this “Python Platform” using just a metapackage and typical dependency information and wouldn’t need to deal with any of the mess that the bundled ensurepip module has created. This metapackage would just depend on Python, and each of the included third party libraries in the platform. This isn’t really something that python-dev itself would have to do (since it requires no changes to Python itself), however there’s benefit to python-dev doing it both in the official-ness of it, and getting it put onto www.python.org and the documentation. Ideally I think we’d change the downloads to push the Python Platform for installers over the actual Python runtime.

...

I feel like moving tkinter&co onto PyPI is the more controversial suggestion :)

I don’t have an opinion specific to tkinter & co, other than I think a smaller standard library and a larger external ecosystem combined with the platform thing I described above is a better direction to go in. I’ve long wanted such a thing. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Steve Dower

9:28 p.m.

On 13Aug2015 1348, Donald Stufft wrote:

...

On August 13, 2015 at 4:29:11 PM, Steve Dower (steve.dower@python.org) wrote:

...
Didn't see Donald's email, so I'm replying via Alex's reply.

Hopefully you get this, I got some error about SPF last time I sent the email :/

I'm on a new mail server, so it's entirely possible that it's one of my settings. Can you forward the full error to me off-list?

...

...
So basically we could add a requirements.txt into the core CPython repo and installers could trigger it (with user permission) on install? We could call it "python-platform.txt", and even add an "ensurepythonplatform" module to run the install (all names are only suggestions).

I'd still like to bundle wheels with the latest available versions at build so that non-networked installs can get the packages, if not necessarily the latest. This seems like a better long term approach than one-ensure-module-per-package.

Well, I’d probably make the “Python Platform” sort of a super-installer that has inside of it the Python installer and then a requirements.txt or whatever along with whatever bundled wheels it needs. Leave the old Python installers alone and continue to generate them (as a sort of “Minimal” installation option). The benefit to having a super installer over top of the core installers, is that you can update and version this super installer independently of the Python installers, so we could do releases more often, maybe every 3-6 months or so. This would essentially just be pulling in the latest Python version, and the latest versions of each of the bundled libraries. To be clear, the bundled libraries would be bundled into the super installer itself, not into Python.

The idea being, that this is something that gets layered overtop of the “Python Runtime” (aka Python the language and standard library) and doesn’t require anything special or any changes to Python itself. I *think* this will make it easier for downstream redistributors because they could implement this “Python Platform” using just a metapackage and typical dependency information and wouldn’t need to deal with any of the mess that the bundled ensurepip module has created. This metapackage would just depend on Python, and each of the included third party libraries in the platform.

This isn’t really something that python-dev itself would have to do (since it requires no changes to Python itself), however there’s benefit to python-dev doing it both in the official-ness of it, and getting it put onto www.python.org and the documentation. Ideally I think we’d change the downloads to push the Python Platform for installers over the actual Python runtime.

It's already a fairly crowded marketplace, at least on Windows. Anaconda, Canopy, WinPython, Pythonxy and Portable Python all come to mind, but none of them have reliably replaced the official Python installer. In large part, I suspect this is because they do too much - most include the scipy stack and at least one (typically 4-5) editors. What might be interesting is if we installed the meta-package into the Lib directory rather than Lib/site-packages. That also opens up the possibility of removing old/deprecated modules from the stdlib but having an install option to restore them. Which helps break up the stdlib a bit and make the core Python install lighter, but still isn't really suitable for something like requests. I also don't see any real gain in making it a separate thing from the main installer unless you're planning 10+ packages to be in there. I would expect an incredibly small amount of packages to be available - those with no significant competitors, extremely broad uses, and completely portable. Cheers, Steve

...

...
I feel like moving tkinter&co onto PyPI is the more controversial suggestion :)

I don’t have an opinion specific to tkinter & co, other than I think a smaller standard library and a larger external ecosystem combined with the platform thing I described above is a better direction to go in. I’ve long wanted such a thing.

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Donald Stufft

9:50 p.m.

On August 13, 2015 at 5:29:15 PM, Steve Dower (steve.dower@python.org) wrote:

...

It's already a fairly crowded marketplace, at least on Windows. Anaconda, Canopy, WinPython, Pythonxy and Portable Python all come to mind, but none of them have reliably replaced the official Python installer. In large part, I suspect this is because they do too much - most include the scipy stack and at least one (typically 4-5) editors.

I think that’s also discounting the *huge* benefit of something being offered as the official Python thing. I think a lot of people are hesitant to install those other things because they come someone else, or they aren’t even aware of them because they searched for Python on google and got the official installer.

...

What might be interesting is if we installed the meta-package into the Lib directory rather than Lib/site-packages. That also opens up the possibility of removing old/deprecated modules from the stdlib but having an install option to restore them. Which helps break up the stdlib a bit and make the core Python install lighter, but still isn't really suitable for something like requests.

Right, I don’t think it’s suitable at all to install something that is typically installed from PyPI into anything other than Lib/site-packages.

...

I also don't see any real gain in making it a separate thing from the main installer unless you're planning 10+ packages to be in there. I would expect an incredibly small amount of packages to be available - those with no significant competitors, extremely broad uses, and completely portable.

Can we release new, different, installers for say, a hypothetical 3.5.0 without cutting a new release of 3.5.0 that include updated versions of the bundled software? If so, how will people know they are using the latest version of that installer? I know that requests will not be very happy being bundled like that if getting a new version is tied to a new version of Python being released. This is one of the primary benefits of a separate installer, the installer (really the set of things that get installed) gets a version number. You can release it independently of a Python release, so if requests has a security issue then you can just roll out a new version of the platform installer without that affecting Python at all. Another major benefit of a separate installer that layers ontop of Python is reducing (or eliminating) the friction it will cause with downstream redistributors. The “sort of stdlib, sort of not” status of pip is weird and has caused a bit of a problem. I’m still dealing with the fallout of that, and while we had to do it that way in order for it to work inside of virtual environments, I don’t think we need to do this that way. I also think the separation just makes way more sense, when you have ensure* it means that things like python-pip depend on Python, but then Python also depends on them. I don’t think that Python the runtime needs to depends on requests and I think that doing that is going the wrong way. What this really is, is just a collection of preinstalled packages, so treating it like that seems like a better option. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Wes Turner

11:03 p.m.

On Thu, Aug 13, 2015 at 4:50 PM, Donald Stufft wrote:

...

On August 13, 2015 at 5:29:15 PM, Steve Dower (steve.dower@python.org) wrote:

...
It's already a fairly crowded marketplace, at least on Windows. Anaconda, Canopy, WinPython, Pythonxy and Portable Python all come to mind, but none of them have reliably replaced the official Python installer. In large part, I suspect this is because they do too much - most include the scipy stack and at least one (typically 4-5) editors.

I think that’s also discounting the *huge* benefit of something being offered as the official Python thing. I think a lot of people are hesitant to install those other things because they come someone else, or they aren’t even aware of them because they searched for Python on google and got the official installer.

...
What might be interesting is if we installed the meta-package into the Lib directory rather than Lib/site-packages. That also opens up the possibility of removing old/deprecated modules from the stdlib but having an install option to restore them. Which helps break up the stdlib a bit and make the core Python install lighter, but still isn't really suitable for something like requests.

Right, I don’t think it’s suitable at all to install something that is typically installed from PyPI into anything other than Lib/site-packages.

...
I also don't see any real gain in making it a separate thing from the main installer unless you're planning 10+ packages to be in there. I would expect an incredibly small amount of packages to be available - those with no significant competitors, extremely broad uses, and completely portable.

Can we release new, different, installers for say, a hypothetical 3.5.0 without cutting a new release of 3.5.0 that include updated versions of the bundled software? If so, how will people know they are using the latest version of that installer? I know that requests will not be very happy being bundled like that if getting a new version is tied to a new version of Python being released.

This is one of the primary benefits of a separate installer, the installer (really the set of things that get installed) gets a version number. You can release it independently of a Python release, so if requests has a security issue then you can just roll out a new version of the platform installer without that affecting Python at all.

Another major benefit of a separate installer that layers ontop of Python is reducing (or eliminating) the friction it will cause with downstream redistributors. The “sort of stdlib, sort of not” status of pip is weird and has caused a bit of a problem. I’m still dealing with the fallout of that, and while we had to do it that way in order for it to work inside of virtual environments, I don’t think we need to do this that way.

I also think the separation just makes way more sense, when you have ensure* it means that things like python-pip depend on Python, but then Python also depends on them. I don’t think that Python the runtime needs to depends on requests and I think that doing that is going the wrong way. What this really is, is just a collection of preinstalled packages, so treating it like that seems like a better option.

Challenges - [ ] repeatable build scripts (to ensure reproducible environments) - tox - Pip - Dockerfiles (for specific operating system) - Installer script / PACKAGES (to be installed/called be each Dockerfile) - What about windows? - [ ] installing third party packages (python -m ensurepip; pip install -U pip; pip install -r requirements.txt) If you're suggesting that Python should test and maintain a distribution of specific PyPi packages, which commands do I need to add to my Dockerfiles, and OSX/Windows? - https://github.com/ipython/ipython/wiki/Install:-Docker#anaconda--ipython-co... - https://www.python.org/dev/buildbot/ - https://wiki.python.org/moin/BuildbotOnWindows (this is probably out of date; download and build which packages every time?)

...

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Wes Turner

11:13 p.m.

"The SciPy Stack" http://www.scipy.org/install.html http://www.scipy.org/stackspec.html https://westurner.org/tools/#scipy-stack http://docs.continuum.io/anaconda/pkg-docs tk http://www.tcl.tk/ Linux Mac8.5.18 requests http://docs.python-requests.org/en/latest/index.html2.7.0 https://www.enthought.com/products/canopy/package-index/ requests http://docs.python-requests.org/en/latest/2.7.0 On Thu, Aug 13, 2015 at 6:03 PM, Wes Turner wrote:

...

On Thu, Aug 13, 2015 at 4:50 PM, Donald Stufft wrote:

...
On August 13, 2015 at 5:29:15 PM, Steve Dower (steve.dower@python.org) wrote:

...
It's already a fairly crowded marketplace, at least on Windows. Anaconda, Canopy, WinPython, Pythonxy and Portable Python all come to mind, but none of them have reliably replaced the official Python installer. In large part, I suspect this is because they do too much - most include the scipy stack and at least one (typically 4-5) editors.

I think that’s also discounting the *huge* benefit of something being offered as the official Python thing. I think a lot of people are hesitant to install those other things because they come someone else, or they aren’t even aware of them because they searched for Python on google and got the official installer.

...
What might be interesting is if we installed the meta-package into the Lib directory rather than Lib/site-packages. That also opens up the possibility of removing old/deprecated modules from the stdlib but having an install option to restore them. Which helps break up the stdlib a bit and make the core Python install lighter, but still isn't really suitable for something like requests.

Right, I don’t think it’s suitable at all to install something that is typically installed from PyPI into anything other than Lib/site-packages.

...
I also don't see any real gain in making it a separate thing from the main installer unless you're planning 10+ packages to be in there. I would expect an incredibly small amount of packages to be available - those with no significant competitors, extremely broad uses, and completely portable.

Can we release new, different, installers for say, a hypothetical 3.5.0 without cutting a new release of 3.5.0 that include updated versions of the bundled software? If so, how will people know they are using the latest version of that installer? I know that requests will not be very happy being bundled like that if getting a new version is tied to a new version of Python being released.

This is one of the primary benefits of a separate installer, the installer (really the set of things that get installed) gets a version number. You can release it independently of a Python release, so if requests has a security issue then you can just roll out a new version of the platform installer without that affecting Python at all.

Another major benefit of a separate installer that layers ontop of Python is reducing (or eliminating) the friction it will cause with downstream redistributors. The “sort of stdlib, sort of not” status of pip is weird and has caused a bit of a problem. I’m still dealing with the fallout of that, and while we had to do it that way in order for it to work inside of virtual environments, I don’t think we need to do this that way.

I also think the separation just makes way more sense, when you have ensure* it means that things like python-pip depend on Python, but then Python also depends on them. I don’t think that Python the runtime needs to depends on requests and I think that doing that is going the wrong way. What this really is, is just a collection of preinstalled packages, so treating it like that seems like a better option.

Challenges

- [ ] repeatable build scripts (to ensure reproducible environments) - tox - Pip - Dockerfiles (for specific operating system) - Installer script / PACKAGES (to be installed/called be each Dockerfile) - What about windows? - [ ] installing third party packages (python -m ensurepip; pip install -U pip; pip install -r requirements.txt)

If you're suggesting that Python should test and maintain a distribution of specific PyPi packages, which commands do I need to add to my Dockerfiles, and OSX/Windows?

- https://github.com/ipython/ipython/wiki/Install:-Docker#anaconda--ipython-co... - https://www.python.org/dev/buildbot/ - https://wiki.python.org/moin/BuildbotOnWindows (this is probably out of date; download and build which packages every time?)

...
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Wes Turner

11:15 p.m.

And then for OS system packages: whohas "requests" whohas "python-requests" https://github.com/whohas/whohas On Thu, Aug 13, 2015 at 6:13 PM, Wes Turner wrote:

...

"The SciPy Stack" http://www.scipy.org/install.html http://www.scipy.org/stackspec.html https://westurner.org/tools/#scipy-stack

http://docs.continuum.io/anaconda/pkg-docs

tk http://www.tcl.tk/ Linux Mac8.5.18 requests http://docs.python-requests.org/en/latest/index.html2.7.0

https://www.enthought.com/products/canopy/package-index/

requests http://docs.python-requests.org/en/latest/2.7.0

On Thu, Aug 13, 2015 at 6:03 PM, Wes Turner wrote:

...
On Thu, Aug 13, 2015 at 4:50 PM, Donald Stufft wrote:

...
On August 13, 2015 at 5:29:15 PM, Steve Dower (steve.dower@python.org) wrote:

...
It's already a fairly crowded marketplace, at least on Windows. Anaconda, Canopy, WinPython, Pythonxy and Portable Python all come to mind, but none of them have reliably replaced the official Python installer. In large part, I suspect this is because they do too much - most include the scipy stack and at least one (typically 4-5) editors.

I think that’s also discounting the *huge* benefit of something being offered as the official Python thing. I think a lot of people are hesitant to install those other things because they come someone else, or they aren’t even aware of them because they searched for Python on google and got the official installer.

...
What might be interesting is if we installed the meta-package into the Lib directory rather than Lib/site-packages. That also opens up the possibility of removing old/deprecated modules from the stdlib but having an install option to restore them. Which helps break up the stdlib a bit and make the core Python install lighter, but still isn't really suitable for something like requests.

Right, I don’t think it’s suitable at all to install something that is typically installed from PyPI into anything other than Lib/site-packages.

...
I also don't see any real gain in making it a separate thing from the main installer unless you're planning 10+ packages to be in there. I would expect an incredibly small amount of packages to be available - those with no significant competitors, extremely broad uses, and completely portable.

Can we release new, different, installers for say, a hypothetical 3.5.0 without cutting a new release of 3.5.0 that include updated versions of the bundled software? If so, how will people know they are using the latest version of that installer? I know that requests will not be very happy being bundled like that if getting a new version is tied to a new version of Python being released.

This is one of the primary benefits of a separate installer, the installer (really the set of things that get installed) gets a version number. You can release it independently of a Python release, so if requests has a security issue then you can just roll out a new version of the platform installer without that affecting Python at all.

Another major benefit of a separate installer that layers ontop of Python is reducing (or eliminating) the friction it will cause with downstream redistributors. The “sort of stdlib, sort of not” status of pip is weird and has caused a bit of a problem. I’m still dealing with the fallout of that, and while we had to do it that way in order for it to work inside of virtual environments, I don’t think we need to do this that way.

I also think the separation just makes way more sense, when you have ensure* it means that things like python-pip depend on Python, but then Python also depends on them. I don’t think that Python the runtime needs to depends on requests and I think that doing that is going the wrong way. What this really is, is just a collection of preinstalled packages, so treating it like that seems like a better option.

Challenges

- [ ] repeatable build scripts (to ensure reproducible environments) - tox - Pip - Dockerfiles (for specific operating system) - Installer script / PACKAGES (to be installed/called be each Dockerfile) - What about windows? - [ ] installing third party packages (python -m ensurepip; pip install -U pip; pip install -r requirements.txt)

If you're suggesting that Python should test and maintain a distribution of specific PyPi packages, which commands do I need to add to my Dockerfiles, and OSX/Windows?

- https://github.com/ipython/ipython/wiki/Install:-Docker#anaconda--ipython-co... - https://www.python.org/dev/buildbot/ - https://wiki.python.org/moin/BuildbotOnWindows (this is probably out of date; download and build which packages every time?)

...
----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Steve Dower

11:25 p.m.

On 13Aug2015 1450, Donald Stufft wrote:

...

On August 13, 2015 at 5:29:15 PM, Steve Dower (steve.dower@python.org) wrote:

...
It's already a fairly crowded marketplace, at least on Windows. Anaconda, Canopy, WinPython, Pythonxy and Portable Python all come to mind, but none of them have reliably replaced the official Python installer. In large part, I suspect this is because they do too much - most include the scipy stack and at least one (typically 4-5) editors.

I think that’s also discounting the *huge* benefit of something being offered as the official Python thing. I think a lot of people are hesitant to install those other things because they come someone else, or they aren’t even aware of them because they searched for Python on google and got the official installer.

Agreed, but that sort of endorsement needs to be handled very very carefully.

...

...
What might be interesting is if we installed the meta-package into the Lib directory rather than Lib/site-packages. That also opens up the possibility of removing old/deprecated modules from the stdlib but having an install option to restore them. Which helps break up the stdlib a bit and make the core Python install lighter, but still isn't really suitable for something like requests.

Right, I don’t think it’s suitable at all to install something that is typically installed from PyPI into anything other than Lib/site-packages.

Yes, but if the aim is to lighten the stdlib, then installing the heavier stdlib into its usual place would make the transition easier. Not a huge concern right now though.

...

...
I also don't see any real gain in making it a separate thing from the main installer unless you're planning 10+ packages to be in there. I would expect an incredibly small amount of packages to be available - those with no significant competitors, extremely broad uses, and completely portable.

Can we release new, different, installers for say, a hypothetical 3.5.0 without cutting a new release of 3.5.0 that include updated versions of the bundled software? If so, how will people know they are using the latest version of that installer? I know that requests will not be very happy being bundled like that if getting a new version is tied to a new version of Python being released.

We could, but I don't want to, mainly to avoid the "which version am I on" problem. What I want is for the question to be "do I have the latest version" and the answer to be "pip install -U ...". I would also insist that something like requests should do that on initial install anyway, so you only get the bundled version if you install offline.

...

This is one of the primary benefits of a separate installer, the installer (really the set of things that get installed) gets a version number. You can release it independently of a Python release, so if requests has a security issue then you can just roll out a new version of the platform installer without that affecting Python at all.

Another major benefit of a separate installer that layers ontop of Python is reducing (or eliminating) the friction it will cause with downstream redistributors. The “sort of stdlib, sort of not” status of pip is weird and has caused a bit of a problem. I’m still dealing with the fallout of that, and while we had to do it that way in order for it to work inside of virtual environments, I don’t think we need to do this that way.

Obviously by becoming a downstream redistributor you no longer affect the other downstream redistributors (unless they see you as competition, which is more of a concern on Windows where distros generally don't have platform lock-in).

...

I also think the separation just makes way more sense, when you have ensure* it means that things like python-pip depend on Python, but then Python also depends on them. I don’t think that Python the runtime needs to depends on requests and I think that doing that is going the wrong way. What this really is, is just a collection of preinstalled packages, so treating it like that seems like a better option.

Maybe this should just be a Windows (and maybe Mac) only thing then. Distros that provide python3-requests (and maybe eventually python3-tkinter/idle) can choose to make that a default install if they like, so upstream doesn't need to cover it, but until a distro becomes more popular than the python.org release we won't see that on Windows. The tcl/tk dependency is also most blatant on Windows as well, since we apparently depend on the system's version on every other platform. Cheers, Steve

...

----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Rustom Mody

15 Aug 15 Aug

7:54 a.m.

On Friday, August 14, 2015 at 12:19:44 AM UTC+5:30, Alex Grönholm wrote:

...

13.08.2015, 20:35, Donald Stufft kirjoitti:

...
One possible thing to look at for prior art, is what Haskell does. They don’t have a bunch of ensure* modules or anything like it, instead they have their compiler (which is like “Haskell Core” and then on top of that they layer a bunch of libraries (Called “Haskell Platform”). This platform releases every ~6 months and just includes something like 40 different libraries with it that represent common development tools and widely used libraries [1].

So I guess my question is, instead of continuing down a path where we add more ensure* style modules to the standard library, why not do something similar and have “Python the Language” and “The Python Platform”, and the platform would be the Python language + N “important” or “popular” packages. This could release on a quicker release schedule than Python itself (since it would really be more like a meta package than anything that itself got developed) and would give the ability to ship things like this without the problems that we’ve had with ensurepip. From a downstream perspective they would just package all of this stuff as normal and it would just be available as normal. We could even publish a metapackage on PyPI that had no code of it’s own, but existed simply to list all of the platform packages as dependencies (with ==) and then people could easily depend on the Python “platform” in their own code.

This would essentially involve someone(s) needing to be the gatekeeper of which libraries become part of the Python platform, some small packaging shims to handle the metapackage on PyPI, and then the installer stuff for OSX and Windows (probably nothing for other OSs? Or maybe a tarball? I don’t know). Amen to this. This is EXACTLY where I've hoped Python would go :)

As an idea that's fine As an actual example, Haskell's package system is more broken than most -- See cabal hell https://www.reddit.com/r/haskell/comments/2al3vx/how_do_you_avoid_the_cabal_... In fact Haskell's package system is ironically un-functional Last I knew you can 'cabal install foo' You cant 'cabal uninstall foo' thereafter; only 'cabal unregister foo' Or delete all haskell packages and start over! That's like saying that after apt-get install foo the only way of undoing it is to reinstall linux More generally large non-trivial systems -- not just languages like python, haskell, ruby etc, but even emacs, tex, eclipse, firefox etc -- have their own packaging systems. These are all upstream of distro systems like apt/portage etc which are usually more stable, thoroughly thought out but by definition of downstream, somewhat stale. It would be good if choices like increasing 'ensures' keep in mind the required symbiosis between these two packaging worlds. Ideally -- somewhat utopian -- we can imagine a world of federated package management, where one could say # apt-get subsystem python install django and apt gets django from Pypi rather than from debian/ubuntu repos

Stephen J. Turnbull

17 Aug 17 Aug

12:27 a.m.

Donald Stufft writes:

...

One possible thing to look at for prior art, is what Haskell does.

I haven't done Darcs in anger for several years but I still follow their mailing lists, and it seems like the GHC people are happy breaking the world for Darcs every 6 months. Darcs is a relatively old member of the GHC world having many internal components that since have be superseded by "platform" modules, and apparently the style of much of the core code is considered idiosyncratic, so it may be an unusual case. Still, it doesn't sound to me like Haskell provides anything like the stability promises that Python makes for the language and the stdlib.

Rustom Mody

12:56 a.m.

On Monday, August 17, 2015 at 5:57:48 AM UTC+5:30, Stephen J. Turnbull wrote:

...

Donald Stufft writes:

...
One possible thing to look at for prior art, is what Haskell does.

I haven't done Darcs in anger for several years but I still follow their mailing lists, and it seems like the GHC people are happy breaking the world for Darcs every 6 months. Darcs is a relatively old member of the GHC world having many internal components that since have be superseded by "platform" modules, and apparently the style of much of the core code is considered idiosyncratic, so it may be an unusual case. Still, it doesn't sound to me like Haskell provides anything like the stability promises that Python makes for the language and the stdlib.

Thats an interesting point to bring up.

Yes darcs is an old member of ghc world And yes Haskell provides poor stability guarantees compared to Python And since these two – stable language vs research language – pull in opposite ways the inevitable happened: ghc junked darcs for git for its own development :-)

Andrew Barnert

13 Aug 13 Aug

8:32 p.m.

On Aug 13, 2015, at 10:06, Steve Dower wrote:

...

From my point-of-view, this has been very successful in Python 2.7, 3.4 and will also be successful in 3.5. For Python 3.6, I'd like to do a similar thing with:

* requests * tkinter (including tcl/tk, IDLE, and other dependencies)

Is the latter actually doable? Can tkinter be packaged in such a way that it includes or downloads or (best of all) downloads only if changed an entire Tcl/Tk installation (without affecting any other Tcl/Tk installations)? And, besides the technical question, at least on OS X we recommend ActiveTcl; does their weird licensing allow Python to just download and install that automatically? And, even if that is possible, does that mean I'll end up with 27 copies of ActiveTcl on my laptop (one for each Python and each virtual env) plus Apple's Tcl, instead of the 1 copy I installed manually? With a 30MB download for each one? If all of those are easily solved, then this seems like a cool idea.

Steve Dower

8:52 p.m.

On 13Aug2015 1332, Andrew Barnert wrote:

...

On Aug 13, 2015, at 10:06, Steve Dower wrote:

...
From my point-of-view, this has been very successful in Python 2.7, 3.4 and will also be successful in 3.5. For Python 3.6, I'd like to do a similar thing with:

* requests * tkinter (including tcl/tk, IDLE, and other dependencies)

Is the latter actually doable?

I believe so, but I'm only really familiar with how we currently distribute it on Windows.

...

Can tkinter be packaged in such a way that it includes or downloads or (best of all) downloads only if changed an entire Tcl/Tk installation (without affecting any other Tcl/Tk installations)?

_tkinter.pyd (on Windows) links directly against Tcl and Tk's binaries, and while theoretically the version of Tcl/Tk being used could be replaced, I'm not aware of anyone doing this or even whether it works. It's certainly not trivial, because our tcl and tk binaries are installed right next to _tkinter, so you need to break your own installation in order to do it. My idea was that whichever part of the package includes _tkinter would also include the matched version of Tix, Tcl and Tk. There may be some issues arise out of that (beyond breaking people who assumed these files would never move) - I haven't done a full investigation yet. I do know there are some environment variable settings that can cause problems. As I understand it, nobody should be importing anything other than `tkinter`, which means we have a chance to resolve those easily, but a change like this would be very likely to break people who aren't using the public documented interfaces. (I'm totally okay with breaking them, BTW, and this would be a 3.5->3.6 change, not 3.5.1.)

...

And, besides the technical question, at least on OS X we recommend ActiveTcl; does their weird licensing allow Python to just download and install that automatically?

If we're currently downloading and installing that automatically, then yes (or at least I'd assume so). If we're not, then we will continue to not do it.

...

And, even if that is possible, does that mean I'll end up with 27 copies of ActiveTcl on my laptop (one for each Python and each virtual env) plus Apple's Tcl, instead of the 1 copy I installed manually?

I'd expect not. If a separate install is a valid setup, then I'd expect that to continue to be the case. But as I say, I'm most familiar with Windows where that isn't something you can do. Cheers, Steve

...

With a 30MB download for each one?

If all of those are easily solved, then this seems like a cool idea.

Terry Reedy

14 Aug 14 Aug

4:34 a.m.

On 8/13/2015 1:06 PM, Steve Dower wrote:

...

I'd like to propose expanding the list of 3rd-party packages we bundle and install by default. (Obviously this does not apply to platforms that repackage Python and can do whatever they want, but on Windows and Mac we are fully responsible for these.)

Currently, we bundle pip (and some of its dependencies - let's avoid that particular discussion right now please, it's on python-dev) and install it by default in a way that lets users easily update to the latest version. Including pip in the standard library would lock users into a specific version for the lifetime of that Python version, which would be a bad thing.

From my point-of-view, this has been very successful in Python 2.7, 3.4 and will also be successful in 3.5. For Python 3.6, I'd like to do a similar thing with:

* requests

Not in stdlib, so easier availability is a plus

...

* tkinter (including tcl/tk, IDLE, and other dependencies)

In stdlib, heavily used by beginners, who will not be helped by the change and who may possibly be harmed by reduced availability. Why did you pick the tkinter group instead some obsolete and little used modules, such as asyncore and asynchat? I think any discussion of breaking up the stdlib should wait until the new workflow is chosen and implemented.

...

tkinter is worth more discussion :) For the remainder of this email, I'll use "tkinter" as shorthand to refer to Tcl, Tk, Tix, _tkinter, tkinter, idlelib/IDLE, PyDoc, turtledemo and any other dependencies or dependents that I missed.

turtle itself.

...

In my experience, few Python scripts depend on or assume tkinter is available.

No person's experience is representative. On Stackoverflow, tkinter has a higher rate of tkinter questions than for, say, requests or pil. I suspect its usage is also higher than those modules among students learning Python.

...

tkinter is already an optional item in the Windows installer

I am pretty sure that this is due to the large size of tcl/tk. This is, of course, less relevant now than 20 years ago. The offline docs and test suite (/lib/test/) are also optional.

...

(maybe Mac too? I don't know) and there are certainly installations of Python out there that don't have it. From this side, nothing would actually change by installing tkinter into site-packages rather than Lib.

There are multiple questions here: where is x developed, where is it installed from, and where is it installed to, and what is it installed with. Educational machines tend to be as locked down as corporate machines. Many places allow 'python and its stdlib' as a package, but no other python packages. If such policies excludes installing into site-packages, then the proposed change is crippling. [Shifting focus from 'tkinter' to Idle...]

...

(One impact may be the start menu shortcuts for IDLE and PyDoc.

For students who are not allowed to use a console, the icons are essential.

...

IDLE is already allowed to make enhancements in maintenance branches (https://www.python.org/dev/peps/pep-0434/),

PEP 434 formalized what had more or less been the practice for several years before.

...

and we have recently received patches that are to be applied to *four* branches.

So what? This is not limited to Idle. There are, of course, *four* branches because we have not dropped 2.7 and started 3.6 earlier than usual.

...

The freedom to enhance IDLE is greatly improved by making it a PyPI installable package

Are you volunteering to do this? Pardon my skepticism, but you said yourself that you work at Microsoft on "a direct competitor to Idle".

...

and disconnecting it from the stdlib's schedule.

You already explained above that the stdlib schedule is pretty much not an issue for Idle. And as you already know, Idle is already on track to get a makeover in the next year *with things as they are*. Anyway, just as there are separate lists for other specialized topics, like packaging, time zones, and core workflow, there is a separate list, Idle-sig*, for discussing how to improve Idle. Anyone interested in this should join us there. *mirrored on news.gmane.org as gmane.comp.python.idle with no mail.python.org subscription required. -- Terry Jan Reedy

Steve Dower

3:52 p.m.

On 13Aug2015 2134, Terry Reedy wrote:

...

On 8/13/2015 1:06 PM, Steve Dower wrote:

...
I'd like to propose expanding the list of 3rd-party packages we bundle and install by default. (Obviously this does not apply to platforms that repackage Python and can do whatever they want, but on Windows and Mac we are fully responsible for these.)

Currently, we bundle pip (and some of its dependencies - let's avoid that particular discussion right now please, it's on python-dev) and install it by default in a way that lets users easily update to the latest version. Including pip in the standard library would lock users into a specific version for the lifetime of that Python version, which would be a bad thing.

From my point-of-view, this has been very successful in Python 2.7, 3.4 and will also be successful in 3.5. For Python 3.6, I'd like to do a similar thing with:

* requests

Not in stdlib, so easier availability is a plus

...
* tkinter (including tcl/tk, IDLE, and other dependencies)

In stdlib, heavily used by beginners, who will not be helped by the change and who may possibly be harmed by reduced availability.

It's the biggest (only?) application in the standard library. idlelib is not documented, and I describe below how this wouldn't necessarily reduce availability or ease of entry for anyone.

...

Why did you pick the tkinter group instead some obsolete and little used modules, such as asyncore and asynchat?

I don't want to promote those to PyPI packages, I'd rather just remove them completely.

...

I think any discussion of breaking up the stdlib should wait until the new workflow is chosen and implemented.

Fair, but this is also a good way to test alternate workflows as well as simplify things for the people working on IDLE (and I acknowledge I haven't raised it on the IDLE list yet, but if the response is "never take it out of the standard library" then there's no point asking people "would you prefer to contribute to IDLE outside of the Python stdlib").

...

...
tkinter is worth more discussion :) For the remainder of this email, I'll use "tkinter" as shorthand to refer to Tcl, Tk, Tix, _tkinter, tkinter, idlelib/IDLE, PyDoc, turtledemo and any other dependencies or dependents that I missed.

turtle itself.

Thank you.

...

...
In my experience, few Python scripts depend on or assume tkinter is available.

No person's experience is representative. On Stackoverflow, tkinter has a higher rate of tkinter questions than for, say, requests or pil. I suspect its usage is also higher than those modules among students learning Python.

I really don't like it when StackOverflow is used for this sort of metric (it's actually fairly popular at work right now, and also for comparing R and Python). High question rates on StackOverflow indicate that people had problems they weren't able to solve using the documentation. A more interesting number would be the number of users who have received badges for high quality and popular answers, as that would indicate a thriving community. (And I haven't checked, but I'd expect Django and matplotlib to far exceed tkinter on either of these metrics.)

...

...
tkinter is already an optional item in the Windows installer

I am pretty sure that this is due to the large size of tcl/tk. This is, of course, less relevant now than 20 years ago. The offline docs and test suite (/lib/test/) are also optional.

As far as I'm concerned, it is precisely because of the large size. Python is being installed on so many more servers, VMs and containers now that size reduction is a perfectly legitimate goal IMHO.

...

...
(maybe Mac too? I don't know) and there are certainly installations of Python out there that don't have it. From this side, nothing would actually change by installing tkinter into site-packages rather than Lib.

There are multiple questions here: where is x developed, where is it installed from, and where is it installed to, and what is it installed with. Educational machines tend to be as locked down as corporate machines. Many places allow 'python and its stdlib' as a package, but no other python packages. If such policies excludes installing into site-packages, then the proposed change is crippling.

* where x is developed is up to whoever is doing the development. Provided a package appears on PyPI, it will work for my proposal. * it would be installed from a wheel bundled into the main installer, exactly as for pip and setuptools currently. * it would be installed with pip (so to "ensuretkinter" you'd have to "ensurepip" - I don't see this being a major issue. If someone wants to lock down package installation, they need to lock down users and not the Python installation) I've heard of plenty of places that disallow packages (I get to deal with a lot of enterprises through my work), but anything in the core installer is okay. tkinter would be in the core installer still, just like today.

...

[Shifting focus from 'tkinter' to Idle...]

...
(One impact may be the start menu shortcuts for IDLE and PyDoc.

For students who are not allowed to use a console, the icons are essential.

Please don't quote part of my sentence and add a full stop so it looks like I stopped there. Here is what I originally wrote: (One impact may be the start menu shortcuts for IDLE and PyDoc, but provided the entry points into those tools are kept stable we can continue adding shortcuts from the installer. People who omit tkinter and then install it later would not get shortcuts. But since they omitted it from the installer, they probably don't want them - they likely just got a package that has tkinter as a dependency.) As you can see, I have acknowledged this is a problem, proposed a solution, identified a subsequent problem, and stated why I don't think it's blocking. Pretending that I'm simply trying to remove the entry point is unfair and dishonest.

...

...
IDLE is already allowed to make enhancements in maintenance branches (https://www.python.org/dev/peps/pep-0434/),

PEP 434 formalized what had more or less been the practice for several years before.

...
and we have recently received patches that are to be applied to *four* branches.

So what? This is not limited to Idle. There are, of course, *four* branches because we have not dropped 2.7 and started 3.6 earlier than usual.

As I said earlier, the difference between Idle and the (majority of the) rest of the stdlib is that Idle is an application.

...

...
The freedom to enhance IDLE is greatly improved by making it a PyPI installable package

Are you volunteering to do this? Pardon my skepticism, but you said yourself that you work at Microsoft on "a direct competitor to Idle".

I'm certainly willing to help get it started. Right now one of my jobs (along with the other Windows core devs) is maintaining the tcl/tk builds and the sub-installer for tkinter. This change would simplify that part of my volunteer efforts, and so I am willing to invest in it. That's not saying I'm going to become an Idlecontributor. The "direct competitor" comment was intended as tongue in cheek (it's from a more private email thread, for anyone wondering where it showed up), but I do acknowledge that one of our (free - not a core business) products provides similar functionality to Idle. It is difficult to make any sort of design decision transparently enough that suggestions of sabotage can be silenced (I hope that's what I'm achieving here - it's certainly the point).

...

...
and disconnecting it from the stdlib's schedule.

You already explained above that the stdlib schedule is pretty much not an issue for Idle. And as you already know, Idle is already on track to get a makeover in the next year *with things as they are*.

My experience with our own product is that being disconnected from Visual Studio's schedule made it much easier to do makeovers whenever we want. I believe Idle (as an application) and tkinter (as a wrapper around a 3rd-party library) would also benefit from this. (As an aside, so would the ssl and hashlib modules, except those both have other dependencies in the stdlib.)

...

Anyway, just as there are separate lists for other specialized topics, like packaging, time zones, and core workflow, there is a separate list, Idle-sig*, for discussing how to improve Idle. Anyone interested in this should join us there.

*mirrored on news.gmane.org as gmane.comp.python.idle with no mail.python.org subscription required.

Thanks for the invite. As I mentioned earlier, if there's opposition here to bundling tkinter rather than having it as a core part of the stdlib, I don't see any reason discussing things like PyPI packages and improved workflows on Idle-sig right now. Cheers, Steve

Nick Coghlan

19 Aug 19 Aug

9:37 a.m.

On 15 August 2015 at 01:52, Steve Dower wrote:

...

On 13Aug2015 2134, Terry Reedy wrote:

...
On 8/13/2015 1:06 PM, Steve Dower wrote:

...
I'd like to propose expanding the list of 3rd-party packages we bundle and install by default. (Obviously this does not apply to platforms that repackage Python and can do whatever they want, but on Windows and Mac we are fully responsible for these.)

Currently, we bundle pip (and some of its dependencies - let's avoid that particular discussion right now please, it's on python-dev) and install it by default in a way that lets users easily update to the latest version. Including pip in the standard library would lock users into a specific version for the lifetime of that Python version, which would be a bad thing.

From my point-of-view, this has been very successful in Python 2.7, 3.4 and will also be successful in 3.5. For Python 3.6, I'd like to do a similar thing with:

* requests

Not in stdlib, so easier availability is a plus

...
* tkinter (including tcl/tk, IDLE, and other dependencies)

In stdlib, heavily used by beginners, who will not be helped by the change and who may possibly be harmed by reduced availability.

It's the biggest (only?) application in the standard library. idlelib is not documented, and I describe below how this wouldn't necessarily reduce availability or ease of entry for anyone.

There's no reason to have a GUI toolkit installed on a headless server, so many (all?) Linux distros also already separate it out into a distinct package: $ python -V Python 2.7.10 $ python -c "import tkinter" Traceback (most recent call last): File "<string>", line 1, in <module> ImportError: No module named tkinter Distros like Raspbian include it in the base install in order to also provide IDLE. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

M.-A. Lemburg

14 Aug 14 Aug

11:22 a.m.

On 13.08.2015 19:06, Steve Dower wrote:

...

I'd like to propose expanding the list of 3rd-party packages we bundle and install by default. (Obviously this does not apply to platforms that repackage Python and can do whatever they want, but on Windows and Mac we are fully responsible for these.)

Currently, we bundle pip (and some of its dependencies - let's avoid that particular discussion right now please, it's on python-dev) and install it by default in a way that lets users easily update to the latest version. Including pip in the standard library would lock users into a specific version for the lifetime of that Python version, which would be a bad thing.

From my point-of-view, this has been very successful in Python 2.7, 3.4 and will also be successful in 3.5. For Python 3.6, I'd like to do a similar thing with:

* requests * tkinter (including tcl/tk, IDLE, and other dependencies)

requests is already installed as part of pip, along with a whole set of other packages (but not exposed at the top-level), so moving it to its own ensure package wouldn't really change much in terms of approach. The problem I see with requests is that they sometimes have glitches in their releases causing them not to be usable, so the version that gets "ensured" would need some extra testing by whoever manages the list of packages. Also notes that the pre-packaged version in pip is not managed by the package manager (because it doesn't see it), so you will sooner or later end up with multiple requests package copies in your site-packages. Not sure about tkinter. Requiring newbies to run an ensure script to be able to run IDLE doesn't sound like a good idea. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 14 2015)

...

...
...
Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2015-08-12: Released mxODBC 3.3.4 ... http://egenix.com/go80 2015-08-22: FrOSCon 2015 ... 8 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Donald Stufft

1:44 p.m.

On August 14, 2015 at 7:22:56 AM, M.-A. Lemburg (mal@egenix.com) wrote:

...

requests is already installed as part of pip, along with a whole set of other packages (but not exposed at the top-level), so moving it to its own ensure package wouldn't really change much in terms of approach.

This isn’t really true except in a very abstract sense. Yes, requests exists because of pip, but it exists inside of pip (``import pip._vendor.requests``) not anywhere where normal people should be importing it. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

M.-A. Lemburg

2:17 p.m.

On 14.08.2015 15:44, Donald Stufft wrote:

...

On August 14, 2015 at 7:22:56 AM, M.-A. Lemburg (mal@egenix.com) wrote:

...
requests is already installed as part of pip, along with a whole set of other packages (but not exposed at the top-level), so moving it to its own ensure package wouldn't really change much in terms of approach.

This isn’t really true except in a very abstract sense. Yes, requests exists because of pip, but it exists inside of pip (``import pip._vendor.requests``) not anywhere where normal people should be importing it.

Right. What I'm saying is that requests is already installed, so moving it from inside pip to top-level isn't much of a change in terms of "do we want requests to be installed via ensure or not". -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 14 2015)

...

...
...
Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

Steve Dower

3:57 p.m.

On 14Aug2015 0422, M.-A. Lemburg wrote:

...

On 13.08.2015 19:06, Steve Dower wrote:

...
I'd like to propose expanding the list of 3rd-party packages we bundle and install by default. (Obviously this does not apply to platforms that repackage Python and can do whatever they want, but on Windows and Mac we are fully responsible for these.)

Currently, we bundle pip (and some of its dependencies - let's avoid that particular discussion right now please, it's on python-dev) and install it by default in a way that lets users easily update to the latest version. Including pip in the standard library would lock users into a specific version for the lifetime of that Python version, which would be a bad thing.

From my point-of-view, this has been very successful in Python 2.7, 3.4 and will also be successful in 3.5. For Python 3.6, I'd like to do a similar thing with:

* requests * tkinter (including tcl/tk, IDLE, and other dependencies)

requests is already installed as part of pip, along with a whole set of other packages (but not exposed at the top-level), so moving it to its own ensure package wouldn't really change much in terms of approach.

The problem I see with requests is that they sometimes have glitches in their releases causing them not to be usable, so the version that gets "ensured" would need some extra testing by whoever manages the list of packages.

I'm interested in this. What sort of glitches are we talking about here? Are they not caught by the requests team's tests? Why would someone else be able to test it better than them? I'd certainly be okay with locking in the version at rc1 time to give people a chance for wider testing. I'd be very nervous about updating any bundled package on the day that the final release is built.

...

Also notes that the pre-packaged version in pip is not managed by the package manager (because it doesn't see it), so you will sooner or later end up with multiple requests package copies in your site-packages.

pip has decided to vendor requests to avoid issues like this. It's unfortunate, but it is the best way to ensure that you can update requests securely even if you get a broken version.

...

Not sure about tkinter. Requiring newbies to run an ensure script to be able to run IDLE doesn't sound like a good idea.

Maybe I misunderstand how the ensure scripts work on other platforms? On Windows (and in the makefile), the installation runs it for them. Only people who edit the makefile and build from source would have to run it manually, and I'm fairly sure you don't get to claim to be a newbie at that point :) Of course, if distros disable the ensure scripts, it's on them to make sure their users have access to the packages they need. Distros can already remove Idle/tkinter if they want to. Cheers, Steve

M.-A. Lemburg

4:41 p.m.

On 14.08.2015 17:57, Steve Dower wrote:

...

On 14Aug2015 0422, M.-A. Lemburg wrote:

...
On 13.08.2015 19:06, Steve Dower wrote:

...
I'd like to propose expanding the list of 3rd-party packages we bundle and install by default. (Obviously this does not apply to platforms that repackage Python and can do whatever they want, but on Windows and Mac we are fully responsible for these.)

Currently, we bundle pip (and some of its dependencies - let's avoid that particular discussion right now please, it's on python-dev) and install it by default in a way that lets users easily update to the latest version. Including pip in the standard library would lock users into a specific version for the lifetime of that Python version, which would be a bad thing.

From my point-of-view, this has been very successful in Python 2.7, 3.4 and will also be successful in 3.5. For Python 3.6, I'd like to do a similar thing with:

* requests * tkinter (including tcl/tk, IDLE, and other dependencies)

requests is already installed as part of pip, along with a whole set of other packages (but not exposed at the top-level), so moving it to its own ensure package wouldn't really change much in terms of approach.

The problem I see with requests is that they sometimes have glitches in their releases causing them not to be usable, so the version that gets "ensured" would need some extra testing by whoever manages the list of packages.

I'm interested in this. What sort of glitches are we talking about here?

E.g. 2.5.2 -> 2.5.3

...

Are they not caught by the requests team's tests? Why would someone else be able to test it better than them?

No, but someone will have to decide which version is stable enough to put into the ensure package.

...

I'd certainly be okay with locking in the version at rc1 time to give people a chance for wider testing. I'd be very nervous about updating any bundled package on the day that the final release is built.

...
Also notes that the pre-packaged version in pip is not managed by the package manager (because it doesn't see it), so you will sooner or later end up with multiple requests package copies in your site-packages.

pip has decided to vendor requests to avoid issues like this. It's unfortunate, but it is the best way to ensure that you can update requests securely even if you get a broken version.

Right, and so the question is not so much: "do we want ensure to install requests (and all the other pip and requests dependencies) ?" but rather: "why not expose those bundled version as top-level installs ?"

...

...
Not sure about tkinter. Requiring newbies to run an ensure script to be able to run IDLE doesn't sound like a good idea.

Maybe I misunderstand how the ensure scripts work on other platforms? On Windows (and in the makefile), the installation runs it for them.

Depends on which Python version you are talking about. For Python 2.7, ensurepip is not run during installation. For Python 3.4, it is enabled per default. On Unix platforms, you have to run "python2 -m ensurepip" to have it installed, or configure Python 2.7 with --with-ensurepip to have it run during installation (default is not to run ensurepip during install).

...

Only people who edit the makefile and build from source would have to run it manually, and I'm fairly sure you don't get to claim to be a newbie at that point :)

Of course, if distros disable the ensure scripts, it's on them to make sure their users have access to the packages they need. Distros can already remove Idle/tkinter if they want to.

-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 14 2015)

...

...
...
Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

Cory Benfield

15 Aug 15 Aug

2:06 a.m.

On 14 August 2015 at 12:41, M.-A. Lemburg wrote:

...

...
...
The problem I see with requests is that they sometimes have glitches in their releases causing them not to be usable, so the version that gets "ensured" would need some extra testing by whoever manages the list of packages.

I'm interested in this. What sort of glitches are we talking about here?

E.g. 2.5.2 -> 2.5.3

For those who don't want to look this up, the error was that we updated our bundled certificates, which caused cert validation failures on websites offering certain trust chains. This would be difficult/impossible to find with pre-release testing, except by sheer good luck, because it only affected a small number of websites that have no common thread between them. This is inevitable with any form of network protocol implementation, sadly: we tend to hit unexpected edge cases in our dependencies (in this case, OpenSSL's trust chain logic).

...

...
Are they not caught by the requests team's tests? Why would someone else be able to test it better than them?

No, but someone will have to decide which version is stable enough to put into the ensure package.

I cannot speak for the project yet (all three maintainers are currently on holiday, so team communication is not particularly high bandwidth at the moment!), but I suspect we'd be really worried about any system that does not obtain the most recent release of requests, or that cannot respond quickly to security releases in requests. Cory

M.-A. Lemburg

16 Aug 16 Aug

3:03 p.m.

On 15.08.2015 04:06, Cory Benfield wrote:

...

On 14 August 2015 at 12:41, M.-A. Lemburg wrote:

...
...
...
The problem I see with requests is that they sometimes have glitches in their releases causing them not to be usable, so the version that gets "ensured" would need some extra testing by whoever manages the list of packages.

I'm interested in this. What sort of glitches are we talking about here?

E.g. 2.5.2 -> 2.5.3

For those who don't want to look this up, the error was that we updated our bundled certificates, which caused cert validation failures on websites offering certain trust chains. This would be difficult/impossible to find with pre-release testing, except by sheer good luck, because it only affected a small number of websites that have no common thread between them. This is inevitable with any form of network protocol implementation, sadly: we tend to hit unexpected edge cases in our dependencies (in this case, OpenSSL's trust chain logic).

Sorry, should have added some more context. Thanks for adding it.

...

...
...
Are they not caught by the requests team's tests? Why would someone else be able to test it better than them?

No, but someone will have to decide which version is stable enough to put into the ensure package.

I cannot speak for the project yet (all three maintainers are currently on holiday, so team communication is not particularly high bandwidth at the moment!), but I suspect we'd be really worried about any system that does not obtain the most recent release of requests, or that cannot respond quickly to security releases in requests.

The problem here is that the ensure package would include one particular package version and install this per default (with an option to update it to the most recent release, if possible during install, as is done for ensurepip in Python 3.4). I doubt that people will regularly run a package update on all their virtualenvs and Python installations to get the most recent requests version, so this needs to be taken into account somehow. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 16 2015)

...

...
...
Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2015-08-12: Released mxODBC 3.3.4 ... http://egenix.com/go80 2015-08-22: FrOSCon 2015 ... 6 days to go ::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Paul Moore

11:08 p.m.

On 14 August 2015 at 17:41, M.-A. Lemburg wrote:

...

...
pip has decided to vendor requests to avoid issues like this. It's unfortunate, but it is the best way to ensure that you can update requests securely even if you get a broken version.

Right, and so the question is not so much: "do we want ensure to install requests (and all the other pip and requests dependencies) ?" but rather: "why not expose those bundled version as top-level installs ?"

It's important to note that pip vendors the modules it does for a number of reasons - one, to ensure that they are always available, two so that there's no circular dependency (we need requests available to upgrade requests), and three, to ensure we have a stable, tested version (so that we don't have to field "pip is broken" bugs where someone has an in-development version of requests installed, for example). The first of the above two reasons is one that could be handled by ensure making these dependencies available at the top level, but the second and third advantages would be lost if that happened. We have had enough issues raised with pip because users have problems with the system OpenSSL installation (which we have to depend on, and which we can't control) that I, for one, would be reluctant for us to expose ourselves to yet more potential issues in the same vein. (Albeit much less likely to occur, as the circumstances which would cause a problem are pretty unusual). And someone would need to audit all of the uses of (currently) vendored modules in pip, to verify that if they were made top-level dependencies, they could upgrade themselves in place without (say) importing a previously unused sub-module after the uninstall had occurred but before the subsequent install. And keep that audit up to date as changes occur with pip to ensure no problems are introduced via new PRs. Paul

Barry Warsaw

14 Aug 14 Aug

3:07 p.m.

On Aug 13, 2015, at 10:06 AM, Steve Dower wrote:

...

I'd like to propose expanding the list of 3rd-party packages we bundle and install by default. (Obviously this does not apply to platforms that repackage Python and can do whatever they want, but on Windows and Mac we are fully responsible for these.)

Indeed. Can we *please* keep the source tarball pure? Even ensurepip gives (some) downstreams plenty of headaches. I'm totally fine with the OS X and Windows installers bundling whatever makes users on those platforms have a better experience. Cheers, -Barry

Steve Dower

4 p.m.

On 14Aug2015 0807, Barry Warsaw wrote:

...

On Aug 13, 2015, at 10:06 AM, Steve Dower wrote:

...
I'd like to propose expanding the list of 3rd-party packages we bundle and install by default. (Obviously this does not apply to platforms that repackage Python and can do whatever they want, but on Windows and Mac we are fully responsible for these.)

Indeed. Can we *please* keep the source tarball pure? Even ensurepip gives (some) downstreams plenty of headaches.

I'm totally fine with the OS X and Windows installers bundling whatever makes users on those platforms have a better experience.

I'm fine with restricting it to the installers we build ourselves too. I guess the core of the discussion (which I got wrong in my quote above) is about turning some 1st-party packages into 3rd-party packages as far as the source tarball is concerned. How would downstreams react to having to get tkinter/Idle/etc. from a separate repo? Cheers, Steve

...

Cheers, -Barry

Petr Viktorin

4:27 p.m.

On Fri, Aug 14, 2015 at 6:00 PM, Steve Dower wrote:

...

How would downstreams react to having to get tkinter/Idle/etc. from a separate repo?

It shouldn't be much of a problem. In Fedora, we already provide tkinter in a separate package from the rest of Python.

Terry Reedy

3:24 p.m.

On 8/13/2015 1:06 PM, Steve Dower wrote:

...

From my point-of-view, this has been very successful in Python 2.7, 3.4 and will also be successful in 3.5. For Python 3.6, I'd like to do a similar thing with:

* requests

The idea, I take it, it to make it easier for people, especially beginners to have a more fully-loaded installation when the installer quits, without having to find the console and run pip. The requests example points to presenting users with a curated list of packages they might want to install. A non-exclusive alternative is to present a list of packages users have already installed with other versions. "You have the following 3rd party modules installed for 3.4. Which of these would you like installed for 3.5? [x] numpy [x] pillow [x] pygame ... " This would save people from having to learn about and use a requirements list. For 2 to 3 upgrades, additional work would be required to check whether packages have py 3 versions, and if not, whether there are py 3 replacements. Pillow can be suggested and installed as a plug-compatible replacement for PIL. Most other cases are harder. But any work the installer can do to make upgrades easier would be very helpful. -- Terry Jan Reedy

3172

Age (days ago)

3178

Last active (days ago)

List overview

Download

32 comments

15 participants

participants (15)

Alex Grönholm
Andrew Barnert
Barry Warsaw
Cory Benfield
Donald Stufft
Joonas Liik
M.-A. Lemburg
Nick Coghlan
Paul Moore
Petr Viktorin
Rustom Mody
Stephen J. Turnbull
Steve Dower
Terry Reedy
Wes Turner

More "ensure*" packages

tags

participants (15)