Specific packaging goals and a tentative timeline
We have a lot of initiatives going every which way at the moment, so I figured it would be a good idea to get a common perception of what we consider to be the important near term goals and a realistic timeline for improving the packaging ecosystem (in particular, the timing relative to the CPython 3.4 release cycle). One of the big things I'd like us to do is ensure we separate out "urgent" things that are coupled to the 3.4 release cycle (like ensuring in-place upgrades for pip work on Windows) from the "important but not urgent" things where we can take more time (like metadata 2.0) This is kinda long, but if people aren't used to long emails from me by now, they haven't been paying attention ;) (PyPA devs - please forward this to pypa-dev for discussion with the folks that don't frequent distutils-sig) My current impression is that the goals below should be fairly realistic. Already done or very close to done (Yay!): * improved PyPI SSL support * setuptools/distribute merger * easy_install SSL verification * setuptools support for additional hashes beyond md5 * pip 1.4 release with SSL verification and initial wheel support (soon!) Before Python 3.4 feature freeze (currently November 23, 2013) * decide on a bundling or explicit bootstrapping scheme for pip (this still needs a PEP to help clarify the pros and cons of the various alternatives) * get RM & installer builder consensus on that scheme * make any necessary updates to CPython (e.g. possibly adding Lib/getpip.py) * (hopefully) add support for indirect imports (see http://mail.python.org/pipermail/import-sig/2013-July/000645.html for the draft PEP - thanks Eric for taking this from a rough idea in email to a concrete proposal!) Before Python 3.4 first release candidate (currently Jan 18, 2014) * pip 1.5 available (or at least release candidates) * setuptools releases as needed * improved handling of in-place pip upgrades on Windows * improved handling of pip/setuptools/pkg_resources division of responsibility * both pip and setuptools available as cross platform wheel files on PyPI * Key requirement: "pip uninstall setuptools" must be supported & must not fundamentally break pip (but may disable installation from anything other than wheel files) * Highly desirable: possible to install pkg_resources without installing setuptools * Highly desirable: possible to install setuptools without the easy_install script (just the script, having the implementation in the setuptools.commands subpackage is fine) Following Python 3.4 final release (currently Feb 22, 2014) * further proposals target pip 1.6 - decoupled from CPython release cycle * metadata 2.0 (PEP 426/440) * sdist 2.0 and wheel 1.1 * installation database format update * revisit distlib-based pip (assuming 1.5 isn't based on a vendored distlib) * revisit TUF-for-PyPI (that's more likely to be pip 1.7 timeframe, though...) Independent activities & miscellaneous suggestions * maybe suggest "pip install distlib" over pip gaining its own programmatic API? * PEP 8 cleanup (including clarification of what constitutes an internal API) * improved PyPI upload API (Donald's working on this) * getting Warehouse to a point where it can be brought online as "pypi-next.python.org" * TUF-for-PyPI exploration (the TUF folks seems to have this well in hand) * improved local PyPI hosting (especially devpi) Specifically on the "bundle or bootstrap pip" front, I'll note that due to the concerns regarding how bundling pip with the CPython MSI installer may interact with in-place upgrades, I'm leaning back towards explicit bootstrapping, with an option to run the bootstrap as part of the installation process for both CPython 3.4+ and the Python Launcher for Windows. Doing that also gives Linux distros something they can patch in the system Python to direct users towards the appropriate system package manager command. Regardless, we still need the various bundling-or-bootstrap alternatives that aren't covered in PEP 439 extracted from the list archives and turned into a PEP that compares them and suggests a preferred solution. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 19 July 2013 14:06, Nick Coghlan <ncoghlan@gmail.com> wrote:
Already done or very close to done (Yay!):
* improved PyPI SSL support * setuptools/distribute merger * easy_install SSL verification * setuptools support for additional hashes beyond md5 * pip 1.4 release with SSL verification and initial wheel support (soon!)
Marcus pointed out that should be: * pip 1.3 release with SSL verification support * pip 1.4 release with initial wheel support (soon!) I also missed out: * PyPI relocation to OSU/OSL * PyPI CDN * Massive reduction in external link scraping (PEP 438) Good things have already been done, more good things are coming, we just need to pick a sensible order and time frame to avoid burning anyone out :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 19 July 2013 05:06, Nick Coghlan <ncoghlan@gmail.com> wrote:
* both pip and setuptools available as cross platform wheel files on PyPI
Just to point out, this is the one that needs a solution to the "script wrappers" issues. At the moment, nominally cross-platform/architecture wheels are actually not so, because the wrapper scripts are fundamentally platform specific. Apologies if everyone already understood this fact. Paul
On 19 July 2013 05:06, Nick Coghlan <ncoghlan@gmail.com> wrote:
* (hopefully) add support for indirect imports (see http://mail.python.org/pipermail/import-sig/2013-July/000645.html for the draft PEP - thanks Eric for taking this from a rough idea in email to a concrete proposal!)
Looking at the import-sig archives, it looks like the list is essentially dead (I obviously unsubscribed some time ago, as I hadn't received the referenced email). Might be worth discussing this on python-dev? Paul
On 19 July 2013 18:17, Paul Moore <p.f.moore@gmail.com> wrote:
On 19 July 2013 05:06, Nick Coghlan <ncoghlan@gmail.com> wrote:
* (hopefully) add support for indirect imports (see http://mail.python.org/pipermail/import-sig/2013-July/000645.html for the draft PEP - thanks Eric for taking this from a rough idea in email to a concrete proposal!)
Looking at the import-sig archives, it looks like the list is essentially dead (I obviously unsubscribed some time ago, as I hadn't received the referenced email).
There hasn't been a lot to talk about since the namespace package discussions :)
Might be worth discussing this on python-dev?
Given the functionality it aims to replace, here would probably be a better place for preliminary discussions to thrash out a detailed proposal. Eric and I were actually wondering about whether or not to cross-post it. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Fri, Jul 19, 2013 at 4:31 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 19 July 2013 18:17, Paul Moore <p.f.moore@gmail.com> wrote:
On 19 July 2013 05:06, Nick Coghlan <ncoghlan@gmail.com> wrote:
* (hopefully) add support for indirect imports (see http://mail.python.org/pipermail/import-sig/2013-July/000645.html for the draft PEP - thanks Eric for taking this from a rough idea in email to a concrete proposal!)
Looking at the import-sig archives, it looks like the list is essentially dead (I obviously unsubscribed some time ago, as I hadn't received the referenced email).
There hasn't been a lot to talk about since the namespace package discussions :)
Might be worth discussing this on python-dev?
Given the functionality it aims to replace, here would probably be a better place for preliminary discussions to thrash out a detailed proposal. Eric and I were actually wondering about whether or not to cross-post it.
I agree it should start there (I have already replied to Eric's draft PEP). All of the people with any form of intimate knowledge of how import works are on that list so I think it's still best to hash out any sticking points there before taking to python-dev for decision-making.
* decide on a bundling or explicit bootstrapping scheme for pip (this still needs a PEP to help clarify the pros and cons of the various alternatives)
if we improve things enough so that the get-pip.py experience is reliable and robust (and handles setuptools if not bundled), then might that be enough for now? (see the options here: https://github.com/pypa/pip/issues/1049) i.e. improve pip's installer experience, and then come back around to bundling/bootstrap with python.
On 20 July 2013 02:31, Marcus Smith <qwcode@gmail.com> wrote:
* decide on a bundling or explicit bootstrapping scheme for pip (this still needs a PEP to help clarify the pros and cons of the various alternatives)
if we improve things enough so that the get-pip.py experience is reliable and robust (and handles setuptools if not bundled), then might that be enough for now? (see the options here: https://github.com/pypa/pip/issues/1049) i.e. improve pip's installer experience, and then come back around to bundling/bootstrap with python.
If we can figure out a consistent download-and-run experience that doesn't assume the availability of curl or wget, I think so. Alternatively (as I think Donald suggested?) a simple download-and-run Windows executable or installer would be friendlier for people that may not be used to configuring the Windows command line (even if it still relied on get-pip.py to do the heavy lifting in terms of actually getting pip onto the system). It doesn't help that the Microsoft provided UI for configuring environment variables hasn't seen any serious improvements in the better part of two decades :P Also, if we'd like to do cert verification in the bootstrap script, keep in mind the fact that Python 2.6+ supports executable zip archives, so long as they have a __main__.py file at the top level. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Jul 19, 2013, at 11:27 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 20 July 2013 02:31, Marcus Smith <qwcode@gmail.com> wrote:
* decide on a bundling or explicit bootstrapping scheme for pip (this still needs a PEP to help clarify the pros and cons of the various alternatives)
if we improve things enough so that the get-pip.py experience is reliable and robust (and handles setuptools if not bundled), then might that be enough for now? (see the options here: https://github.com/pypa/pip/issues/1049) i.e. improve pip's installer experience, and then come back around to bundling/bootstrap with python.
If we can figure out a consistent download-and-run experience that doesn't assume the availability of curl or wget, I think so. Alternatively (as I think Donald suggested?) a simple download-and-run Windows executable or installer would be friendlier for people that may not be used to configuring the Windows command line (even if it still relied on get-pip.py to do the heavy lifting in terms of actually getting pip onto the system). It doesn't help that the Microsoft provided UI for configuring environment variables hasn't seen any serious improvements in the better part of two decades :P
Also, if we'd like to do cert verification in the bootstrap script, keep in mind the fact that Python 2.6+ supports executable zip archives, so long as they have a __main__.py file at the top level.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia _______________________________________________ Distutils-SIG maillist - Distutils-SIG@python.org http://mail.python.org/mailman/listinfo/distutils-sig
Folks are aware that get-pip.py includes an entire pip installation griped inside of it right? So if we ship get-pip.py we're shipping pip, which we'll then use to install pip over the network … instead of just shipping pip. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Folks are aware that get-pip.py includes an entire pip installation griped inside of it right?
So if we ship get-pip.py we're shipping pip, which we'll then use to install pip over the network … instead of just shipping pip.
yes, it has pip in it. I mentioned it as something to improve for the present, not ship. that would be odd
On 20 July 2013 13:52, Marcus Smith <qwcode@gmail.com> wrote:
Folks are aware that get-pip.py includes an entire pip installation griped inside of it right?
So if we ship get-pip.py we're shipping pip, which we'll then use to install pip over the network … instead of just shipping pip.
yes, it has pip in it. I mentioned it as something to improve for the present, not ship. that would be odd
Actually, the main thing that made me realised that bundling pip with the installer for something else was a potentially flawed notion was that it would mean we'd be inflicting on *ourselves* exactly the same problem that already exists on Linux: two different file management tools (PEP 376 installers and the system package manager) fighting over the same Python installation. It wouldn't be as severe (since it would only affect one project), but it's still a potentially ugly mess. By contrast, the explicit bootstrap (whether using the current get-pip.py structure or a zip archive with a __main__.py file and whether executed at install time or later) ensures that the site-packages for that particular Python installation remains under the control of the PEP 376 installers (including pip). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 20 July 2013 04:27, Nick Coghlan <ncoghlan@gmail.com> wrote:
It doesn't help that the Microsoft provided UI for configuring environment variables hasn't seen any serious improvements in the better part of two decades :P
AFAIK, for setting environment variables, the Unix UI hasn't changed in a lot longer than that... (Sorry, knee-jerk reaction to digs at Windows there :-)) Paul
On 20 July 2013 17:40, Paul Moore <p.f.moore@gmail.com> wrote:
On 20 July 2013 04:27, Nick Coghlan <ncoghlan@gmail.com> wrote:
It doesn't help that the Microsoft provided UI for configuring environment variables hasn't seen any serious improvements in the better part of two decades :P
AFAIK, for setting environment variables, the Unix UI hasn't changed in a lot longer than that...
(Sorry, knee-jerk reaction to digs at Windows there :-))
Hey, I come by my hatred of that <censored> panel honestly! When a flat text file with an arcane name is a UI upgrade... :) Cheers, Nick. P.S. For those that don't know (which is probably most people), I was a professional C/C++/Python programmer on Windows (starting with NT4!) from late 2000 through to early 2011. I just haven't used it for *personal* software development since 2004 or so, when I decided learning Linux was easier than trying to get CPython to build on Windows with the very limited free tools that existed in the days prior to Visual Studio Express :) -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 19 July 2013 14:06, Nick Coghlan <ncoghlan@gmail.com> wrote:
Independent activities & miscellaneous suggestions
* maybe suggest "pip install distlib" over pip gaining its own programmatic API? * PEP 8 cleanup (including clarification of what constitutes an internal API) * improved PyPI upload API (Donald's working on this) * getting Warehouse to a point where it can be brought online as "pypi-next.python.org" * TUF-for-PyPI exploration (the TUF folks seems to have this well in hand) * improved local PyPI hosting (especially devpi)
A significant one I left out here: * Getting the "Python Packaging User's Guide" up to scratch, and deferring to that in the docs on python.org This is hard to really get going *right now*, since it's tricky to provide clear guidelines while we're still trying to figure out exactly what the recommended approach *is*. However, I'm hopeful that as the core approach stabilises over the next few months we'll be able to use the guide as a place to capture those instructions, to the point where we're prepared to start pointing more people to it as the "one obvious place" to get info about the Python packaging ecosystem. The reason I consider it independent of the CPython release cycle is that we're pretty flexible with doc updates, and the online docs for all branches are refreshed daily. So whenever we deem the packaging user guide ready for broad consumption, we can adjust the material on docs.python.org to defer to it. For offline docs, this documented deprecation of the bundled docs will then get captured in the next CPython maintenance release (those that need to deal with isolated networks and working with an entirely private index server installation are such a special case that we can leave them to deal with their own documentation, especially since they have the option of making a private fork of the upstream packaging guide). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 19 July 2013 14:06, Nick Coghlan <ncoghlan@gmail.com> wrote:
We have a lot of initiatives going every which way at the moment, so I figured it would be a good idea to get a common perception of what we consider to be the important near term goals and a realistic timeline for improving the packaging ecosystem (in particular, the timing relative to the CPython 3.4 release cycle).
What do people think of the idea of my writing something up as an actual "Python packaging road map" and adding it to the user guide? (It would probably replace https://python-packaging-user-guide.readthedocs.org/en/latest/future.html) Note that I wouldn't actually *do* this until after I get back from Flock to Fedora (the conference is mid-August, I get home late August since I'm taking some time off for a holiday), so we have plenty of time to discuss the general approach before it gets written up anywhere more "official". Just wanted to give people a change to think about the idea. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
What do people think of the idea of my writing something up as an actual "Python packaging road map" and adding it to the user guide? (It would probably replace https://python-packaging-user-guide.readthedocs.org/en/latest/future.html)
Note that I wouldn't actually *do* this until after I get back from Flock to Fedora (the conference is mid-August, I get home late August since I'm taking some time off for a holiday), so we have plenty of time to discuss the general approach before it gets written up anywhere more "official". Just wanted to give people a change to think about the idea.
I think it would be great. I was hoping for something like that. Marcus
participants (5)
-
Brett Cannon -
Donald Stufft -
Marcus Smith -
Nick Coghlan -
Paul Moore