[Distutils] Specific packaging goals and a tentative timeline

[Nick Coghlan]

We have a lot of initiatives going every which way at the moment, so I
figured it would be a good idea to get a common perception of what we
consider to be the important near term goals and a realistic timeline
for improving the packaging ecosystem (in particular, the timing
relative to the CPython 3.4 release cycle).

One of the big things I'd like us to do is ensure we separate out
"urgent" things that are coupled to the 3.4 release cycle (like
ensuring in-place upgrades for pip work on Windows) from the
"important but not urgent" things where we can take more time (like
metadata 2.0)

This is kinda long, but if people aren't used to long emails from me
by now, they haven't been paying attention ;)

(PyPA devs - please forward this to pypa-dev for discussion with the
folks that don't frequent distutils-sig)

My current impression is that the goals below should be fairly realistic.

Already done or very close to done (Yay!):

    * improved PyPI SSL support
    * setuptools/distribute merger
    * easy_install SSL verification
    * setuptools support for additional hashes beyond md5
    * pip 1.4 release with SSL verification and initial wheel support (soon!)

Before Python 3.4 feature freeze (currently November 23, 2013)

    * decide on a bundling or explicit bootstrapping scheme for pip
(this still needs a PEP to help clarify the pros and cons of the
various alternatives)
    * get RM & installer builder consensus on that scheme
    * make any necessary updates to CPython (e.g. possibly adding Lib/getpip.py)
    * (hopefully) add support for indirect imports (see
http://mail.python.org/pipermail/import-sig/2013-July/000645.html for
the draft PEP - thanks Eric for taking this from a rough idea in email
to a concrete proposal!)

Before Python 3.4 first release candidate (currently Jan 18, 2014)

    * pip 1.5 available (or at least release candidates)
    * setuptools releases as needed
    * improved handling of in-place pip upgrades on Windows
    * improved handling of pip/setuptools/pkg_resources division of
responsibility
    * both pip and setuptools available as cross platform wheel files on PyPI
    * Key requirement: "pip uninstall setuptools" must be supported &
must not fundamentally break pip (but may disable installation from
anything other than wheel files)
    * Highly desirable: possible to install pkg_resources without
installing setuptools
    * Highly desirable: possible to install setuptools without the
easy_install script (just the script, having the implementation in the
setuptools.commands subpackage is fine)

Following Python 3.4 final release (currently Feb 22, 2014)

    * further proposals target pip 1.6 - decoupled from CPython release cycle
    * metadata 2.0 (PEP 426/440)
    * sdist 2.0 and wheel 1.1
    * installation database format update
    * revisit distlib-based pip (assuming 1.5 isn't based on a vendored distlib)
    * revisit TUF-for-PyPI (that's more likely to be pip 1.7
timeframe, though...)

Independent activities & miscellaneous suggestions

    * maybe suggest "pip install distlib" over pip gaining its own
programmatic API?
    * PEP 8 cleanup (including clarification of what constitutes an
internal API)
    * improved PyPI upload API (Donald's working on this)
    * getting Warehouse to a point where it can be brought online as
"pypi-next.python.org"
    * TUF-for-PyPI exploration (the TUF folks seems to have this well in hand)
    * improved local PyPI hosting (especially devpi)

Specifically on the "bundle or bootstrap pip" front, I'll note that
due to the concerns regarding how bundling pip with the CPython MSI
installer may interact with in-place upgrades, I'm leaning back
towards explicit bootstrapping, with an option to run the bootstrap as
part of the installation process for both CPython 3.4+ and the Python
Launcher for Windows. Doing that also gives Linux distros something
they can patch in the system Python to direct users towards the
appropriate system package manager command.

Regardless, we still need the various bundling-or-bootstrap
alternatives that aren't covered in PEP 439 extracted from the list
archives and turned into a PEP that compares them and suggests a
preferred solution.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia