[Distutils] a plea for backward-compatibility / smooth transitions (was: Re: Migrating Hashes from MD5 to SHA256)

[holger krekel]

Hi Nick, Donald, all,

On Sun, Jul 28, 2013 at 22:23 +1000, Nick Coghlan wrote:
> On 28 July 2013 20:55, Donald Stufft <donald at stufft.io> wrote:
> > Ok so given that:
> >
> >     - There's a readably available solution for Python 2.4+ with the likelihood
> >        being that most users are either using it or using an older version which
> >        doesn't support SSL.
> >     - The number of folks likely to be on Python 2.3 and wanting to install things
> >        from PyPI is likely to be very small.
> >     - There's possibly a future solution for Python 2.3
> >     - The safety margins for MD5 are gone and cryptographers heavily suggest
> >        moving away from it.

Please detail the actual attack scenario wrt PyPI/installer processes.

> >     - A revised scheme will break backwards compatibility with the versions of
> >       the tooling that do support a stronger hash.
>
> >
> > I'm going to go ahead and make this change unless someone comes out and
> > contests moving PyPI to SHA256. I'll give it a bit to make sure no one does
> > have an issue with the move.

Actually, i strongly object further backward-incompatible changes.

Please (generally) find a way to introduce improvements without breaking
existing installation processes at the same time.

For example, in this case pip/easy_install could indicate to PYPI what
kind of hashes it accepts (through a header or query param or whatever)
and PyPI could serve it but we'd default to MD5 for now if nothing else 
was requested.  Please also consider the PEP438 vetted registration of
externals+hashses in this context.  Once things and tools are working
nicely we can switch to serving a non-MD5 hash as default after a 
sufficient grace period.

> +1, this sounds like a good way forward for the existing PyPI interfaces.
> 
> We can do something better once the focus shifts from "make the status
> quo not broken" to making the next generation interfaces a reality
> (PEP 426 et al).

Just switching the hashes is likely to break things.  Do you want to take
a bet on that?  Just switching to SSL broke things.  Just switching
pip-1.4 broke things.  Just switching to newer setuptools broke things.
For me the fall-out of all the well-intentioned changes has been 
frustrating lately.

For example, one of my projects documented how to generate a
supervisor-controled deployment and suddenly "pip install supervisor"
does not work anymore because pip-1.4 doesn't consider any existing
supervisor distributions as non-dev.  You can argue that's supervisor's
fault but in the end it doesn't matter: What used to work and was
documented is now broken, users complaining or simply turning away, with
a re-inforced view of "python packaging sucks" or "this tool sucks, the
docs are wrong".

There are >1000 times more people using python packaging tools than
there are ones following the recent distutils-sig/Donald/Nick verdicts
or tool release notes.  And heck, even for me being quite involved in all
this stuff, having written a related PEP, it's really hard to see what's
coming and to prevent breakage for my tools's/projects's users.
Therefore, to be honest, at this point i am almost afraid of what is
released/deployed next in PyPI/Installer land.  I'd rather like to 
get into a "can't wait to get the next release" mode.

best,
holger