[Distutils] devpi-server-4.0: fixing the pip-8.1.2 issue / pep 503 compliance

[holger krekel]

devpi-server-4.0: fixing the pip-8.1.2 problem / PEP503 compliance
============================================================================

We've made available critically important releases of the devpi private packaging
available.  If you are not using "devpi" yet then you can may just read
http://doc.devpi.net  and forget about the rest of this announcement.

This is for the many who experienced the "pip doesn't install packages
anymore with devpi" problem.  First of all, you may temporarily pin "pip" 
to avoid the problem on the client side:
 
    pip install pip==8.1.1

This is obviously a crutch but gives you some time to perform the
export/import cycle required for hosting private packages via
devpi-server-4.0 and being compatible with pip-8.1.2.

If you are using devpi-server as a pure pypi.python.org cache you don't 
need to perform export/import and can just delete your server directory 
($HOME/.devpi/server by default) before you install and start up 
devpi-server-4.0.

If you are hosting private packages on devpi you will need to perform an
export/import cycle of your server state in order to run devpi-server-4.0.
The "4.0" in this case only signals this export/import need -- no other
big changes are coming with 4.0.  At the end of this announcement we explain 
some details of why we needed to go for a 4.0 and not just a micro bugfix release.


To export from devpi-server-3.X
--------------------------------

upgrade to the new devpi-server-3.1.2 before you export, like this:

    pip install "devpi-server<4.0" 

Now stop your server and run:

    devpi-server --export EXPORTDIR --serverdir SERVERDIR

where EXPORTDIR should be a fresh new directory and SERVERDIR
should be the server state directory ($HOME/.devpi/server by default).


To export from devpi-server-2.X
--------------------------------

Upgrade to the latest devpi-server-2.X release:

    pip install "devpi-server<3.0" devpi-common>=2.0.10

Here we force the devpi-common dependency to not accidentally
be "devpi-common==2.0.9" which could lead to problems.

Now stop your server and run:

    devpi-server --export EXPORTDIR --serverdir SERVERDIR

where EXPORTDIR should be a fresh new directory and SERVERDIR
should be the server state directory ($HOME/.devpi/server by default).


to import state into devpi-server-4.0
----------------------------------------

Upgrade to the latest devpi-server-4.X release:

    pip install "devpi-server<5.0" devpi-web

If you don't use "devpi-web" you can leave it out from the pip command.
Check you have the right version:

    devpi-server --version

Now import from your previously created EXPORTDIR:

    devpi-server --serverdir SERVERDIR_NEW --import EXPORTDIR

This will take a while if you have many indexes or lots of documentation --
devpi-web will create a search index over all of it during import.

You are now good to go -- pip works again!


devpi-client also has a 2.6.3
--------------------------------

We also published a minor bugfix "devpi-client-2.6.3" release which
should work with both devpi-server-2.6 and devpi-server-4.0 as we
are generally trying to keep devpi-client forward/backward compatible.
You only need to install devpi-client-2.6.3 if you also install
devpi-server into the same virtual environment.  Otherwise using
devpi-client-2.6.2 with both devpi-server-2.6 and devpi-server-4.0
probably works fine as well.

background on the pip/devpi bug for the curious
-----------------------------------------------

Besides devpi, also artifactory and other private index servers
have experienced failures with pip-8.1.2.  The change from 8.1.1
was that pip now asks for PEP503-normalized names when requesting
the simple page from an index.  Previously "-" and "." would be
allowed but with the new normalization "." is substituted with "-".
Now "pip install zope.interface" triggers a request to 
"+simple/zope-interface" and devpi in turns asks 
pypi.python.org/simple/zope-interface and gets an answer
with lots of "zope.interface-*.tar.gz" release links. But those
are not matched because without PEP503 "zope.interface" and "zope-interface"
are different things.  Moreover, pypi.python.org used to redirect 
to the "true" name but does not do this anymore which contributed
to the eventual problem.

We decided to go for 4.0 because since 3.0 we base database
keys on normalized project names -- and this normalization is
used in like 20-30 code places across the devpi system and plugins.
Trying to be clever and avoid the export/import and trick "pip-8.1.2"
into working looked like a can of worms.  Now with devpi-server-4.0
we are using proper PEP503 specified normalization so should be safe.

best,
holger and florian

P.S.: we offer support contracts btw and thank in particular
Dolby Laboratories, YouGov Inc and BlueYonder GmbH who funded a lot of
the last year's devpi work and now agreed to be named in public - and
no, we didn't get around to make a flashy web site yet.  For now,
just mail holger at merlinux to discuss support and training options.