[Distutils] PEP 470 Round 2 - Using Multi Index Support for External to PyPI Package File Hosting

[Nick Coghlan]

On 7 Jun 2014 06:08, "Donald Stufft" <donald at stufft.io> wrote:
>
>
> On Jun 6, 2014, at 9:41 AM, holger krekel <holger at merlinux.eu> wrote:
> >
> > Once you care for ACLs for indexes and releases you have a number
> > of issues to consider, it's hardly related to PEP470/PEP438.
>
> It is related, because it means that the exact same mechanisms can be
used,
> people don’t have to learn two different ways of specifying externally
hosted
> projects. In fact it also teaches them how to specify mirrors and the
like as well
> something that any devpi user is already going to have to learn how to do.

This is the key benefit of PEP 470 from my perspective: some aspects of the
Python packaging ecosystem suffer from a bad case of "too many ways to do
it", and if we're ever going to fix that, we need to be ruthless in culling
redundant concepts.

Specifying custom indexes is a feature with a lot of use cases - local
mirrors and private indexes being two of the big ones. By contrast,
external references from the simple API duplicate a small subset of the
custom index functionality in a way that introduces a whole slew of new
concepts that still need to be documented and learned, even if the advice
is "don't use that, use custom indexes instead".

As far as dev-pi goes, if it's only mirroring links rather than externally
hosted files today, then in the future, it will still automatically mirror
the external index URLs. Dependency update scanners could follow those
links automatically, even if pip install doesn't check them by default.

One other nice consequence of PEP 470 should make it easier for
organisations to flag and investigate cases where they're relying on an
upstream source other than PyPI, regardless of whether they care about the
details of their dependencies' hosting for speed, reliability or legal
reasons.

>From a migration perspective, how hard would it be to automate generation
of a custom index page on pythonhosted.org for projects currently relying
on external references? That would still let us make the client changes
without needing to special case PIL.

Also, it occurred to me that while the latest/any split matters for new
users, we still need to consider the impact on projects which have pinned
dependencies on older versions of packages that were previously externally
hosted, but have moved to PyPI for more recent releases. I still think
dropping the external reference feature from the simple API in favour of
improving the custom index support is the right to do, but a couple of
*client side* examples of handling the migration could help clarify the
consequences for the existing users that may be affected.

For example, perhaps we should keep "--allow-all-external", but have it
mean that pip automatically adds new custom index URLs given for the
requested packages. Even if it emitted a deprecation warning, clients using
it would keep working in the face of the proposed changes to the simple API
link handling.

Regards,
Nick.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140607/81bcaa4c/attachment-0001.html>