[Catalog-sig] V3 PEP-draft for transitioning to pypi-hosting of release files

Wed Mar 13 20:08:32 CET 2013

On Mar 13, 2013, at 2:57 PM, "M.-A. Lemburg" <mal at egenix.com> wrote:

> On 13.03.2013 12:21, holger krekel wrote:
>> Hi all,
>> 
>> after some more discussions and hours spend by Carl Meyer (who is now
>> co-authoring the PEP) and me, here is a new V3 pre-submit draft.  
>> It is now more ambitious than the previous draft as should be obvious
>> from the modified abstract (and Carl Meyers and Philip's earlier
>> interactions on this list).  There also are more details of how
>> the current link-scraping works among other improvements and incorporations
>> of feedback from discussions here.
>> 
>> We intend to submit this draft tonight to the PEP editors.  
>> 
>> Feedback now and later remains welcome.  I am sure there are issues to 
>> be sorted and clarified, among them the versioning-API suggestion by 
>> Marc-Andre.
>> 
>> Thanks for everybody's support and feedback so far,
>> holger
>> 
>> 
>> PEP: XXX
>> Title: Transitioning to release-file hosting on PyPI
>> Version: $Revision$
>> Last-Modified: $Date$
>> Author: Holger Krekel <holger at merlinux.eu>, Carl Meyer <carl at oddbird.net>
>> Discussions-To: catalog-sig at python.org
>> Status: Draft (PRE-submit V3)
>> Type: Process
>> Content-Type: text/x-rst
>> Created: 10-Mar-2013
>> Post-History:
>> 
>> 
>> Abstract
>> ========
>> 
>> This PEP proposes a backward-compatible two-phase transition process to speed
>> up, simplify and robustify installing from the pypi.python.org (PyPI)
>> package index.  To ease the transition and minimize client-side
>> friction, **no changes to distutils or existing installation tools are
>> required in order to benefit from the transition phases, which is to
>> result in faster, more reliable installs for most existing packages**.
>> 
>> The first transition phase implements easy and explicit means for
>> a package maintainter to control which release file links are 
>> served to present-day installation tools.  The first phase also
>> includes the implementation of analysis tools for present-day packages,
>> to support communication with package maintainers and the automated
>> setting of default modes for controling release file links.   
>> 
>> The second transition phase will result in the current PYPI index 
>> to only serve PYPI-hosted files by default.  Externally hosted files
>> will still be automatically discoverable through a second index. 
>> Present-day installation tools will be able to continue working
>> by specifying this second index.  New versions of installation
>> tools shall default to only install packages from PYPI unless
>> the user explicitely wishes to include non-PYPI sites.
> 
> I must say, don't like this change in motivation compared
> to V1 and V2.
> 
> The original of the discussion was to make PyPI more secure
> and the installation process faster and more reliable
> by moving away from crawling arbitrary external web pages.
> 
> Both can be had by:
> 
> * limiting the crawling to package author defined specific
>  URLs, with added hashes to make sure that the URLs and
>  their target content is not modified (this is the securing
>  external downloads part - see here for an example:
>  https://pypi.python.org/pypi/egenix-pyopenssl/0.13.1.1.0.1.5),
>  and
> 
> * adding a way for the package authors to say "PyPI, please go
>  ahead and cache/copy my distributions files" (this is the
>  increase download reliability part - can be had by doing
>  opt-in CDN caching/proxying of external links via PyPI)
> 
> Now, with V3 of the proposal, you are moving towards a system
> that basically says "do it this way, or stay out of our eco
> system", which, in my book, is not what the Python eco system
> is all about.
> 

I don't see how? The -with-externals index will still contain all the existing links, and indeed PJ Elby has already stated that setuptools will move to support this index by default but with proper warnings to people so they know they are installing a package off site.

This allows existing tools to be moved to a secure by default position. Allows future tools to choose if they want to enable the existing behavior through use of -with-externals (hopefully with a warning or opt-in sort of thing like laid out by PJE, but it's certainly not required). And even allows users of existing tools to opt into the old behavior via the -i option.

Maybe i'm missing it but in what way does this force authors to "do it this way or stay out of our eco system" since all the same options are available as there are today?

> Your V2 was much more inviting in this respect.
> 
> -- 
> Marc-Andre Lemburg
> eGenix.com
> 
> Professional Python Services directly from the Source  (#1, Mar 13 2013)
>>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
> ________________________________________________________________________
> 
> ::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::
> 
>   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>           Registered at Amtsgericht Duesseldorf: HRB 46611
>               http://www.egenix.com/company/contact/
> _______________________________________________
> Catalog-SIG mailing list
> Catalog-SIG at python.org
> http://mail.python.org/mailman/listinfo/catalog-sig

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/catalog-sig/attachments/20130313/a2911175/attachment-0001.pgp>