[Catalog-sig] Next generation package infrastructure (was: Deprecate External Links)

[M.-A. Lemburg]

On 28.02.2013 09:43, Nick Coghlan wrote:
> On Thu, Feb 28, 2013 at 6:12 PM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 28.02.2013 07:39, Nick Coghlan wrote:
>>> 1. The next generation metadata infrastructure will NOT support
>>> external hosting of files indexed on PyPI - if you don't upload the
>>> archive files to PyPI, they won't be included in the next generation
>>> metadata. If you want external hosting, you will need to run a
>>> separate index (this is similar to the yum model - you can host files
>>> wherever you want, but you need to run "yum createrepo" yourself to
>>> generate the metadata, and instruct users on how to get their
>>> installers to retrieve your metadata. The big difference between PyPI
>>> and the yum model is that the default index still won't be curated at
>>> all, so there's no review process to get through if you want to use
>>> it, thus less need for external hosting).
>>
>> Could you elaborate on this ?
>>
>> AFAIK, the metadata only works on package names, regardless of where
>> an installer finds them.
> 
> Caveat: this is NOT a final design, and people that aren't me will be
> working out the exact details. It is, however, how I want it to work.

:-)

> The next generation metadata publication infrastructure is likely to
> be based on TUF, and thus will consist of pregenerated, signed
> metadata served as static files. Installers will just download
> metadata files, sdists and wheels (and probably eggs and tarballs),
> and never need to contact an active web service. The only "active" web
> service technically required will be one to regularly refresh the
> signed timestamp file that prevents certain kinds of attacks based on
> providing old, insecure versions of software (a cron job running on
> the server hosting the metadata would suffice for this task). PyPI
> itself will have another active service to automatically regenerate
> the metadata when files are uploaded by maintainers. The delegation of
> trust within the framework will be defined only for files hosted by
> PyPI - it will not be extended to allow the declaration of external
> URLs as a source for the target files.
> 
> Publishers will still be able to publish on external sites, but they
> will need to generate their own metadata, and distributions published
> that way won't be indexed in the next generation metadata on PyPI.
> This is the same way yum repos work - the metadata for each repo only
> covers SRPMs and RPMs hosted in that repo. If you want to download
> software from somewhere else, you have to add another repo definition
> in the client so it knows where to look for the metadata. APT works in
> a similar fashion.

Thanks for the added details. This sounds like a major overhaul
of the whole package infrastructure, you have in mind there :-)

In order for this to work out, you will need to get the
support of people hosting packages externally and address
their concerns.

The current discussion has been too dogmatic for my taste.
A more pragmatic approach would likely be a more reasonable
and successful way to achieve a transition.


Just as aside:

The reason why apt, yum, zypper, etc. have different
repos is that those tools work based on trust which is achieved
by reviews of the packages in those repos. As user you trust
the review process and want to be sure that you are getting
the packages that have actually been reviewed.

With PyPI, the situation is different, since we don't have
a review process for packages. The only guarantee we can provide
is that the packages that you download via tools from PyPI
or elsewhere match the ones that the authors of the packages
created. This requires more trust on the authors, so we better
make sure that they are happy with it ;-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 28 2013)
>>> Python Projects, Consulting and Support ...   http://www.egenix.com/
>>> mxODBC.Zope/Plone.Database.Adapter ...       http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/