[Distutils] Fwd: The state of PyPI

[Tarek Ziadé]

I have sent that to the PSF list because there's a PSF project about PyPI infra.

But someone complained, saying that I was doing this discussion
"behind closed doors"

SInce this is not my goal, I am now spamming more lists...


---------- Forwarded message ----------
From: Tarek Ziadé <ziade.tarek at gmail.com>
Date: Tue, Sep 27, 2011 at 10:37 AM
Subject: The state of PyPI
To: PSF Members List <psf-members at python.org>
Cc: Richard Jones <r1chardj0n3s at gmail.com>, Steve Holden <steve at holdenweb.com>


Hey

This is just a mail that summarizes the current state of PyPI, the
existing features, and what can be done next to improve stuff.

I am sending this in the PSF members list because we had a project of
an infrastructure going on, and I want to make sure all involved
parties are in the same page.

1/ stability and high availability
2/ private mirrors
3/ private projects
4/ tutorial ?


= stability and high availability =

we went in two directions to improve PyPI :

1/ add the mirroring protocol
2/ make the PyPI server more reliable by pushing its storage in a
redundant cloud.

== mirroring ==

The mirroring protocol (PEP 381) is implemented on server-side, I've
worked with Martin on this, and we have mirrors now:

Look at http://pypi.python.org/mirrors

Also, there's a client that anyone can use to set up a mirror:
http://pypi.python.org/pypi/pep381client

The idea is that anyone in the community willing to maintain a mirror
can do so. We add the mirror in the CND, and make it available for
client tools to use. What's really missing right now is more
integration on client-side.

- Pip supports the mirroring protocol, and can fall back to a mirror,
but I am not 100% sure this is a default behavior.  (please correct me
if it is now)
- Buildout knows how to use *another server* than the main PyPI, so
can manually switch to a mirror, but I don't think it's transparent.
It should.
- Distribute/Setuptools does not do anything for this, and should.
- everything is already implemented in packaging/distutils2

The effect of the mirrors is that PyPI being down should not impact
the community. This will be true once all tool are transparently using
the mirrors.

== better infra ==

I think the project is staled right now.


= private mirrors =

Having a private mirror makes a lot of sense, when companies need to
make sure their build systems are not relying on external services
like PyPI or a mirror. It's also a good way to dramatically reduce the
load for the community servers.

The idea is that a Jenkins server that builds hundreds of Python apps
every hour should not hammer PyPI.

We have everything needed these days to set up this kind of system,
with zc.buildout or pip good practices.

What we need is a good tutorial or a guide [*]

= private projects =

The part that we do not address in the community is private projects:
since we don't have any permissions/group/roles system in PyPI,
everything is public.

One way to solve this is to have a local repository for private
packages, that is looked by tools like pip or easy_install, with the
--find-links option.


What we need is a good tutorial or a guide [*]

= tutorial =

[*] If this helps, I am willing to work on a tutorial day for Pycon
US, that goes through all of this, to help people set up their dev.
environment the best way possible.

The material could then be published at python.org/pypi to help out.

I know Richard has some material already, so maybe this could be a
joint tutorial ?

HTH

Cheers
Tarek





--
Tarek Ziadé | http://ziade.org



-- 
Tarek Ziadé | http://ziade.org