[Distutils] fail-over and index merging for tools like setuptools and pyinstall

[Tarek Ziadé]

Hi ! Sorry for the crosspost, but this mail concerns both lists,

There are two points that need to be discussed to finish the work on
the PyPI proposal started in D.C. (http://wiki.python.org/PEP_374):

- the fail-over mechanism
- merging several indexes

*Fail over*

PyPI will provide a static page that lists all its mirrors. Each line
of this file describes a mirror.
It provides the root url, followed by the relative url of:

  - the index : the root of the package index
  - the last-modified page : a static text file that gives the date of
the last sync.
  - the local stats page: a static text file that gives the number of
downloads of a file, per package,  per user-agent
  - the global stats page, calculated by pypi that gives the grand
total of all downloads (the sum of PyPI local stats + mirrors local
stats)
  - the mirrors page, that lists all mirrors

For example:

    http://example.com/pypi,index,last-modified,local-stats,stats,mirrors
    http://example2.com/pypi,index,last-modified,local-stats,stats,mirrors

(see the proposal doc for more info)

This mirror list says for example that a mirror is available at
http://example.com/pypi/index, and that its last modified
date is available at http://example.com/pypi/last-modified.

On client side it means that it is possible to list mirrors of a given
package index to implement a fail-over
mechanism. Moreover, it makes it possible to select the nearest mirror.

*Merging several indexes*

Besides fail over, another thing needs to be implemented
on client side: being able to use different indexes.

This is an obvious missing feature: we don't want to push in PyPI all
our customers package.
In the meantime we do want to use tools like distutils, setuptools
etc., the same way with any kind of package.
So using private package indexes easily besides PyPI is needed.

It is now possible in Python 2.6 with the new .pypirc file to define
several indexes.
>From there softwares like PloneSoftwareCenter allows developers to
work with other indexes than PyPI.

But tools like setuptools need to evolve the same way.

Each one of this index can have its own mirrors, as defined
previously, but the client needs to combine all the different
index, into a "super" index.

This can be implemented by working with a sorted list of index. When a
client is looking for a package, it can
look in each index and pick the first package that fits.


Any comments ?

Cheers
Tarek

-- 
Tarek Ziadé | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/