[Python-checkins] r70705 - peps/trunk/pep-0381.txt
python-checkins at python.org
Mon Mar 30 06:53:46 CEST 2009
Date: Mon Mar 30 06:53:39 2009
New Revision: 70705
various fixes, more details
--- peps/trunk/pep-0381.txt (original)
+++ peps/trunk/pep-0381.txt Mon Mar 30 06:53:39 2009
@@ -10,6 +10,7 @@
@@ -35,7 +36,6 @@
by PyPI, in order to get hit counts and the last modified date.
- how a mirror should synchronize with PyPI
- how a client can implement a fail-over mechanism
-- a contact form for Package maintainers
Mirror listing and registering
@@ -46,31 +46,59 @@
added in a mirror list in the PyPI application after it
has been checked to be compliant with the mirroring rules.
-A mirror has to be a hostname. For example:
The mirror list is handled by a DNS entry for this hostname:
When a mirror is added into the DNS, it becomes an official
-IP for `mirrors.pypi.python.org`.
+IP for `mirrors.pypi.python.org`, and requests will be sent
+to the given IP. Therefore the mirror maintainer should not
+change the IP provided. If the IP has to change for any reason,
+the mirror maintainer has to send a mail to catalog-SIG at least
+one week before the change so the DNS entry can be changed on time.
The new mirror also appears at `http://pypi.python.org/mirrors`
which is a human-readable page that gives the list of mirrors.
This page also explains how to register a new mirror.
+PyPI provides statistics on downloads at `/stats`. This page is
+calculated daily by PyPI, by reading all mirrors' local stats and
+The stats are presented in daily or montly files, under `/stats/days`
+and `/stats/months`. Each file is a `bzip2` file with these formats:
+- YYYY-MM-DD.bz2 for daily files
+- YYYY-MM.bz2 for monthly files
Special pages a mirror needs to provide
-A mirror needs to provide four pages, beside the index one:
+A mirror is a strict copy of PyPI, so it provides the same structure
+by copying it.
+- pypi: html version of the package index
+- simple: rest version of the package index
+- packages: packages, stored by Python version, and letters
+- stats : statistics on downloads
+It also needs to provide two specific elements:
Last modified date
@@ -81,12 +109,11 @@
For PyPI, each mirror needs to maintain a URL with simple text content
that represents the last synchronisation date the mirror maintains.
-The date is provided in GMT time, using the ISO 8601 format (see
+The date is provided in GMT time, using the ISO 8601 format [#iso8601]_.
Each mirror will be responsible to maintain its last modified date.
-Conventionally, this page should be reachable at: `/last-modified`.
+This page must be located at : `/last-modified` and must be a
@@ -124,42 +151,7 @@
-Conventionally the name should be `local-stats`, but it can be any
-name provided when the mirror is registered.
-PyPI and each mirror are responsible to provide the grand total page
-at `/stats`. This page is calculated daily by PyPI, by reading all
-mirrors' local stats and summing them.
-Therefore the mirrors should not try to rebuild this stat page but
-simply get the one on PyPI during each synchronization.
-It has the same structure as `local-stats` but also provides counts
-Conventionally the name should be `stats`, but it can be any name
-provided when the mirror is registered.
-Mirrors listing page
-Like `/stats`, each mirror should get and provide a copy of the
-Conventionally the name should be `mirrors`, but it can be any name
-provided when the mirror is registered.
+This page must be located at `/local-stats`.
How a mirror should synchronize with PyPI
@@ -175,7 +167,7 @@
XXX Need to describe the protocol here.
-The zc.pypimirror package [#zcpkg]_ provides an application that
+The z3c.pypimirror package [#zcpkg]_ provides an application that
respects this protocol to browse PyPI.
User-agent request header
@@ -187,10 +179,9 @@
This is also true for all clients like:
-- `zc.buildout <http://pypi.python.org/pypi/zc.buildout>`_
-- `setuptools <http://pypi.python.org/pypi/zc.buildout>`_
-- `pip <http://pypi.python.org/pypi/zc.buildout>`_
+- zc.buildout [#zc.buildout]_.
+- setuptools [#setuptools]_.
+- pip [#pip]_.
XXX user agent registering mechanism at PyPI ?
@@ -198,7 +189,13 @@
Clients that are browsing PyPI should be able to use alternative
-mirrors, by reading the `/mirrors` page at PyPI.
+mirrors, by getting the list of the mirrors using `mirrors.pypi.python.org`.
+ >>> import socket
+ >>> socket.gethostbyname_ex('mirrors.pypi.python.org')[-1]
+ ['18.104.22.168', '22.214.171.124']
The clients so far that could use this mechanism:
@@ -212,9 +209,6 @@
Clients that are browsing PyPI should be able to use a fail-over
mechanism when PyPI or the used mirror is not responding.
-This can be done by parsing the `/mirrors` page of PyPI or the one
-located on any PyPI mirror.
It is up to the client to decide wich mirror should be used, maybe by
looking at its geographical location and its responsivness.
@@ -242,9 +236,8 @@
Softwares that are compatible with PyPI and Distutils so far:
-- `PloneSoftwareCenter <http://plone.org/products/plonesoftwarecenter>`_
- wich is used to run plone.org products section.
-- `EggBasket <http://www.chrisarndt.de/projects/eggbasket>`_
+- PloneSoftwareCenter [#psc]_ wich is used to run plone.org products section.
+- EggBasket [#eggbasket]_.
**An extra package index is not a mirror of PyPI, but can have some
@@ -257,8 +250,10 @@
source of packages. Different indexes should be defined as a sorted
list for the client to look for a package.
-Each independant index can of course provide a list of its mirrors, if
-the `/mirrors` page is available.
+Each independant index can of course provide a list of its mirrors.
+XXX define how to get the hostname for the mirrors of an arbitrary
That permits all combinations at client level, for a reliable
packaging system with all levels of privacy.
@@ -275,6 +270,30 @@
+Martin von Loewis, Georg Brandl.
More information about the Python-checkins