[Python-checkins] distutils2: Add a fallback mechanism to use mirrors
tarek.ziade
python-checkins at python.org
Sun Jul 4 11:48:40 CEST 2010
tarek.ziade pushed 7d8b2a7faeca to distutils2:
http://hg.python.org/distutils2/rev/7d8b2a7faeca
changeset: 318:7d8b2a7faeca
user: Alexis Metaireau <ametaireau at gmail.com>
date: Tue Jun 22 15:46:13 2010 +0200
summary: Add a fallback mechanism to use mirrors
files: docs/source/pypi.rst, src/distutils2/pypi/dist.py, src/distutils2/pypi/errors.py, src/distutils2/pypi/simple.py, src/distutils2/tests/pypiserver/foo_bar_baz/simple/bar/index.html, src/distutils2/tests/pypiserver/foo_bar_baz/simple/baz/index.html, src/distutils2/tests/pypiserver/foo_bar_baz/simple/foo/index.html, src/distutils2/tests/pypiserver/foo_bar_baz/simple/index.html, src/distutils2/tests/test_pypi_simple.py
diff --git a/docs/source/pypi.rst b/docs/source/pypi.rst
--- a/docs/source/pypi.rst
+++ b/docs/source/pypi.rst
@@ -1,6 +1,6 @@
-=================
-The PyPI Module
-=================
+=========================================
+Tools to query PyPI: the PyPI package
+=========================================
Distutils2 comes with a module (eg. `distutils2.pypi`) which contains
facilities to access the Python Package Index (named "pypi", and avalaible on
@@ -15,34 +15,6 @@
Distutils2 provides two python modules to ease the work with those two APIs:
`distutils2.pypi.simple` and `distutils2.pypi.xmlrpc`.
-`distutils2.pypi.dist`
-======================
-
-Both `SimpleIndex` and `XmlRpcIndex` classes works with the classes provided
-in the `pypi.dist` package.
-
-`PyPIDistribution`
-------------------
-
-`PyPIDistribution` is a simple class that defines the following attributes:
-
-:name:
- The name of the package. `foobar` in our exemples here
-:version:
- The version of the package
-:location:
- If the files from the archive has been downloaded, here is the path where
- you can find them.
-:url:
- The url of the distribution
-
-
-`PyPIDistributions`
--------------------
-
-The `dist` module also provides another class, to work with lists of
-`PyPIDistribution` classes.
-
Requesting information via the "simple" API `distutils2.pypi.simple`
====================================================================
@@ -64,8 +36,17 @@
* Things that will end up in too long index processing (like "finding all
distributions with a specific version, no matters the name")
+API
+----
+
+.. autoclass:: distutils2.pypi.simple.SimpleIndex
+ :members:
+
+Usage Exemples
+---------------
+
Request PyPI to get a specific distribution
---------------------------------------------
+++++++++++++++++++++++++++++++++++++++++++++
Supposing you want to scan the PyPI index to get a list of distributions for
the "foobar" project. You can use the "search" method for that.::
@@ -89,7 +70,7 @@
<PyPIDistribution "foobar 1.1">
Download distributions
-----------------------
++++++++++++++++++++++++
As it can get the urls of distributions provided by PyPI, the `Simple` client
also can download the distributions and put it for you in a temporary
@@ -107,20 +88,26 @@
will try another time, then if fails again, raise `MD5HashDoesNotMatchError`.
Requesting external pages
--------------------------
++++++++++++++++++++++++++
The default behavior for distutils2 is to follow the links provided
-by distributions within their metadatas, and find distributions related
+by HTML pages in the "simple index", to find distributions related
downloads.
-When downloading, then, it will first use the packages found ont PyPI, then
-falling back to external webpages if needed.
-
It's possible to tell the PyPIClient to not follow external links by specifying
a list of allowed hosts::
>>> client = SimpleIndex(hosts=("*.python.org"))
+Working with mirrors
++++++++++++++++++++++
+
+The SimpleClient implement a fallback mechanism to switch from one mirror to
+another, the simple way. All you need to do is to provide a list of mirrors to
+it at the instanciation time::
+
+ >>> client = SimpleClient(mirrors=['http://mirror1/', 'http://mirror2/'])
+
Requesting informations via XML-RPC (`distutils2.pypi.XmlRpcIndex`)
==========================================================================
@@ -135,6 +122,40 @@
>>> client = XmlRpcIndex()
+PyPI Distributions
+==================
+
+Both `SimpleIndex` and `XmlRpcIndex` classes works with the classes provided
+in the `pypi.dist` package.
+
+`PyPIDistribution`
+------------------
+
+`PyPIDistribution` is a simple class that defines the following attributes:
+
+:name:
+ The name of the package. `foobar` in our exemples here
+:version:
+ The version of the package
+:location:
+ If the files from the archive has been downloaded, here is the path where
+ you can find them.
+:url:
+ The url of the distribution
+
+.. autoclass:: distutils2.pypi.dist.PyPIDistribution
+ :members:
+
+`PyPIDistributions`
+-------------------
+
+The `dist` module also provides another class, to work with lists of
+`PyPIDistribution` classes. It allow to filter results and is used as a
+container of
+
+.. autoclass:: distutils2.pypi.dist.PyPIDistributions
+ :members:
+
At a higher level
=================
diff --git a/src/distutils2/pypi/dist.py b/src/distutils2/pypi/dist.py
--- a/src/distutils2/pypi/dist.py
+++ b/src/distutils2/pypi/dist.py
@@ -34,9 +34,7 @@
"""Build a Distribution from a url archive (egg or zip or tgz).
:param url: complete url of the distribution
- :param probable_dist_name: the probable name of the distribution. This
- could be useful when multiple name can be assumed from the archive
- name.
+ :param probable_dist_name: A probable name of the distribution.
"""
# if the url contains a md5 hash, get it.
md5_hash = None
diff --git a/src/distutils2/pypi/errors.py b/src/distutils2/pypi/errors.py
--- a/src/distutils2/pypi/errors.py
+++ b/src/distutils2/pypi/errors.py
@@ -23,3 +23,7 @@
class MD5HashDoesNotMatch(DownloadError):
"""Compared MD5 hashes does not match"""
+
+
+class UnableToDownload(PyPIError):
+ """All mirrors have been tried, without success"""
diff --git a/src/distutils2/pypi/simple.py b/src/distutils2/pypi/simple.py
--- a/src/distutils2/pypi/simple.py
+++ b/src/distutils2/pypi/simple.py
@@ -15,7 +15,8 @@
from distutils2.version import VersionPredicate
from distutils2.pypi.dist import PyPIDistribution, PyPIDistributions, \
EXTENSIONS
-from distutils2.pypi.errors import PyPIError, DistributionNotFound
+from distutils2.pypi.errors import PyPIError, DistributionNotFound, \
+ DownloadError, UnableToDownload
from distutils2 import __version__ as __distutils2_version__
# -- Constants -----------------------------------------------
@@ -43,11 +44,13 @@
"""Decorator to add a socket timeout when requesting pages on PyPI.
"""
def _socket_timeout(func):
- def _socket_timeout(*args, **kwargs):
+ def _socket_timeout(self, *args, **kwargs):
old_timeout = socket.getdefaulttimeout()
+ if hasattr(self, "_timeout"):
+ timeout = self._timeout
socket.setdefaulttimeout(timeout)
try:
- return func(*args, **kwargs)
+ return func(self, *args, **kwargs)
finally:
socket.setdefaulttimeout(old_timeout)
return _socket_timeout
@@ -59,7 +62,7 @@
"""
def __init__(self, url=PYPI_DEFAULT_INDEX_URL, hosts=DEFAULT_HOSTS,
- mirrors=[]):
+ mirrors=[], timeout=SOCKET_TIMEOUT):
"""Class constructor.
:param index_url: the url of the simple index to search on.
@@ -70,9 +73,10 @@
:param mirrors: a list of mirrors to check out if problems occurs while
working with the one given in "url"
"""
- self.index_url = url
- self.mirrors = mirrors
- self._hosts = hosts
+ self._index_urls = [url]
+ self._index_urls.extend(mirrors)
+ self._current_index_url = 0
+ self._timeout = timeout
# create a regexp to match all given hosts
self._allowed_hosts = re.compile('|'.join(map(translate, hosts))).match
@@ -99,8 +103,9 @@
requirements.
:param requirements: A project name and it's distribution, using
- version specifiers, as described in PEP345. You can pass either a
- version.VersionPredicate or a string.
+ version specifiers, as described in PEP345.
+ :type requirements: You can pass either a version.VersionPredicate
+ or a string.
"""
requirements = self._get_version_predicate(requirements)
@@ -124,6 +129,8 @@
is given, creates and return one.
Returns the complete absolute path to the downloaded archive.
+
+ :param requirements: The same as the find attribute of `find`.
"""
return self.get(requirements).download(path=temp_path)
@@ -134,7 +141,20 @@
if isinstance(requirements, str):
requirements = VersionPredicate(requirements)
return requirements
+
+ @property
+ def index_url(self):
+ return self._index_urls[self._current_index_url]
+ def _switch_to_next_mirror(self):
+ """Switch to the next mirror (eg. point self.index_url to the next
+ url.
+ """
+ if self._current_index_url < len(self._index_urls):
+ self._current_index_url = self._current_index_url + 1
+ else:
+ raise UnableToDownload("All mirrors fails")
+
def _is_browsable(self, url):
"""Tell if the given URL needs to be browsed or not, according to the
object internal attributes.
@@ -229,9 +249,16 @@
:param name: the name of the project to find the page
"""
- # Browse and index the content of the given PyPI page.
- url = self.index_url + name + "/"
- self._process_url(url, name)
+ try:
+ # Browse and index the content of the given PyPI page.
+ url = self.index_url + name + "/"
+ self._process_url(url, name)
+ except DownloadError:
+ # if an error occurs, try with the next index_url
+ # (provided by the mirrors)
+ self._switch_to_next_mirror()
+ self._distributions.clear()
+ self._process_pypi_page(name)
@socket_timeout()
def _open_url(self, url):
@@ -273,12 +300,12 @@
except urllib2.HTTPError, v:
return v
except urllib2.URLError, v:
- raise PyPIError("Download error for %s: %s" % (url, v.reason))
+ raise DownloadError("Download error for %s: %s" % (url, v.reason))
except httplib.BadStatusLine, v:
- raise PyPIError('%s returned a bad status line. '
+ raise DownloadError('%s returned a bad status line. '
'The server might be down, %s' % (url, v.line))
except httplib.HTTPException, v:
- raise PyPIError("Download error for %s: %s" % (url, v))
+ raise DownloadError("Download error for %s: %s" % (url, v))
def _decode_entity(self, match):
what = match.group(1)
diff --git a/src/distutils2/tests/pypiserver/foo_bar_baz/simple/bar/index.html b/src/distutils2/tests/pypiserver/foo_bar_baz/simple/bar/index.html
new file mode 100644
--- /dev/null
+++ b/src/distutils2/tests/pypiserver/foo_bar_baz/simple/bar/index.html
@@ -0,0 +1,6 @@
+<html><head><title>Links for bar</title></head><body><h1>Links for bar</h1>
+<a rel="download" href="../../packages/source/F/bar/bar-1.0.tar.gz">bar-1.0.tar.gz</a><br/>
+<a rel="download" href="../../packages/source/F/bar/bar-1.0.1.tar.gz">bar-1.0.1.tar.gz</a><br/>
+<a rel="download" href="../../packages/source/F/bar/bar-2.0.tar.gz">bar-2.0.tar.gz</a><br/>
+<a rel="download" href="../../packages/source/F/bar/bar-2.0.1.tar.gz">bar-2.0.1.tar.gz</a><br/>
+</body></html>
diff --git a/src/distutils2/tests/pypiserver/foo_bar_baz/simple/baz/index.html b/src/distutils2/tests/pypiserver/foo_bar_baz/simple/baz/index.html
new file mode 100644
--- /dev/null
+++ b/src/distutils2/tests/pypiserver/foo_bar_baz/simple/baz/index.html
@@ -0,0 +1,6 @@
+<html><head><title>Links for baz</title></head><body><h1>Links for baz</h1>
+<a rel="download" href="../../packages/source/F/baz/baz-1.0.tar.gz">baz-1.0.tar.gz</a><br/>
+<a rel="download" href="../../packages/source/F/baz/baz-1.0.1.tar.gz">baz-1.0.1.tar.gz</a><br/>
+<a rel="download" href="../../packages/source/F/baz/baz-2.0.tar.gz">baz-2.0.tar.gz</a><br/>
+<a rel="download" href="../../packages/source/F/baz/baz-2.0.1.tar.gz">baz-2.0.1.tar.gz</a><br/>
+</body></html>
diff --git a/src/distutils2/tests/pypiserver/foo_bar_baz/simple/foo/index.html b/src/distutils2/tests/pypiserver/foo_bar_baz/simple/foo/index.html
new file mode 100644
--- /dev/null
+++ b/src/distutils2/tests/pypiserver/foo_bar_baz/simple/foo/index.html
@@ -0,0 +1,6 @@
+<html><head><title>Links for foo</title></head><body><h1>Links for foo</h1>
+<a rel="download" href="../../packages/source/F/foo/foo-1.0.tar.gz">foo-1.0.tar.gz</a><br/>
+<a rel="download" href="../../packages/source/F/foo/foo-1.0.1.tar.gz">foo-1.0.1.tar.gz</a><br/>
+<a rel="download" href="../../packages/source/F/foo/foo-2.0.tar.gz">foo-2.0.tar.gz</a><br/>
+<a rel="download" href="../../packages/source/F/foo/foo-2.0.1.tar.gz">foo-2.0.1.tar.gz</a><br/>
+</body></html>
diff --git a/src/distutils2/tests/pypiserver/foo_bar_baz/simple/index.html b/src/distutils2/tests/pypiserver/foo_bar_baz/simple/index.html
new file mode 100644
--- /dev/null
+++ b/src/distutils2/tests/pypiserver/foo_bar_baz/simple/index.html
@@ -0,0 +1,3 @@
+<a href="foo/">foo/</a>
+<a href="bar/">bar/</a>
+<a href="baz/">baz/</a>
diff --git a/src/distutils2/tests/test_pypi_simple.py b/src/distutils2/tests/test_pypi_simple.py
--- a/src/distutils2/tests/test_pypi_simple.py
+++ b/src/distutils2/tests/test_pypi_simple.py
@@ -8,7 +8,7 @@
import unittest2
import urllib2
-from distutils2.tests.pypi_server import use_pypi_server
+from distutils2.tests.pypi_server import use_pypi_server, PyPIServer
from distutils2.pypi import simple
from distutils2.errors import DistutilsError
@@ -222,11 +222,24 @@
self.assertIn("%s/foobar-2.0.tar.gz" % server.full_address,
index._processed_urls) # linked from external homepage (rel)
-# def test_uses_mirrors(self):
-# """When the main repository seems down, try using the given mirrors"""
-# server =
-# mirror =
-#
+ def test_uses_mirrors(self):
+ """When the main repository seems down, try using the given mirrors"""
+ server = PyPIServer("foo_bar_baz")
+ mirror = PyPIServer("foo_bar_baz")
+ mirror.start() # we dont start the server here
+
+ try:
+ # create the index using both servers
+ index = simple.SimpleIndex(server.full_address + "/simple/",
+ hosts=('*',), timeout=1, # set the timeout to 1s for the tests
+ mirrors=[mirror.full_address + "/simple/",])
+
+ # this should not raise a timeout
+ self.assertEqual(4, len(index.find("foo")))
+ except Exception, e:
+ mirror.stop()
+ raise e
+
def test_suite():
return unittest2.makeSuite(PyPISimpleTestCase)
--
Repository URL: http://hg.python.org/distutils2
More information about the Python-checkins
mailing list