[Python-checkins] distutils2: merge konryd works about pypi test server
tarek.ziade
python-checkins at python.org
Sun Jul 4 11:48:39 CEST 2010
tarek.ziade pushed c217e36b984a to distutils2:
http://hg.python.org/distutils2/rev/c217e36b984a
changeset: 284:c217e36b984a
parent: 283:d6099bdc57cd
parent: 275:944a00b90bde
user: Alexis Metaireau <ametaireau at gmail.com>
date: Fri May 28 11:24:53 2010 +0200
summary: merge konryd works about pypi test server
files:
diff --git a/docs/design/pep-0376.txt b/docs/design/pep-0376.txt
--- a/docs/design/pep-0376.txt
+++ b/docs/design/pep-0376.txt
@@ -633,7 +633,7 @@
Distributions installed using existing, pre-standardization formats do not have
the necessary metadata available for the new API, and thus will be
-ignored. Third-party tools may of course to continue to support previous
+ignored. Third-party tools may of course continue to support previous
formats in addition to the new format, in order to ease the transition.
diff --git a/docs/design/wiki.rst b/docs/design/wiki.rst
--- a/docs/design/wiki.rst
+++ b/docs/design/wiki.rst
@@ -282,7 +282,7 @@
mailman/etc/* = {config} # 8
mailman/foo/**/bar/*.cfg = {config}/baz # 9
mailman/foo/**/*.cfg = {config}/hmm # 9, 10
- some-new-semantic.txt = {funky-crazy-category} # 11
+ some-new-semantic.sns = {funky-crazy-category} # 11
The glob definitions are relative paths that match files from the top
of the source tree (the location of ``setup.cfg``). Forward slashes
diff --git a/docs/source/index.rst b/docs/source/index.rst
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -12,6 +12,8 @@
:maxdepth: 2
metadata
+ new_commands
+ test_framework
Indices and tables
==================
diff --git a/docs/source/new_commands.rst b/docs/source/new_commands.rst
new file mode 100644
--- /dev/null
+++ b/docs/source/new_commands.rst
@@ -0,0 +1,65 @@
+========
+Commands
+========
+
+Distutils2 provides a set of commands that are not present in distutils itself.
+You might recognize some of them from other projects, like Distribute or
+Setuptools.
+
+``upload_docs`` - Upload package documentation to PyPI
+======================================================
+
+PyPI now supports uploading project documentation to the dedicated URL
+http://packages.python.org/<project>/.
+
+The ``upload_docs`` command will create the necessary zip file out of a
+documentation directory and will post to the repository.
+
+Note that to upload the documentation of a project, the corresponding version
+must already be registered with PyPI, using the distutils ``register``
+command -- just like the ``upload`` command.
+
+Assuming there is an ``Example`` project with documentation in the
+subdirectory ``docs``, e.g.::
+
+ Example/
+ |-- example.py
+ |-- setup.cfg
+ |-- setup.py
+ |-- docs
+ | |-- build
+ | | `-- html
+ | | | |-- index.html
+ | | | `-- tips_tricks.html
+ | |-- conf.py
+ | |-- index.txt
+ | `-- tips_tricks.txt
+
+You can simply pass the documentation directory path to the ``upload_docs``
+command::
+
+ python setup.py upload_docs --upload-dir=docs/build/html
+
+As with any other ``setuptools`` based command, you can define useful
+defaults in the ``setup.cfg`` of your Python project, e.g.:
+
+.. code-block:: ini
+
+ [upload_docs]
+ upload-dir = docs/build/html
+
+The ``upload_docs`` command has the following options:
+
+``--upload-dir``
+ The directory to be uploaded to the repository. The default value is
+ ``docs`` in project root.
+
+``--show-response``
+ Display the full response text from server; this is useful for debugging
+ PyPI problems.
+
+``--repository=URL, -r URL``
+ The URL of the repository to upload to. Defaults to
+ http://pypi.python.org/pypi (i.e., the main PyPI installation).
+
+
diff --git a/docs/source/test_framework.rst b/docs/source/test_framework.rst
new file mode 100644
--- /dev/null
+++ b/docs/source/test_framework.rst
@@ -0,0 +1,34 @@
+==============
+Test Framework
+==============
+
+When you are testing code that works with distutils, you might find these tools
+useful.
+
+
+``PyPIServer``
+==============
+
+PyPIServer is a class that implements an HTTP server running in a separate
+thread. All it does is record the requests for further inspection. The recorded
+data is available under ``requests`` attribute. The default
+HTTP response can be overriden with the ``default_response_status``,
+``default_response_headers`` and ``default_response_data`` attributes.
+
+
+``PyPIServerTestCase``
+======================
+
+``PyPIServerTestCase`` is a test case class with setUp and tearDown methods that
+take care of a single PyPIServer instance attached as a ``pypi`` attribute on
+the test class. Use it as one of the base classes in you test case::
+
+ class UploadTestCase(PyPIServerTestCase):
+ def test_something(self):
+ cmd = self.prepare_command()
+ cmd.ensure_finalized()
+ cmd.repository = self.pypi.full_address
+ cmd.run()
+
+ environ, request_data = self.pypi.requests[-1]
+ self.assertEqual(request_data, EXPECTED_REQUEST_DATA)
diff --git a/src/distutils2/command/upload_docs.py b/src/distutils2/command/upload_docs.py
new file mode 100644
--- /dev/null
+++ b/src/distutils2/command/upload_docs.py
@@ -0,0 +1,128 @@
+import base64, httplib, os.path, socket, tempfile, urlparse, zipfile
+from cStringIO import StringIO
+from distutils2 import log
+from distutils2.core import PyPIRCCommand
+from distutils2.errors import DistutilsFileError
+
+def zip_dir(directory):
+ """Compresses recursively contents of directory into a StringIO object"""
+ destination = StringIO()
+ zip_file = zipfile.ZipFile(destination, "w")
+ for root, dirs, files in os.walk(directory):
+ for name in files:
+ full = os.path.join(root, name)
+ relative = root[len(directory):].lstrip(os.path.sep)
+ dest = os.path.join(relative, name)
+ zip_file.write(full, dest)
+ zip_file.close()
+ return destination
+
+# grabbed from
+# http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/
+def encode_multipart(fields, files, boundary=None):
+ """
+ fields is a sequence of (name, value) elements for regular form fields.
+ files is a sequence of (name, filename, value) elements for data to be uploaded as files
+ Return (content_type, body) ready for httplib.HTTP instance
+ """
+ if boundary is None:
+ boundary = '--------------GHSKFJDLGDS7543FJKLFHRE75642756743254'
+ CRLF = '\r\n'
+ l = []
+ for (key, value) in fields:
+ l.extend([
+ '--' + boundary,
+ 'Content-Disposition: form-data; name="%s"' % key,
+ '',
+ value])
+ for (key, filename, value) in files:
+ l.extend([
+ '--' + boundary,
+ 'Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename),
+ '',
+ value])
+ l.append('--' + boundary + '--')
+ l.append('')
+ body = CRLF.join(l)
+ content_type = 'multipart/form-data; boundary=%s' % boundary
+ return content_type, body
+
+class upload_docs(PyPIRCCommand):
+
+ user_options = [
+ ('upload-dir=', None, 'directory to upload'),
+ ]
+
+ def initialize_options(self):
+ self.upload_dir = None
+
+ def finalize_options(self):
+ PyPIRCCommand.finalize_options(self)
+ if self.upload_dir == None:
+ build = self.get_finalized_command('build')
+ self.upload_dir = os.path.join(build.build_base, "docs")
+ config = self._read_pypirc()
+ if config != {}:
+ self.username = config['username']
+ self.password = config['password']
+ self.repository = config['repository']
+ self.realm = config['realm']
+
+ def verify_upload_dir(self, upload_dir):
+ index_location = os.path.join(upload_dir, "index.html")
+ if not os.path.exists(index_location):
+ mesg = "No 'index.html found in docs directory (%s)"
+ raise DistutilsFileError(mesg % upload_dir)
+
+
+ def run(self):
+ tmp_dir = tempfile.mkdtemp()
+ name = self.distribution.metadata['Name']
+ self.verify_upload_dir(self.upload_dir)
+ zip_file = zip_dir(self.upload_dir)
+
+ fields = {':action': 'doc_upload', 'name': name}.items()
+ files = [('content', name, zip_file.getvalue())]
+ content_type, body = encode_multipart(fields, files)
+
+ credentials = self.username + ':' + self.password
+ auth = "Basic " + base64.encodestring(credentials).strip()
+
+ self.announce("Submitting documentation to %s" % (self.repository),
+ log.INFO)
+
+ schema, netloc, url, params, query, fragments = \
+ urlparse.urlparse(self.repository)
+ if schema == "http":
+ conn = httplib.HTTPConnection(netloc)
+ elif schema == "https":
+ conn = httplib.HTTPSConnection(netloc)
+ else:
+ raise AssertionError("unsupported schema "+schema)
+
+ try:
+ conn.connect()
+ conn.putrequest("POST", url)
+ conn.putheader('Content-type', content_type)
+ conn.putheader('Content-length', str(len(body)))
+ conn.putheader('Authorization', auth)
+ conn.endheaders()
+ conn.send(body)
+ except socket.error, e:
+ self.announce(str(e), log.ERROR)
+ return
+
+ r = conn.getresponse()
+
+ if r.status == 200:
+ self.announce('Server response (%s): %s' % (r.status, r.reason),
+ log.INFO)
+ elif r.status == 301:
+ location = r.getheader('Location')
+ if location is None:
+ location = 'http://packages.python.org/%s/' % meta.get_name()
+ self.announce('Upload successful. Visit %s' % location,
+ log.INFO)
+ else:
+ self.announce('Upload failed (%s): %s' % (r.status, r.reason),
+ log.ERROR)
diff --git a/src/distutils2/tests/pypi_server.py b/src/distutils2/tests/pypi_server.py
new file mode 100644
--- /dev/null
+++ b/src/distutils2/tests/pypi_server.py
@@ -0,0 +1,63 @@
+import Queue, threading, time, unittest2
+from wsgiref.simple_server import make_server
+
+class PyPIServerTestCase(unittest2.TestCase):
+
+ def setUp(self):
+ super(PyPIServerTestCase, self).setUp()
+ self.pypi = PyPIServer()
+ self.pypi.start()
+
+ def tearDown(self):
+ super(PyPIServerTestCase, self).tearDown()
+ self.pypi.stop()
+
+class PyPIServer(threading.Thread):
+ """Thread that wraps a wsgi app"""
+ def __init__(self):
+ threading.Thread.__init__(self)
+ self.httpd = make_server('', 0, self.pypi_app)
+ self.httpd.RequestHandlerClass.log_request = lambda *_: None
+ self.address = self.httpd.server_address
+ self.request_queue = Queue.Queue()
+ self._requests = []
+ self.default_response_status = "200 OK"
+ self.default_response_headers = [('Content-type', 'text/plain')]
+ self.default_response_data = ["hello"]
+
+ def run(self):
+ self.httpd.serve_forever()
+
+ def stop(self):
+ self.httpd.shutdown()
+ self.join()
+
+ def pypi_app(self, environ, start_response):
+ # record the request
+ if environ.get("CONTENT_LENGTH"):
+ request_data = environ.pop('wsgi.input').read(int(environ['CONTENT_LENGTH']))
+ else:
+ request_data = environ.pop('wsgi.input').read()
+ self.request_queue.put((environ, request_data))
+ # send back a response
+ status, headers, data = self.get_next_response()
+ start_response(status, headers)
+ return data
+
+ def get_next_response(self):
+ return (self.default_response_status,
+ self.default_response_headers,
+ self.default_response_data)
+
+ @property
+ def requests(self):
+ while True:
+ try:
+ self._requests.append(self.request_queue.get_nowait())
+ except Queue.Empty:
+ break
+ return self._requests
+
+ @property
+ def full_address(self):
+ return "http://%s:%s" % self.address
diff --git a/src/distutils2/tests/test_pypi_server.py b/src/distutils2/tests/test_pypi_server.py
new file mode 100644
--- /dev/null
+++ b/src/distutils2/tests/test_pypi_server.py
@@ -0,0 +1,27 @@
+"""Tests for distutils.command.bdist."""
+import unittest2, urllib, urllib2
+from distutils2.tests.pypi_server import PyPIServer
+
+class PyPIServerTest(unittest2.TestCase):
+
+ def test_records_requests(self):
+ server = PyPIServer()
+ server.start()
+ self.assertEqual(len(server.requests), 0)
+
+ data = "Rock Around The Bunker"
+ headers = {"X-test-header": "Mister Iceberg"}
+
+ request = urllib2.Request(server.full_address, data, headers)
+ urllib2.urlopen(request)
+ self.assertEqual(len(server.requests), 1)
+ environ, request_data = server.requests[-1]
+ self.assertIn("Rock Around The Bunker", request_data)
+ self.assertEqual(environ["HTTP_X_TEST_HEADER"], "Mister Iceberg")
+ server.stop()
+
+def test_suite():
+ return unittest2.makeSuite(PyPIServerTest)
+
+if __name__ == '__main__':
+ unittest2.main(defaultTest="test_suite")
diff --git a/src/distutils2/tests/test_upload.py b/src/distutils2/tests/test_upload.py
--- a/src/distutils2/tests/test_upload.py
+++ b/src/distutils2/tests/test_upload.py
@@ -1,34 +1,13 @@
"""Tests for distutils.command.upload."""
# -*- encoding: utf8 -*-
-import sys
-import os
-import unittest2
+import os, unittest2
-from distutils2.command import upload as upload_mod
from distutils2.command.upload import upload
from distutils2.core import Distribution
-from distutils2.tests import support
+from distutils2.tests.pypi_server import PyPIServer, PyPIServerTestCase
from distutils2.tests.test_config import PYPIRC, PyPIRCCommandTestCase
-PYPIRC_LONG_PASSWORD = """\
-[distutils]
-
-index-servers =
- server1
- server2
-
-[server1]
-username:me
-password:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
-
-[server2]
-username:meagain
-password: secret
-realm:acme
-repository:http://another.pypi/
-"""
-
PYPIRC_NOPASSWORD = """\
[distutils]
@@ -40,38 +19,9 @@
username:me
"""
-class FakeOpen(object):
-
- def __init__(self, url):
- self.url = url
- if not isinstance(url, str):
- self.req = url
- else:
- self.req = None
- self.msg = 'OK'
-
- def getcode(self):
- return 200
-
-
-class uploadTestCase(PyPIRCCommandTestCase):
-
- def setUp(self):
- super(uploadTestCase, self).setUp()
- self.old_open = upload_mod.urlopen
- upload_mod.urlopen = self._urlopen
- self.last_open = None
-
- def tearDown(self):
- upload_mod.urlopen = self.old_open
- super(uploadTestCase, self).tearDown()
-
- def _urlopen(self, url):
- self.last_open = FakeOpen(url)
- return self.last_open
+class UploadTestCase(PyPIServerTestCase, PyPIRCCommandTestCase):
def test_finalize_options(self):
-
# new format
self.write_file(self.rc, PYPIRC)
dist = Distribution()
@@ -80,7 +30,7 @@
for attr, waited in (('username', 'me'), ('password', 'secret'),
('realm', 'pypi'),
('repository', 'http://pypi.python.org/pypi')):
- self.assertEquals(getattr(cmd, attr), waited)
+ self.assertEqual(getattr(cmd, attr), waited)
def test_saved_password(self):
# file with no password
@@ -89,44 +39,40 @@
# make sure it passes
dist = Distribution()
cmd = upload(dist)
- cmd.finalize_options()
- self.assertEquals(cmd.password, None)
+ cmd.ensure_finalized()
+ self.assertEqual(cmd.password, None)
# make sure we get it as well, if another command
# initialized it at the dist level
dist.password = 'xxx'
cmd = upload(dist)
cmd.finalize_options()
- self.assertEquals(cmd.password, 'xxx')
+ self.assertEqual(cmd.password, 'xxx')
def test_upload(self):
- tmp = self.mkdtemp()
- path = os.path.join(tmp, 'xxx')
+ path = os.path.join(self.tmp_dir, 'xxx')
self.write_file(path)
command, pyversion, filename = 'xxx', '2.6', path
dist_files = [(command, pyversion, filename)]
- self.write_file(self.rc, PYPIRC_LONG_PASSWORD)
# lets run it
pkg_dir, dist = self.create_dist(dist_files=dist_files, author=u'dédé')
cmd = upload(dist)
cmd.ensure_finalized()
+ cmd.repository = self.pypi.full_address
cmd.run()
# what did we send ?
- self.assertIn('dédé', self.last_open.req.data)
- headers = dict(self.last_open.req.headers)
- self.assert_(headers['Content-length'] > 2000)
- self.assertTrue(headers['Content-type'].startswith('multipart/form-data'))
- self.assertEquals(self.last_open.req.get_method(), 'POST')
- self.assertEquals(self.last_open.req.get_full_url(),
- 'http://pypi.python.org/pypi')
- self.assertTrue('xxx' in self.last_open.req.data)
- auth = self.last_open.req.headers['Authorization']
- self.assertFalse('\n' in auth)
+ environ, request_data = self.pypi.requests[-1]
+ self.assertIn('dédé', request_data)
+ self.assertIn('xxx', request_data)
+ self.assert_(environ['CONTENT_LENGTH'] > 2000)
+ self.assertTrue(environ['CONTENT_TYPE'].startswith('multipart/form-data'))
+ self.assertEqual(environ['REQUEST_METHOD'], 'POST')
+ self.assertNotIn('\n', environ['HTTP_AUTHORIZATION'])
def test_suite():
- return unittest2.makeSuite(uploadTestCase)
+ return unittest2.makeSuite(UploadTestCase)
if __name__ == "__main__":
unittest2.main(defaultTest="test_suite")
diff --git a/src/distutils2/tests/test_upload_docs.py b/src/distutils2/tests/test_upload_docs.py
new file mode 100644
--- /dev/null
+++ b/src/distutils2/tests/test_upload_docs.py
@@ -0,0 +1,167 @@
+"""Tests for distutils.command.upload_docs."""
+# -*- encoding: utf8 -*-
+import httplib, os, os.path, tempfile, unittest2, zipfile
+from cStringIO import StringIO
+
+from distutils2.command import upload_docs as upload_docs_mod
+from distutils2.command.upload_docs import (upload_docs, zip_dir,
+ encode_multipart)
+from distutils2.core import Distribution
+
+from distutils2.errors import DistutilsFileError
+
+from distutils2.tests import support
+from distutils2.tests.pypi_server import PyPIServer, PyPIServerTestCase
+from distutils2.tests.test_config import PyPIRCCommandTestCase
+
+
+EXPECTED_MULTIPART_OUTPUT = "\r\n".join([
+'---x',
+'Content-Disposition: form-data; name="a"',
+'',
+'b',
+'---x',
+'Content-Disposition: form-data; name="c"',
+'',
+'d',
+'---x',
+'Content-Disposition: form-data; name="e"; filename="f"',
+'',
+'g',
+'---x',
+'Content-Disposition: form-data; name="h"; filename="i"',
+'',
+'j',
+'---x--',
+'',
+])
+
+PYPIRC = """\
+[distutils]
+index-servers = server1
+
+[server1]
+repository = %s
+username = real_slim_shady
+password = long_island
+"""
+
+class UploadDocsTestCase(PyPIServerTestCase, PyPIRCCommandTestCase):
+
+ def setUp(self):
+ super(UploadDocsTestCase, self).setUp()
+ self.dist = Distribution()
+ self.dist.metadata['Name'] = "distr-name"
+ self.cmd = upload_docs(self.dist)
+
+ def test_generates_uploaddir_if_not_given(self):
+ self.assertEqual(self.cmd.upload_dir, None)
+ self.cmd.ensure_finalized()
+ self.assertEqual(self.cmd.upload_dir, os.path.join("build", "docs"))
+
+ def prepare_sample_dir(self):
+ sample_dir = tempfile.mkdtemp()
+ os.mkdir(os.path.join(sample_dir, "some_dir"))
+ self.write_file(os.path.join(sample_dir, "some_dir", "index.html"), "Ce mortel ennui")
+ self.write_file(os.path.join(sample_dir, "index.html"), "Oh la la")
+ return sample_dir
+
+ def test_zip_dir(self):
+ source_dir = self.prepare_sample_dir()
+ compressed = zip_dir(source_dir)
+
+ zip_f = zipfile.ZipFile(compressed)
+ self.assertEqual(zip_f.namelist(), ['index.html', 'some_dir/index.html'])
+
+ def test_encode_multipart(self):
+ fields = [("a", "b"), ("c", "d")]
+ files = [("e", "f", "g"), ("h", "i", "j")]
+ content_type, body = encode_multipart(fields, files, "-x")
+ self.assertEqual(content_type, "multipart/form-data; boundary=-x")
+ self.assertEqual(body, EXPECTED_MULTIPART_OUTPUT)
+
+ def prepare_command(self):
+ self.cmd.ensure_finalized()
+ self.cmd.repository = self.pypi.full_address
+ self.cmd.upload_dir = self.prepare_sample_dir()
+ self.cmd.username = "username"
+ self.cmd.password = "password"
+
+ def test_upload(self):
+ self.prepare_command()
+ self.cmd.run()
+
+ self.assertEqual(len(self.pypi.requests), 1)
+ environ, request_data = self.pypi.requests[-1]
+
+ self.assertIn("content", request_data)
+ self.assertIn("Basic", environ['HTTP_AUTHORIZATION'])
+ self.assertTrue(environ['CONTENT_TYPE'].startswith('multipart/form-data;'))
+
+ action, name, content =\
+ request_data.split("----------------GHSKFJDLGDS7543FJKLFHRE75642756743254")[1:4]
+
+ # check that we picked the right chunks
+ self.assertIn('name=":action"', action)
+ self.assertIn('name="name"', name)
+ self.assertIn('name="content"', content)
+
+ # check their contents
+ self.assertIn("doc_upload", action)
+ self.assertIn("distr-name", name)
+ self.assertIn("some_dir/index.html", content)
+ self.assertIn("Ce mortel ennui", content)
+
+ def test_https_connection(self):
+ https_called = False
+ orig_https = upload_docs_mod.httplib.HTTPSConnection
+ def https_conn_wrapper(*args):
+ https_called = True
+ return upload_docs_mod.httplib.HTTPConnection(*args) # the testing server is http
+ upload_docs_mod.httplib.HTTPSConnection = https_conn_wrapper
+ try:
+ self.prepare_command()
+ self.cmd.run()
+ self.assertFalse(https_called)
+
+ self.cmd.repository = self.cmd.repository.replace("http", "https")
+ self.cmd.run()
+ self.assertFalse(https_called)
+ finally:
+ upload_docs_mod.httplib.HTTPSConnection = orig_https
+
+ def test_handling_response(self):
+ calls = []
+ def aggr(*args):
+ calls.append(args)
+ self.pypi.default_response_status = '403 Forbidden'
+ self.prepare_command()
+ self.cmd.announce = aggr
+ self.cmd.run()
+ message, _ = calls[-1]
+ self.assertIn('Upload failed (403): Forbidden', message)
+
+ calls = []
+ self.pypi.default_response_status = '301 Moved Permanently'
+ self.pypi.default_response_headers.append(("Location", "brand_new_location"))
+ self.cmd.run()
+ message, _ = calls[-1]
+ self.assertIn('brand_new_location', message)
+
+ def test_reads_pypirc_data(self):
+ self.write_file(self.rc, PYPIRC % self.pypi.full_address)
+ self.cmd.repository = self.pypi.full_address
+ self.cmd.ensure_finalized()
+ self.assertEqual(self.cmd.username, "real_slim_shady")
+ self.assertEqual(self.cmd.password, "long_island")
+
+ def test_checks_index_html_presence(self):
+ self.prepare_command()
+ os.remove(os.path.join(self.cmd.upload_dir, "index.html"))
+ self.assertRaises(DistutilsFileError, self.cmd.run)
+
+def test_suite():
+ return unittest2.makeSuite(UploadDocsTestCase)
+
+if __name__ == "__main__":
+ unittest2.main(defaultTest="test_suite")
--
Repository URL: http://hg.python.org/distutils2
More information about the Python-checkins
mailing list