Mailman 3 October 2012 - Distutils-SIG

distlib - experience using the library
by Paul Moore 29 Oct '12

29 Oct '12

In order to get a feel for distlib, I spent a short time today writing a quick script to solve a problem I currently have. The results will hopefully be of some use in pointing out some "real world" experience with the library. The problem is as follows: I maintain a small library of sdists downloaded from PyPI, for offline use and to speed things up when I'm building virtualenvs. A maintenance issue with this approach is keeping the library reasonably up to date. It's not a huge issue, so I've never invested much time in writing anything to solve it, but it seemed a reasonable task for distlib. First of all, the positives: 1. It really was "a short time" I spent. Less time than it took me to write this mail. The locator API is beautifully easy to use, and fits this use case perfectly. 2. Although the resulting code is a quick hack, it's clean and readable. 3. Extending the code looks very straightforward. I haven't tried yet, but adding extra indexes, adding a download capability, all look simple. Areas where I found things lacking: I'd really like to have a means of getting all distributions from a locator. Something like the distlib.locators.get_all_distribution_names function, but on a locator. It would only be available on certain locators (specifically, the XMLRPC and directory ones) but for my script, it would have saved writing a directory scanning function and duplicating the filename parsing from the locator library. The dance to create a Distribution object from some data md = Metadata() md['Name'] = p md['Version'] = v dist = Distribution(md) cries out for a convenience method. On the other hand, I'm not sure how much advantage there is in using Distribution objects over just passing round the raw name and version strings for code as simple as mine. The version API seems very complex. I was expecting to just do something like Version(string) to get a sortable version object. Given that some of my packages use deliberately non-standard custom versions (2.3.x20121023) that's probably insufficient, but for this application I'm not really interested in validating versions. Using distutils.version.LooseVersion actually works better for me because it "just works". Is there some way to get the forgiving convenience of if distutils.version.LooseVersion(v1) > distutils.version.LooseVersion(v2): # whatever with distlib? But regardless, these are minor quibbles. The overall experience was very positive. Paul.

2 2

Re: [Distutils] distlib locator compares reasonably well with pip's PackageFinder
by Daniel Holth 28 Oct '12

28 Oct '12

Just call it a dist. Try not to overload the name quite as much as egg. On Oct 28, 2012 12:46 PM, "PJ Eby" <pje(a)telecommunity.com> wrote:

1 0

distlib locator compares reasonably well with pip's PackageFinder
by Vinay Sajip 28 Oct '12

28 Oct '12

As an extended test of the locator functionality in distlib, I compared the results from the locate() API with the results from pip's PackageFinder. The results seem encouraging: out of 24938 packages tested, only 510 (2%) gave different results from pip. Out of the 510, many are due to: 1. distlib (currently) looks for archives which are machine-independent, and skips archives which contain machine-dependent strings in their names. 2. distlib (currently) skips archives automatically generated by sites like GitHub and BitBucket, since the download archive doesn't always contain version information in the name of the download. Examples: http://github.com/user/project/tarball/master http://bitbucket.org/user/project/get/tip.zip In a (much) smaller number of cases, the differences are because locate() found an archive where pip didn't. But for 98% of the projects registered on PyPI, locate() returns identical results to pip's PackageFinder.find_requirement, and does it slightly faster in most cases :-) The comparison script, along with the list of differences, is at https://gist.github.com/3951634 Regards, Vinay Sajip

2 5

[ANN] pypiserver 1.0.0 - minimal private pypi server
by Ralf Schmitt 26 Oct '12

26 Oct '12

Hi, I've just uploaded pypiserver 1.0.0 to the python package index. pypiserver is a minimal PyPI compatible server. It can be used to serve a set of packages and eggs to easy_install or pip. pypiserver is easy to install (i.e. just 'pip install pypiserver'). It doesn't have any external dependencies. http://pypi.python.org/pypi/pypiserver/ should contain enough information to easily get you started running your own PyPI server in a few minutes. The code is available on github: https://github.com/schmir/pypiserver Changes in version 1.0.0 ------------------------ - add passlib and waitress to pypi-server-standalone - upgrade bottle to 0.11.3 - Update scripts/opensuse/pypiserver.init - Refuse to re upload existing file - Add 'console_scripts' section to 'entry_points', so 'pypi-server.exe' will be created on Windows. - paste_app_factory now use the the password_file option to create the app. Without this the package upload was not working. - Add --fallback-url argument to pypi-server script to make it configurable. -- Cheers Ralf

2 1

__build__ as a temp build directory for setup.py
by anatoly techtonik 25 Oct '12

25 Oct '12

What do you think about this? http://bugs.python.org/issue16299

4 5

Distribute + Sphinx Fail in Python 3.2/3.3 on Windows
by anatoly techtonik 22 Oct '12

22 Oct '12

I can't install Sphinx with pristine installation of Python 3.2 and Python 3.3 on Windows and it looks like there some bugs in distribute_setup.py and Distribute versions. Can anybody take a look? I think it's rather critical. https://bitbucket.org/birkenfeld/sphinx/issue/1022/doesnt-install-with-pyth… If it is bug in Distribute, maybe distribute_setup.py should be patched to check for known issues for default version specified inside and upgrade itself instead of giving users bad times? -- anatoly t.

1 0

distribute buildbot
by Adam GROSZER 22 Oct '12

22 Oct '12

Hello, Are there any buildbots for distribute? -- Best regards, Adam GROSZER -- Quote of the day: Men do not reject the Bible because it contradicts itself but because it contradicts them. - The Defender

5 4

semver (again)
by Daniel Holth 21 Oct '12

21 Oct '12

I like semver. My favorite part is point 12: Precedence MUST be calculated by separating the version into major, minor, patch, pre-release, and build identifiers in that order. Major, minor, and patch versions are always compared numerically. Pre-release and build version precedence MUST be determined by comparing each dot separated identifier as follows: identifiers consisting of only digits are compared numerically and identifiers with letters or dashes are compared lexically in ASCII sort order. Numeric identifiers always have lower precedence than non-numeric identifiers. Example: 1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0-rc.1+build.1 < 1.0.0 < 1.0.0+0.3.7 < 1.3.7+build < 1.3.7+build.2.b8f12d7 < 1.3.7+build.11.e0f985a. (No non-lexicographic comparisons). Unlike PEP-386 it would also allow DVCS revision ids to be included in the version string. Taking a look at pypi, ~1k of the latest releases do not parse as semver (if you extend it by making 1.0 equivalent to 1.0.0, as all semver versions have at least (major, minor, patch)). This is similar to the number of latest package versions that don't parse as PEP-386. I also found this blog post. http://blog.ziade.org/2009/07/03/dropping-pep-386-versions-comparison/ How much would it cost to endorse semver? Daniel H

4 6

Re: [Distutils] semver (again)
by Vinay Sajip 21 Oct '12

21 Oct '12

Did you mean to reply off-list? I assume it was Gmail again :-) > From: Daniel Holth <dholth(a)gmail.com> >To: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> >Sent: Sunday, 21 October 2012, 14:33 >Subject: Re: [Distutils] semver (again) > >Of course concessions must be made to produce a practical system. Does your semver allow 1.0 in addition to 1.0.0? The SemanticVersion in distlib is currently strict and doesn't allow x.y, but the test script which produced those results did a simplistic check for matching x.y and tacked on a .0 in that case, before deciding on a pass/fail for semver. >Really what we hope to gain is a good way to order pre and post releases having the same major-minor-patch. The "legacy" scheme (setuptools/distribute) will do that, won't it? (For some value of "good", of course.) Of course it has drawbacks, as mentioned in PEP 386: it doesn't constrain version numbers to follow a sensible scheme. Of course, to enforce any additional constraints would need PyPI metadata checks to be added, which would no doubt annoy the minority of people who name their versions like "02 Jun 2010", "pre-release", "0.1.5dev-20111031", or "0.5.0-protobuf-2.4". Regards, Vinay Sajip

1 0

Re: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427
by Daniel Holth 21 Oct '12

21 Oct '12

I'm really happy with moving the troublesome Description: field to the payload. Email Policy() is useful too. http://hg.python.org/peps/rev/74868fe8ba17 Metadata 1.3 is a ... +format with no maximum line length, followed by a blank line and an +arbitrary payload. It is parseable by the ``email`` module with an +appropriate ``email.policy.Policy()``. description is deprecated... +Since Metadata 1.3 the recommended place for the description is in the +payload section of the document, after the last header. The description +needs no special formatting when included in the payload. includes a handy... # Metadata 1.3 demo from email.generator import Generator from email import header from email.parser import Parser from email.policy import Compat32 from email.utils import _has_surrogates class MetadataPolicy(Compat32): max_line_length = 0 continuation_whitespace = '\t' def _sanitize_header(self, name, value): if not isinstance(value, str): return value if _has_surrogates(value): raise NotImplementedError() else: return value def _fold(self, name, value, sanitize): body = ((self.linesep+self.continuation_whitespace) .join(value.splitlines())) return ''.join((name, ': ', body, self.linesep)) if __name__ == "__main__": import sys import textwrap pkg_info = """\ Metadata-Version: 1.3 Name: package Version: 0.1.0 Summary: A package. Description: Description =========== A description of the package. """ m = Parser(policy=MetadataPolicy()).parsestr(pkg_info) m['License'] = 'GPL' description = m['Description'] description_lines = description.splitlines() m.set_payload(description_lines[0] + '\n' + textwrap.dedent('\n'.join(description_lines[1:])) + '\n') del m['Description'] # Correct if sys.stdout.encoding == 'UTF-8': Generator(sys.stdout, maxheaderlen=0).flatten(m)

1 0