Mailman 3 October 2012 - Distutils-SIG

distlib updated - comments sought
by Vinay Sajip 05 Oct '12

05 Oct '12

I've updated distlib as follows: 1. I've added support for the equivalent of "pkg_resources.resource_filename" for returning a true filesystem path for a resource. I've added info about this in the "Distlib's design" document, though the reference API has not been fully updated - I thought I'd see if there was any feedback about the approach I've used. Code and tests have been added, though. 2. I've added the PyPI client code from distutils2 in distlib.pypi. I only tweaked it a little to run under 2.x and 3.x from a single code base, ensuring that all the tests passed. I hope to update it with better support for finding archives not hosted on PyPI, using hints. I'm also considering the following changes, which I'd like some feedback on: * Merging the distlib.depgraph package into the distlib.database package, as they are fairly interlinked (e.g. adding requires() and required_by() methods to Distribution). * Removing depgraph.main() and its test, because I don't think it belongs in the library layer, but is better implemented as a utility script. Please let me have your comments. Regards, Vinay Sajip

3 6

distlib updated with resources API
by Vinay Sajip 04 Oct '12

04 Oct '12

I've updated distlib[1] with a resources API - functionality for accessing data files co-located with code in Python packages. This is missing from the stdlib and currently people use pkg_resources to achieve this. The design and implementation allows for accessing resources from packages imported from the file system or from .zip files, and is intended to be extensible to other PEP 302-compliant import systems. The update includes: A short tutorial showing how to use the API. [2] A discussion of the design of the API. [3] Reference documentation for the API. [4] I'd be interested in any and all feedback on this - be it on design, code, docs. Please regard it as very much a first draft - it would appear to cover pretty much the same functionality as the corresponding part of pkg_resources [5]. Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/distlib/ [2] http://distlib.readthedocs.org/en/latest/tutorial.html#using-the-resource-a… [3] http://distlib.readthedocs.org/en/latest/internals.html#the-resources-api [4] http://distlib.readthedocs.org/en/latest/reference.html#the-distlib-resourc… [5] http://packages.python.org/distribute/pkg_resources.html#basic-resource-acc…

9 61

pkg_resources API use according to Nullege search engine
by Daniel Holth 02 Oct '12

02 Oct '12

I thought it would be interesting to get an idea of the relative popularity of the various pkg_resources calls in the interest of emulating a subset. So I looked it up on Nullege. The absolute popularity is certainly different from the numbers below. I think of require() and load_entry_point() used in console scripts as implementing a dynamic linker, so for example if some code crashes when the KDEWallet bindings are installed, you would still be able to run it by omitting those from sys.path. Gem certainly works this way. I'm not 100% sure pkg_resources works that way. Those functions traverse the dependency graph at runtime which is fast because it doesn't include the "search pypi" stuff. resource_filename is very popular. I would have thought resource_stream would be more popular. Unless your package is zipped resource_filename is trivial to implement. If you are writing an installer, you can unpack a single distribution to a folder, run find_distributions() on that folder, and get a Distribution() object with the dependencies as a dict. -- Daniel H pkg_resources APIs by number of Nullege Call() counts: 'require', 2284 'run_script', 0 'get_provider', 32 'get_distribution', 103 'load_entry_point', 31 'get_entry_map', 11 'get_entry_info', 4 'iter_entry_points', 291 'resource_string', 175 'resource_stream', 155 'resource_filename', 713 'resource_listdir', 71 'resource_exists', 67 'resource_isdir', 18 'declare_namespace', 643; obsoleted by Python 3.3 or pkgutil 'working_set', 55 (all samples); not a function 'add_activation_listener', 3 'find_distributions', 25; needed by installers 'set_extraction_path', 2 'cleanup_resources', 4 'get_default_cache', 2 'Environment', 51 'WorkingSet', 17 'ResourceManager', 27 'Distribution', 22; needed by installers 'Requirement.parse', 524 'EntryPoint.parse', 45 'ResolutionError', 6 samples 'VersionConflict', 13 samples 'DistributionNotFound', 41 samples 'UnknownExtra', 8 samples 'ExtractionError', 0 'parse_requirements', 32 'parse_version', 100 'safe_name', trivial regexp replace 'safe_version', "" 'get_platform', 1 'compatible_platforms', 0 'yield_lines', 15; trivial 'split_sections', 9; .ini-style [section] parser 'safe_extra', 0; another regexp replacement 'to_filename', 9; another text replacement 'ensure_directory', 40 'normalize_path', 42 # Constants that control which kinds of dists are preferred 'EGG_DIST', 'BINARY_DIST', 'SOURCE_DIST', 'CHECKOUT_DIST', 'DEVELOP_DIST', 'IMetadataProvider', 'IResourceProvider', 'FileMetadata', 'PathMetadata', 23 'EggMetadata', 2 'EmptyProvider', 'empty_provider', 'NullProvider', subclassed a few times for unit test mocks 'EggProvider', 0 'DefaultProvider', 21 mentions 'ZipProvider', 1 subclass 'register_finder', 'register_namespace_handler', 'register_loader_type', 6; one user is a code signing library 'fixup_namespace_packages', 0 'get_importer', 0 'AvailableDistributions']

4 11

Struggling with a modular buildout configuration
by Lele Gaifax 02 Oct '12

02 Oct '12

Hi all, I'm trying to setup a modular buildout configuration, but I'm hitting my head on an invisible wall... Basically, I'd like to have a top-level buildout.cfg that "collects" settings from various "subprojects"; in particular, each subproject may "augment" a set of eggs, some of which may be defined as "develop" (in buildout parlance). Finally, with a single "zc.recipe.egg" I wanna install a custom interpreter with all those eggs inside. So, I have the top-level configuration that uses "extends" to include a "projects.cfg": [buildout] extends = base.cfg projects.cfg that in turn brings in the various subprojects snippets thru "extends": [buildout] extends = src/this-project.cfg src/that-project.cfg these snippets typically contain a section like [projects] eggs += this-project-egg some-other-dependency-egg develop += src/this-project parts += this-project-specific-part [this-project-specific-part] ... The problem I'm having is that for incomprensible (to me, of course :) reason some of those settings do not "emerge" at the top level (others do!), as if a later "extends" is clobbering a previous one. I built a minimalistic example [1] that shows the unexpected behaviour; doing a "buildout annotate" I get something like the following: Annotated sections ================== [buildout] ... develop= ${projects:develop} /tmp/tb/buildout.cfg parts= ${projects:parts} /tmp/tb/buildout.cfg ... [extra-parts-from-proj2] [projects] develop= src/proj2 += /tmp/tb/src/proj2/devel.cfg eggs= extra-needed-by-proj1 proj1 proj2 += /tmp/tb/src/proj1/devel.cfg += /tmp/tb/src/proj1/base.cfg += /tmp/tb/src/proj2/base.cfg parts= extra-parts-from-proj2 += /tmp/tb/src/proj2/devel.cfg [python] eggs= ${projects:eggs} /tmp/tb/base.cfg interpreter= python /tmp/tb/base.cfg recipe= zc.recipe.egg /tmp/tb/base.cfg If you look at the [projects] section, you can see that while the "eggs" setting carries the three expected values (one for each subproject plus one extra), the "develop" setting contains just a single value, the one injected by proj2, and the one that should be coming from proj1 is missing. If I omit "src/proj2" from the extends option, then the "develop" setting correctly contains just "src/proj1"... Of course I tried hard to understand what's going, even walking into buildout 1.5.2 code with pdb. I tried also with the latest development version (the upcoming 1.6) as well as the github branch (IIUC it derivates from 1.4), but all exhibit the same behaviour. The 1.6 code is slightly different, and at first it seemed doing the right thing, but at a closer look it really moved the problem from the "develop" setting "eggs". As far as I can tell, there is something strange happening in the _update_section() function when the _open() function handles the "extends" setting: at some point it "loses" the cognition that a certain assignment is an expansion (tecnically, that the key is actually "develop +", instead of "develop"), so the final "s1.update(s2)" overrides it, instead of augmenting it... Of course I may be doing the wrong thing or missing the obvious, and if so I'll be happy if you can shed some light on me. Thanks in advance for any help and sorry for the long post, ciao, lele. [1] http://artiemestieri.tn.it/~lele/test-buildout.tar.gz -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele(a)metapensiero.it | -- Fortunato Depero, 1929.

3 6

Re: [Distutils] pkg_resources API use according to Nullege search engine
by Vinay Sajip 01 Oct '12

01 Oct '12

> > Of course it's not really a mystery; a tremendous number of Apis only accept > a file name. That's why it's there. And only something well connected to > the importer system knows how and whether the extraction is needed. > Why is that the case? If the bytes are available, then whoever needs can write them to a temp file, pass that name, and clean up afterwards. Is managing a cache of arbitrary files extracted from zips is a mainstream use case which distlib needs to support? Regards, Vinay Sajip

1 0

Re: [Distutils] [Python-Dev] distlib updated with resources API
by Vinay Sajip 01 Oct '12

01 Oct '12

Yes, and I've added its package.yaml to the original Gist at https://gist.github.com/3803556 Regards, Vinay Sajip >________________________________ > From: Daniel Holth <dholth(a)gmail.com> >To: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> >Sent: Monday, 1 October 2012, 15:27 >Subject: Re: [Distutils] [Python-Dev] distlib updated with resources API > >Have you read the IPython setup.py? >https://github.com/ipython/ipython/blob/master/setup.py > > >

1 0

Differences in PEP386 and setuptools
by Donald Stufft 01 Oct '12

01 Oct '12

I've been comparing how PEP386/distutils2.version treats versions and how pkg_resources from setuptools treats versions and it confirmed a worry of me with the way that dev is treated in PEP386. In PEP386 you have several kinds of "top level" releases, these are alpha, beta, candidate, and final and they are sorted as you would expect. On top of that it has pre and post releases for any of those top level releases. The pre releases are tagged with a ".dev" and post releases are tagged with a ".post". They are sorted immediately before/after the "main" release that they are pre/post for. In pkg_resources dev is treated as it's own "main" release and sorts it in front of an alpha. This means that given the versions: ["0.1.dev1", "0.1a1", "0.1b1", "0.1c1", "0.1"] PEP386 will sort them as: ["0.1a1", "0.1b1", "0.1c1", "0.1.dev1", "0.1"] and pkg_resources will sort them as: ["0.1.dev1", "0.1a1", "0.1b1", "0.1c1", "0.1"] To further complicate things the most common usage I've personally seen in the wild is that of "0.1dev" or "0.1dev1" which the author expects to sort before an alpha (In this case distutils2.version throws an error, but the suggest function is able to turn it into 0.1.dev1). I think this difference is going to cause confusion, especially during the the transition period when you're going to have people using both pkg_resources and the new PEP386 functions. Since PEP386 is only in the "Accepted" stage, and isn't part of the official implementation yet, is it at all possible to revise it? Ideally I think to follow with the prior art, and people's expectations "dev" should be moved to a "main" release type sorted before an alpha, and to take it's place as a pre release modifier perhaps something like "pre" can be used instead (e.g. "0.1.pre1").

9 24

ANN wheel-0.10.1
by Daniel Holth 01 Oct '12

01 Oct '12

I'm pleased to announce wheel version 0.10.1. In this version of wheel, Paul Moore has done a lot of work on wheel's reference installer, a simple installer that can only install from local wheel files. The wheel installer can now accept pip-style requirements files but it doesn't automatically install dependencies, and it does not uninstall conflicting packages i.e. previous versions of the same package. If you need a mature full-featured installer you should use pip. A notable feature of this version of wheel is that you can use it even when your virtualenv does not contain setuptools or distribute. For example, /tmp/venv/bin/python wheel-0.10.1-py2.py3-none-any.whl/wheel install wheel-0.10.1-py2.py3-none-any.whl executes wheel's wheel to install itself. This feature should be handy for bootstrapping. Wheel's own scripts still use pkg_resources, so you should invoke the tool with "python -m wheel" if you do not want to install distribute or setuptools. On my machine, it takes about 0.7 seconds to install IPython from a wheel.

1 0