Mailman 3 October 2012 - Distutils-SIG

Distribute 0.6.29 released
by Jason R. Coombs 20 Oct '12

20 Oct '12

I'm pleased to announce the release of Distribute 0.6.29. It's been far too long since the last release. This release brings a great deal of improvement <https://bitbucket.org/tarek/distribute/src/216ed5285c66202fe87fdf8c3be8ed01 c3df6956/CHANGES.txt?at=default> , particularly around Python 3.3. Many thanks to all the people who contributed to this release. Of course, if you encounter issues, please report them at the project site <https://bitbucket.org/tarek/distribute/> . Regards, Jason R. Coombs

1 0

Re: [Distutils] [Python-Dev] accept the wheel PEPs 425, 426, 427
by Daniel Holth 18 Oct '12

18 Oct '12

On Thu, Oct 18, 2012 at 3:10 PM, Antoine Pitrou <solipsis(a)pitrou.net> wrote: > On Thu, 18 Oct 2012 14:35:19 -0400 > Benjamin Peterson <benjamin(a)python.org> wrote: >> 2012/10/18 Daniel Holth <dholth(a)gmail.com>: >> > Let me know what I need to do to get it accepted, if anything needs to >> > be added or revised, or, preferably, that it is awesome and you want >> > to use it ASAP. >> >> Traditionally, you send the peps to python-dev, so people can bikeshed inline. > > Or at least send the URLs, it's helpful. The texts seemed a bit long for direct inclusion. http://www.python.org/dev/peps/pep-0425/ http://www.python.org/dev/peps/pep-0426/ http://www.python.org/dev/peps/pep-0427/

2 1

[issue146] exclude doesn't for for find_packages
by ranamalo 18 Oct '12

18 Oct '12

New submission from ranamalo: find_packages(exclude=('settings')) for examples does not exclude the file when making an egg: python setup.py bdist_egg Am I using it wrong? ---------- messages: 696 nosy: ranamalo priority: bug status: unread title: exclude doesn't for for find_packages _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue146> _______________________________________________

1 0

Re: [Distutils] distlib updated with "locators" API
by Vinay Sajip 18 Oct '12

18 Oct '12

Daniel Holth <dholth <at> gmail.com> writes: > When you are using setuptools, setup.py can be used to generate > declarative metadata with the command "python setup.py egg_info". You > can add --egg-base <dir> to put the static metadata in a different > directory. Setuptools even makes it easy to add new metadata files to > the .egg-info folder by using a plugin to write each file. The > majority of the sdists on pypi contain pre-built versions of this > static metadata in .egg-info folders. Can it be correctly said that all of the metadata passed to setup() is in .egg-info? How far can we rely on the accuracy of the content of .egg-info dirs in sdists on PyPI? I have found hundreds of projects where even basic metadata is missing or incorrect - for example, the version. > Unfortunately setup.py is used to build not one, but many different > dists (the most common difference of course being different > requirements) depending on any number of factors, so pip rebuilds the > static metadata before using it. Metadata 1.2+ of course allow us to > put the most common "reasons for different dependencies" conditions in > the static metadata itself. Unfortunately, that doesn't help with nearly all the stuff currently on PyPI. Regards, Vinay Sajip

7 9

Simple API for location of distributions and requirements on PyPI
by Vinay Sajip 18 Oct '12

18 Oct '12

>>> This bit of text prevents Gmane from thinking that I'm top-posting :-( I've added a convenience function to distlib.locators called locate(), to which you pass a requirement. The returned Distribution object allows you to query its requirements. Example of usage: >>> from distlib.locators import locate >>> from pprint import pprint >>> flask = locate('flask') >>> flask <Distribution Flask (0.9) [http://pypi.python.org/packages/source/F/Flask/Flask-0.9.tar.gz]> >>> flask.get_requirements('install') ['Werkzeug (>= 0.7)', 'Jinja2 (>= 2.4)'] >>> reqs = flask.get_requirements('install') >>> werkzeug, jinja2 = [locate(r) for r in reqs] >>> werkzeug <Distribution Werkzeug (0.8.3) [http://pypi.python.org/packages/source/W/Werkzeug/Werkzeug-0.8.3.tar.gz]> >>> jinja2 <Distribution Jinja2 (2.6) [http://pypi.python.org/packages/source/J/Jinja2/Jinja2-2.6.tar.gz]> >>> authy = locate('authy') >>> authy.get_requirements('install') ['httplib2 (>= 0.7, < 0.8)', 'simplejson'] >>> reqs = authy.get_requirements('install') >>> rds = [locate(r) for r in reqs] >>> pprint(rds) [<Distribution httplib2 (0.7.6) [http://httplib2.googlecode.com/files/httplib2-0.7.6.tar.gz]>, <Distribution simplejson (2.6.2) [http://pypi.python.org/packages/source/s/simplejson/simplejson-2.6.2.tar.gz]>] >>> Feedback welcome. Regards, Vinay Sajip

2 1

distlib updated with "locators" API
by Vinay Sajip 15 Oct '12

15 Oct '12

I've updated distlib with an experimental API for locating distributions. A locator is just something whose "get_project" method you invoke with a "project" name such as "flask" or "sqlalchemy", and it returns a dictionary whose keys are versions of the project which were located, and the corresponding value is a tuple whose first element is a Metadata object and whose second element is a list of dicts pointing to archives for that version of the project. I've implemented three locators with some basic tests and docs: * DirectoryLocator - looks in a file system directory tree for archives * PyPIRPCLocator - uses PyPI's XML-RPC interface to locate archives * SimpleScrapingLocator - uses PyPI's "simple" web scraping interface to locate archives - similar to pip's PackageFinder A very basic script showing how to use it is at https://gist.github.com/3886402 There's plenty more work to do - if there's mileage in this idea, instead of returning Metadata and lists of dicts, Distribution instances can be returned which can be queried for dependencies in a more natural way. At the moment, this (dependency querying) isn't fully doable because all the metadata is not declarative. However, the work I've been doing with the package.yaml metadata format indicates that setup()-kwarg metadata can be automatically pulled out of setup.py for many (most) of the projects on PyPI. Feedback welcome. Regards, Vinay Sajip

2 1

distlib updated to include "entry point" functionality
by Vinay Sajip 10 Oct '12

10 Oct '12

I've now updated distlib to include equivalent functionality for pkg_resources' entry points. There is no ability to install packages on-demand via "require()" functionality, but distributions are able to register things they implement, and the capability to query what's available is also present. The relevant information is stored in a dist-info file REGISTRY which has the same format as entry_points.txt files. There is no code at present to write these files; that will come later when focusing on what installer software will need from distlib. The distlib.depgraph code has now been merged into distlib.database, as indicated in an earlier post. I've also added a section on migrating from older APIs. This currently covers pkg_resources' resource and entry point APIs. The resources API now covers extraction of files from .zip files and caching of the extracted files in the file system. I noticed that pkg_resources has a function cleanup_resources() to clear the cache, but this is a no-op. Can someone tell me why this is? Regards, Vinay Sajip

5 16

Re: [Distutils] distlib updated - comments sought
by Paul Moore 09 Oct '12

09 Oct '12

On 5 October 2012 10:27, Vinay Sajip <vinay_sajip(a)yahoo.co.uk> wrote: >> +1 on this. Can you share a little more on your hint idea? I have a > >> specific use case in mind, and would like to know if it's covered. >> It's an intranet webpage that hosts packages in a very odd format, >> unfortunately, so I need to write code to get the packages, just >> providing URLs isn't possible. > > For example, setuptools uses "dependency_links" which indicate URLs which lead to downloadable packages. I haven't thought it through in detail, but my package.yaml format preserves those links. I would aim to design in such a way that custom schemes aren't hard to support. One more area I need to look at is pip's PackageFinder, which is the minimum standard we need to support. [Sorry, I hadn't meant to go off-list - blame gmail...] Right, that's not going to be enough for me. I have a web page where files are hosted at http://my.domain/datastore/VARYING_ENCODED_ID/filename.tar.gz. The problem is that VARYING_ENCODED_ID changes regularly. It's possible to calculate it in code, but only based on the current date and the package name :-( It really isn't worth asking why, but I'm stuck with it. I'd envisage a design consisting of independent "Locators", one for a local directory, one for PyPI using XMLRPC, one for PyPI's "simple" webpage interface (IMO, we need both PyPI types, as the simple interface generates a lot of useless web scraping - look at the simple page for lxml, for example...). Users could add locators for any custom stores as needed. I wrote a 30-minute proof of concept, which is at https://bitbucket.org/pmoore/tools/src/67b33c15efad/bin/Locator.py if you're interested. Plan B for something like this would be to use one of the "run a local PyPI server" packages in existence, and write a front-end to the web store that I can run locally, and point to as an extra PyPI index. That may well be a more general solution, but I don't think any of the existing PyPI server packages supports this type of usage, so it would need some work. Paul.

5 21

Re: [Distutils] distlib updated - comments sought
by Stanley A. Klein 08 Oct '12

08 Oct '12

The issue I had raised was attention to the needs of Linux packaging and the Filesystem Hierarchy Standard, and apparently that is under consideration. The response I received (from "Daniel Holth" <dholth(a)gmail.com>) said in part "The FHS issues are on the list, which is why there are "resource categories" that can be installed wherever in new-packaging land." I assume that by an "installer" you mean the combination of package building (done by e.g., rpmbuild) and installation (done by e.g., rpm or yum). The mapping of files from the Pypi structure to the target directories is done at package-build time and not usually at install-time. What a function like bdist_rpm does is to create an rpm spec and turn the package building function over to rpmbuild. The file mapping is defined in the rpm spec and takes place during the rpmbuild processing. What I didn't notice in the distlib documentation were low level functions that would facilitate the allocation of Pypi files/directories to target files/directories to help in preparation of the rpm spec. Stan Klein On Fri, October 5, 2012 1:24 pm, Vinay Sajip <vinay_sajip(a)yahoo.co.uk> wrote: > Message: 3 > Date: Fri, 5 Oct 2012 15:49:04 +0000 (UTC) > From: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> > To: Distutils-Sig(a)Python.Org > Subject: Re: [Distutils] distlib updated - comments sought > > Stanley A. Klein <sklein <at> cpcug.org> writes: > >> I looked at the documentation and can't completely follow what you are >> doing, although I was told in another email that the issue I raised was >> being addressed in pkg_resources. > > The goal of distlib is to provide a library of low-level functions > relating > to packaging and distribution of Python software, and it does not contain > any > installers (though it will contain low-level code which could be used by > an > installer). The distlib code aims to serve as an implementation of the > various > packaging PEPs, such that if higher level tools use distlib, they should > be > able to both save writing code (and thereby gain time) as well as achieve > better interoperability with other packaging tools. > > The goal of distlib is not to be a complete packaging solution (which > packaging > tried to be for Python 3.3, but it proved to be too ambitious a goal). > > Which issue are you referring to, which you were told was addressed by > pkg_resources? From the section starting "Here is what I need", I couldn't > see how pkg_resources could fulfil those needs by itself. > > >> Here is what I need: > [snip] >> I can't completely see how I would do this using what is described in >> the >> distlib documentation. > > What you need is an installer, and distlib doesn't contain functionality > to do > installation. I won't rehash the various discussions around the needs of > Linux > distro packagers, except to say that the PEPs have taken into account the > needs > of distro packagers and the ability to map various parts of a distribution > to > various target directories in a distro-specific manner. > > Regards, > > Vinay Sajip >

4 4

Re: [Distutils] distlib updated - comments sought
by Stanley A. Klein 05 Oct '12

05 Oct '12

I looked at the documentation and can't completely follow what you are doing, although I was told in another email that the issue I raised was being addressed in pkg_resources. Here is what I need: I'm trying to build an rpm package from a Pypi tar.gz. The tar file contains several files and subdirectories. I need to map them as follows: a. Send the package modules to /usr/lib/pythonx.x/site-packages/name_of_pkg b. Send the executable command scripts to /usr/bin or /usr/sbin as appropriate c. Send any readme files, license files, doc directories, example directories, test directories, and other such files/directories to /usr/share/doc/name_of_pkg d. Generally, send the various files to places prescribed in the Filesystem Hierarchy Standard for Unix/Linux. I can't completely see how I would do this using what is described in the distlib documentation. Stan Klein On Fri, October 5, 2012 6:00 am, Vinay Sajip <vinay_sajip(a)yahoo.co.uk> wrote: > Date: Thu, 4 Oct 2012 22:28:01 +0000 (UTC) > From: Vinay Sajip <vinay_sajip(a)yahoo.co.uk> > To: Distutils-Sig(a)Python.Org > Subject: [Distutils] distlib updated - comments sought > > I've updated distlib as follows: > > 1. I've added support for the equivalent of > "pkg_resources.resource_filename" > for returning a true filesystem path for a resource. I've added info > about > this in the "Distlib's design" document, though the reference API has > not > been fully updated - I thought I'd see if there was any feedback about > the > approach I've used. Code and tests have been added, though. > > 2. I've added the PyPI client code from distutils2 in distlib.pypi. I only > tweaked it a little to run under 2.x and 3.x from a single code base, > ensuring that all the tests passed. I hope to update it with better > support > for finding archives not hosted on PyPI, using hints. > > I'm also considering the following changes, which I'd like some feedback > on: > > * Merging the distlib.depgraph package into the distlib.database package, > as > they are fairly interlinked (e.g. adding requires() and required_by() > methods > to Distribution). > > * Removing depgraph.main() and its test, because I don't think it belongs > in the > library layer, but is better implemented as a utility script. > > Please let me have your comments. > > Regards, > > Vinay Sajip >

2 1