Mailman 3 June 2008 - Distutils-SIG

zc.buildout 1.1?
by Hanno Schlichting 30 Jun '08

30 Jun '08

Hi buildout-people, I have noticed us slacking a bit on the proper release management side of buildout as of late. It seems we have chosen to use the 1.0.x releases for introducing new features which has led to at least one completely broken release already. How do people feel about declaring the 1.0.x line as stable and maintenance only and do new feature work in a 1.1 release instead? Thanks, Hanno

2 1

patches for stdeb: autodetect Debian dependencies
by zooko 30 Jun '08

30 Jun '08

Folks: Thanks for stdeb! I'm using stdeb to produce debian packages for a few Python projects -- zfec [1], pycryptopp [2], pyutil [3], and argparse [4]. It works nicely. I intend to use it in the future to produce debian packages for allmydata.org Tahoe, the Least-Authority Filesystem [5]. Along the way I added this feature to stdeb: use apt-file to discover which debian package(s) provide $DISTNAME.egg-info of the required version number, and automatically produce debian metadata showing that the new debian package depends on those debian packages. For this to work requires that you have the "apt-file" command available and that you have run "apt-file update" in order to acquire the apt- file database. Please see the patch for details. With this patch, there is no "manual intervention" required to produce a good Debian package from a good Python package (for these five projects that I'm using it on). I just run stdeb then run the debian build-package command and then I'm done. We are going to script our buildbot to automatically do this whenever a new patch is committed to revision control and the unit tests pass. This is really cool! Stdeb is almost mature enough that every well- packaged Python package can automatically be converted into a well- packaged Debian package! I also ported stdeb to Python 2.4, set zip_ok=False so that it could install on Ubuntu dapper, and made one tiny clean-up that was suggested by lintian. The patch is appended. Thanks! Regards, Zooko [1] http://allmydata.org/trac/zfec [2] http://allmydata.org/trac/pycryptopp [3] http://allmydata.org/trac/pyutil [4] http://argparse.python-hosting.com/ [5] http://allmydata.org Fri Jun 27 11:38:23 MST 2008 zooko(a)zooko.com * remove print statements used for debugging Thu Jun 12 10:41:50 MST 2008 zooko(a)zooko.com * include an implementation of check_call so that this will work with Python 2.4 Thu Jun 12 10:40:36 MST 2008 zooko(a)zooko.com * setup.cfg: zip_ok = False Zipping your eggs causes various problems. I have seen about four or five such problems. I just now added one to the list -- installing stdeb with "./setup.py install" when there is already a version of stdeb installed fails on Ubuntu dapper (setuptool s-0.6a9) unless you set zip_ok = False. Tue May 27 11:16:05 MST 2008 zooko(a)zooko.com * automatically produce Debian "Depends:" metadata from setuptools "install_requires" metadata Wed May 21 15:49:10 MST 2008 zooko(a)zooko.com * don't build-depend on "-1" of python-setuptools lintian says that it is a bad idea to depend on "-1" versions. Thu May 15 16:03:04 MST 2008 zooko(a)zooko.com * more details in exception message diff -u -r --exclude=_darcs dw/setup.cfg autodeps/setup.cfg --- dw/setup.cfg 2008-05-15 15:31:42.000000000 -0700 +++ autodeps/setup.cfg 2008-06-27 11:35:40.000000000 -0700 @@ -1,3 +1,6 @@ [egg_info] tag_build = .dev tag_svn_revision = 1 + +[easy_install] +zip_ok = False diff -u -r --exclude=_darcs dw/setup.py autodeps/setup.py --- dw/setup.py 2008-05-15 15:32:13.000000000 -0700 +++ autodeps/setup.py 2008-06-27 11:39:26.000000000 -0700 @@ -1,5 +1,3 @@ -#!/usr/bin/env python - import setuptools from setuptools import setup diff -u -r --exclude=_darcs dw/stdeb/command/sdist_dsc.py autodeps/ stdeb/command/sdist_dsc.py --- dw/stdeb/command/sdist_dsc.py 2008-05-21 06:33:07.000000000 -0700 +++ autodeps/stdeb/command/sdist_dsc.py 2008-06-27 11:35:40.000000000 -0700 @@ -80,6 +80,10 @@ if self.extra_cfg_file is not None: cfg_files.append(self.extra_cfg_file) + try: + install_requires = open(os.path.join (egg_info_dirname,'requires.txt'),'rU').read() + except EnvironmentError: + install_requires = () debinfo = DebianInfo( cfg_files=cfg_files, module_name = module_name, @@ -93,6 +97,8 @@ long_description = self.distribution.get_long_description(), patch_file = self.patch_file, patch_level = self.patch_level, + install_requires = install_requires, + setup_requires = (), # XXX How do we get the setup_requires? ) if debinfo.patch_file != '' and self.patch_already_applied: raise RuntimeError('A patch was already applied, but another ' diff -u -r --exclude=_darcs dw/stdeb/util.py autodeps/stdeb/util.py --- dw/stdeb/util.py 2008-05-21 06:33:07.000000000 -0700 +++ autodeps/stdeb/util.py 2008-06-27 11:39:00.000000000 -0700 @@ -1,11 +1,12 @@ # # This module contains most of the code of stdeb. # -import sys, os, shutil, sets, select +import re, sys, os, shutil, sets, select import ConfigParser import subprocess import tempfile import stdeb +import pkg_resources from stdeb import log, __version__ as __stdeb_version__ __all__ = ['DebianInfo','build_dsc','expand_tarball','expand_zip', @@ -13,6 +14,15 @@ 'apply_patch','repack_tarball_with_debianized_dirname', 'expand_sdist_file'] +import exceptions +class CalledProcessError(exceptions.Exception): pass + +def check_call(*popenargs, **kwargs): + retcode = subprocess.call(*popenargs, **kwargs) + if retcode == 0: + return + raise CalledProcessError(retcode) + stdeb_cmdline_opts = [ ('dist-dir=', 'd', "directory to put final built distributions in (default='deb_dist')"), @@ -48,7 +58,7 @@ def process_command(args, cwd=None): if not isinstance(args, (list, tuple)): raise RuntimeError, "args passed must be in a list" - subprocess.check_call(args, cwd=cwd) + check_call(args, cwd=cwd) def recursive_hardlink(src,dst): dst = os.path.abspath(dst) @@ -111,6 +121,80 @@ result = cmd.stdout.read().strip() return result +def get_deb_depends_from_setuptools_requires(requirements): + depends = [] # This will be the return value from this function. + + requirements = list(pkg_resources.parse_requirements(requirements)) + if not requirements: + return depends + + # Ask apt-file for any packages which have a .egg-info file by these names. + # Note that apt-file appears to think that some packages e.g. setuptools itself have "foo.egg-info/BLAH" files but not a "foo.egg- info" directory. + + egginfore="((%s)(?:-[^/]+)?(?:-py[0-9]\.[0-9.]+)?\.egg-info)" % '|'.join(req.project_name for req in requirements) + + args = ["apt-file", "search", "--ignore-case", "--regexp", egginfore] + try: + cmd = subprocess.Popen(args, stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True) + except Exception, le: + log.error('ERROR running: %s', ' '.join(args)) + raise RuntimeError('exception %s from subprocess %s' % (le,args)) + returncode = cmd.wait() + if returncode: + log.error('ERROR running: %s', ' '.join(args)) + raise RuntimeError('returncode %d from subprocess %s' % (returncode, args)) + + inlines = cmd.stdout.readlines() + + dd = {} # {pydistname: {pydist: set(debpackagename)}} + E=re.compile(egginfore, re.I) + D=re.compile("^([^:]*):", re.I) + eggsndebs = set() + for l in inlines: + if l: + emo = E.search(l) + assert emo, l + dmo = D.search(l) + assert dmo, l + eggsndebs.add((emo.group(1), dmo.group(1))) + + for (egginfo, debname) in eggsndebs: + pydist = pkg_resources.Distribution.from_filename(egginfo) + try: + dd.setdefault(pydist.project_name.lower(), {}).setdefault (pydist, set()).add(debname) + except ValueError, le: + log.warn("I got an error parsing a .egg-info file named \"%s\" from Debian package \"%s\" as a pkg_resources Distribution: % s" % (egginfo, debname, le,)) + pass + + # Now for each requirement, see if a Debian package satisfies it. + ops = {'<':'<<','>':'>>','==':'=','<=':'<=','>=':'>='} + for req in requirements: + reqname = req.project_name.lower() + gooddebs = set() + for pydist, debs in dd.get(reqname, {}).iteritems(): + if pydist in req: + # log.info("I found Debian packages \"%s\" which provides Python package \"%s\", version \"%s\", which satisfies our version requirements: \"%s\"" % (', '.join(debs), req.project_name, ver, req)) + gooddebs |= (debs) + else: + log.info("I found Debian packages \"%s\" which provides Python package \"%s\", version \"%s\", which does not satisfy our version requirements: \"%s\" -- ignoring." % (', '.join (debs), req.project_name, ver, req)) + if not gooddebs: + log.warn("I found no Debian package which provides the required Python package \"%s\" with version requirements \"%s\". Guessing blindly that the name \"python-%s\" will be it, and that the Python package version number requirements will apply to the Debian package." % (req.project_name, req.specs, reqname)) + gooddebs.add("python-" + reqname) + elif len(gooddebs) == 1: + log.info("I found a Debian package which provides the require Python package. Python package: \"%s\", Debian package: \"%s \"; adding Depends specifications for the following version(s): \"%s \"" % (req.project_name, tuple(gooddebs)[0], req.specs)) + else: + log.warn("I found multiple Debian packages which provide the Python distribution required. I'm listing them all as alternates. Candidate debs which claim to provide the Python package \"%s\" are: \"%s\"" % (req.project_name, ', '.join(gooddebs),)) + + alts = [] + for deb in gooddebs: + for spec in req.specs: + # Here we blithely assume that the Debian package versions are enough like the Python package versions that the requirement can be ported straight over... + alts.append("%s (%s %s)" % (deb, ops[spec[0]], spec [1])) + + depends.append(' | '.join(alts)) + + return depends + def make_tarball(tarball_fname,directory,cwd=None): "create a tarball from a directory" if tarball_fname.endswith('.gz'): opts = 'czf' @@ -278,6 +362,8 @@ long_description=NotGiven, patch_file=None, patch_level=None, + install_requires=None, + setup_requires=None, ): if cfg_files is NotGiven: raise ValueError("cfg_files must be supplied") if module_name is NotGiven: raise ValueError("module_name must be supplied") @@ -337,7 +423,9 @@ self.pycentral_showversions=current - build_deps = ['python-setuptools (>= 0.6b3-1)'] + build_deps = ['python-setuptools (>= 0.6b3)'] + build_deps.extend(get_deb_depends_from_setuptools_requires (setup_requires)) + depends = [] depends.append('${python:Depends}') @@ -386,6 +474,7 @@ self.copy_files_lines += '\n\tcp %s %s'% (mime_desktop_file,dest_file) depends.extend(parse_vals(cfg,module_name,'Depends') ) + depends.extend(get_deb_depends_from_setuptools_requires (install_requires)) self.depends = ', '.join(depends) self.description = description

2 1

bug in setuptools 0.6c8/undefined log in entries_finder()/sdist.py
by Andreas Jung 29 Jun '08

29 Jun '08

sdist.py contains this: def entries_finder(dirname, filename): f = open(filename,'rU') data = f.read() f.close() if data.startswith('8'): # subversion 1.4 for record in map(str.splitlines, data.split('\n\x0c\n')[1:]): if not record or len(record)>=6 and record[5]=="delete": continue # skip deleted yield joinpath(dirname, record[0]) elif data.startswith('<?xml'): for match in entries_pattern.finditer(data): yield joinpath(dirname,unescape(match.group(1))) else: log.warn("unrecognized .svn/entries format in %s", dirname) However 'log' is undefined. Andreas -- ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: info(a)zopyx.com - Phone +49 - 7071 - 793376 Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535 Geschäftsführer/Gesellschafter: ZOPYX Limited, Birmingham, UK ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting

2 1

how can I find out what version of an egg buildout has selected?
by Chris Withers 26 Jun '08

26 Jun '08

Hi All, If I have a buildout.cfg such as: [buildout] eggs = some.egg some.other.egg ...what's the correct way to find out what versiosn of these eggs buildout has selected and is using? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

2 1

Two bugs in setuptools-0.6c8-py2.5.egg/setuptools/command/sdist.py (entries_finder)
by Pavel Repin 25 Jun '08

25 Jun '08

Hi, I noticed the following by running setup.py in a non SVN-1.4 working copy. 1. When 'entries_finder' gets confused about the svn working copy .svn entries, it tries to log an error by doing: log.warn(....). But there's no 'log' object. See a simple hack that fixes it (the diff attached). 2. It assumes my working copy format is SVN-1.4. But, I happen to use a SVN 1.5 working copy that contains my setup.py. Does 'entries_finder' simply try to find objects that are under version control minus the ones that have pending delete status? If so, maybe invoking 'svn status' in a child process and parsing its output would be better? -- Pavel Repin

1 0

Adding include path during install
by Zahed Emon 25 Jun '08

25 Jun '08

Hi, I am trying to install PJSIP py-pjsua with distutils setup.py install command. But the while building with visual studio .NET 2003 it complains about missing header files. the files are there I suppose its not finding paths to it. SO is it possible to give the path to include directory? thanks in advance BR Sarker

1 0

buildout: setting envirionment variables for zc.recipe.egg:custom?
by Christian Zagrodnick 24 Jun '08

24 Jun '08

Hi, building lxml requires additional configuration so it finds xslt-config correctly. One possibility is to set the PATH environment variable. The other is to call setup.py --xslt-config=... Using setup.py --xslt-config doesn't fit at all into buildout i think. Setting environment variables before installing an egg on the other hand might be useful. Would it make sense to put this into zc.recipe.egg or would it be better to create some unrelated recipe which only sets envirionment variables? Regards, -- Christian Zagrodnick gocept gmbh & co. kg · forsterstrasse 29 · 06112 halle/saale www.gocept.com · fon. +49 345 12298894 · fax. +49 345 12298891

3 5

easy_install installing old version
by Patrick Lewis 24 Jun '08

24 Jun '08

Hello, I'm having some trouble with easy_install finding and installing old packages, and I'm not sure why. Specifically, with python2.4, easy_install wants to grab the oldest egg file possible, rather than newer egg or tar.gz source distributions. Here is the pypi page: http://pypi.python.org/pypi/registration and here is a sample install (from a fresh virtualenv) (ENV_24_clean)pat-mac:~/Programming/TurboGears/ENV_24_clean plewis$ which easy_install-2.4 /Users/plewis/Programming/TurboGears/ENV_24_clean/bin/easy_install-2.4 (ENV_24_clean)pat-mac:~/Programming/TurboGears/ENV_24_clean plewis$ easy_install-2.4 -U registration Searching for registration Reading http://pypi.python.org/simple/registration/ Reading http://patrickhlewis.googlepages.com/registration.html Reading http://pat.lewis.home.insightbb.com/ Best match: registration 0.11 Processing registration-0.11-py2.4.egg registration 0.11 is already the active version in easy-install.pth I'm not quite sure what's going on here, since http://pypi.python.org/simple/registration/ clearly shows newer versions. If someone could point out what I've done wrong, I'd much appreciate it.

2 2

problem with setuptools when using subversion 1.5
by robert rottermann 23 Jun '08

23 Jun '08

Hi there, I found two problems using setuptools 0.6c8 in conjunction with suversion 1.5 1. the file setuptools/commands/sdist.py checks for '8' as the start of an svn.entires file (line 89). in subversion 1.5 this is '9'. 2. it the tries to log a warning, but log is not imported (line 98). I changed '8' to '9' and all works fine again. seems no further changes in the structure of the file have been incoorporated that would affect the working of setuptools. thanks for your great tools robert

2 1

distutils sprint wrapup
by Tarek Ziadé 22 Jun '08

22 Jun '08

Hello, We sprinted together with Chris on adding doctests with distutils, by taking each command and adding a doctest that demonstrates it. Chris worked under windows and I worked under Mac, which was great to get more robust tests quickly. We covered so far 7 commands with various level of details (egg_infos.txt is more detailed for instance). We think it is the best approach to start things up, as the tests will evolve on each code change later. I guess we will continue covering commands, and wait for Phillip feedback and others. Especially about the amount of "magic" doctesting.py introduces into the doctests syntax. See : http://bazaar.launchpad.net/~setuptools/setuptools-test/main/files That was a very productive and fun sprint, ++ Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

1 0