Mailman 3 November 2009 - Distutils-SIG

[issue96] Invalid setup.py flag in srpm spec file (0.6c11)
by Benjamin Riggs 13 Nov '09

13 Nov '09

New submission from Benjamin Riggs <riggs(a)umn.edu>: line 191-2: %install python setup.py install --single-version-externally-managed --root=$RPM_BUILD_ROOT --record-rpm=INSTALLED_OBJECTS causes: error: option --record-rpm not recognized ---------- messages: 463 nosy: riggs priority: bug status: unread title: Invalid setup.py flag in srpm spec file (0.6c11) _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue96> _______________________________________________

1 0

Install time prefixes and data files
by Wolodja Wentland 13 Nov '09

13 Nov '09

Hi all, I had a discussion with Tarek on IRC yesterday and would like to bring some points to your attention and also write down some conclusions/ideas we had yesterday. I think that the distutils module has a serious flaw that need to be addressed before any further work is done on getting PyPi in a better shape. The main issue is, that there are many ways to customise the installation of a distribution but absolutely nothing to retrieve this information within the setup script or within your programs/libraries. tl;dr: save prefixes in .egg-info/PREFIX and create methods for Distribution to retrieve these and data files Install prefixes / Installation schemes --------------------------------------- There are basically five installation schemes: 1. Default 2. User 3. Home 4. Prefix 5. Custom All these installation schemes taken together enable the user to install pure libraries, non-pure libraries, data files and scripts in exactly the places s/he choose. But this also means that it is impossible to make any assumptions about the locations in your code. I think it is of uttermost importance to work on an implementation that enables users to query this information: * at build time (within setup.py) * at run time (within scripts/libraries) The major problem today is that the information where files of a given distribution are placed is *only* available at build time and not saved anywhere. And even at build time you'll have to get the finalised install command to query that information, which is suboptimal and counter-intuitive. Data files ---------- Especially data files are problematic in this respect because the FHS mandates that these are *not* placed within the packages but at exactly defined places in the file system. The API available to retrieve data files in the *stdlib* pkgutil.get_data() can not cope with this and is therefore not usable. This leads to the following problems: * Linux package maintainers have to implement a lot of patches if they want to place data files in the correct places. * Python developers have to either (i) save the various locations at install time in a place where they can retrieve that information later or (ii) accept that their library/program will break if one of the fancier installation schemes is used. * Python distribution installations that violate the FHS * ... Proposal -------- I propose three things: 1. Save installation prefixes for every installed distribution 2. Define standard infixes for typical data file classes 3. Implement an API that is able to query that information/retrieve data files 1 and 3. Discussion The installation prefixes could for example be saved within a suitable file in .egg-info/ which would mandate changes to PEP 376. I could think of a PREFIX file and the following methods/attribute for e.g. the Distribution class * get_install_prefixes() -> returns a dictionary with all prefixes for this distribution * get_data_file(type, name) -> get data file of given type and name (see (see below) 2. Discussion The FHS differentiates between various classes of files and defines proper location for them. We could define platform dependent standard infixes for the following data file classes for distribution foo: * configuration etc * shared data usr/share/foo/ * readme usr/share/foo/README README, TODO, ... could be automatically discovered * examples usr/share/foo/examples * documentation usr/share/foo/doc * man files usr/share/man * variable var/lib/foo * ... There should be a set of platform dependent prefixes for all classes we agree on *and* it should be possible to change/set their default values with, for example, environment variables or command line options. I call them infixes and not prefixes here, because applications like virtualenv might want to define a different root for these files than the system default one. Tarek suggested to use this scheme like this: setup( ... data_files = [ ('$configuration', ['data/my.conf', 'data/default.conf']), ('$examples', ['doc/examples/do_foo.sample']), ('/i/want/file/here', ['custom_placed_file']), ... ] ) The placeholders will then be replaced with their current value at installation time and the locations written to .egg-info/PREFIX. This would allow/ease a couple of things, namely: * automatic placement of typical files in the right locations * much less work for packaging python applications on *nix distributions -> debhelper could take advantage of all these for example * automatic tests/fixers for different file types I think of automatic lintian style tests on PyPi here that will reject a package if it does not conform to a standard or at least warns the user about it * retrieval of certain types of data easily at run time Distribution.get_data_file('$configuration', 'my.cfg') -> file object * ... thanks! -- .''`. Wolodja Wentland <wentland(a)cl.uni-heidelberg.de> : :' : `. `'` 4096R/CAF14EFC `- 081C B7CD FF04 2BA9 94EA 36B2 8B7F 7D30 CAF1 4EFC

7 13

distutils.core.setup --install-script option in Python 2.6 ?
by Jim Vickroy 13 Nov '09

13 Nov '09

Hello, Please excuse the repost (sent last week), but I'm hoping someone on the list can get me on-track with a distutils problem. I have just upgraded from Python 2.5 to 2.6.4 (MS Windows XP service pack 3). My setup.py scripts (based on distutils.core.setup) no longer work with the --install-script option. Here is the error: * error: option --install-script not recognized The above error occurs when passing either bdist_wininst or bdist_msi parameters to the setup.py scripts. I have been unable to locate any on-line mention of this option not being supported in Python 2.6. I apologize for missing something obvious here, but I am stuck. Thanks in advance for your feedback. -- jv

2 3

Re: [Distutils] People want CPAN :-)
by John Kleint 12 Nov '09

12 Nov '09

>Tarek Ziadé schrieb: > >> For the documentation part I am afraid it will be messy for the end >> users trying to package apps in Python *until* all PEPs >> have made it into Python. >> >> Although, as Ian Bicking says: we could write today some kind of >> all-in-one tutorial so end-users can work out without having to run >> after the documenation in several places. > >Big +1, and I would be more than willing to include it in the standard >Python documentation, *even* if it mostly describes setuptools/distribute/pip. > >When people want to package a library, they *will* look for docs in Python, >and at the moment they only find the distutils reference. While the latter >is necessary, a more howto-like standard document is much needed. > >Georg I have the start of such a document, derived from my own recent experience pulling together all the information needed to release a new Python project. It's aimed at people new to packaging and possibly open source in general, so it's broader than just packaging. It includes things like basic file layout, unit testing, Pylint, getting started with Sphinx, a sample (distutils) setup.py, and how to register with PyPI. I try to pull together all the info a novice would need to have a reasonable shot at sharing their code. It's by no means all-encompassing (e.g. no mention of native extensions), but perhaps someone would be willing to write a complimentary advanced packaging howto. You can find it at: http://infinitemonkeycorps.net/docs/pph/ Let me know what you think. -John

3 2

[issue95] [PATCH] ez_setup.py fails when pkg_resources is installed but setuptools is not
by Zooko O'Whielacronx 12 Nov '09

12 Nov '09

New submission from Zooko O'Whielacronx <zooko(a)zooko.com>: This just happened on Ubuntu Karmic: HACL zooko-desktop:/sdb/zooko/ecometry2rabbit/ecometry2rabbit$ /usr/bin/python setup.py test Traceback (most recent call last): File "setup.py", line 2, in <module> use_setuptools() File "/sdb/zooko/ecometry2rabbit/ecometry2rabbit/ez_setup.py", line 116, in use_setuptools if not setuptools_is_new_enough(min_version): File "/sdb/zooko/ecometry2rabbit/ecometry2rabbit/ez_setup.py", line 88, in setuptools_is_new_enough pkg_resources.require('setuptools >= %s' % (required_version,)) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 626, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 524, in resolve raise DistributionNotFound(req) # XXX put more info here pkg_resources.DistributionNotFound: setuptools>=0.6c7 As you can see from the following, the pkg_resources module is importable even though setuptools is not installed: HACL zooko-desktop:/sdb/zooko/ecometry2rabbit/ecometry2rabbit$ /usr/bin/python -c 'import pkg_resources;print pkg_resources;print pkg_resources.require("setuptools")' <module 'pkg_resources' from '/usr/lib/python2.6/dist-packages/pkg_resources.pyc'> Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 626, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python2.6/dist-packages/pkg_resources.py", line 524, in resolve raise DistributionNotFound(req) # XXX put more info here pkg_resources.DistributionNotFound: setuptools This is because Debian and Ubuntu package pkg_resources separately: HACL zooko-desktop:/sdb/zooko/ecometry2rabbit/ecometry2rabbit$ dpkg --search pkg_resources.py python-pkg-resources: /usr/share/pyshared/pkg_resources.py python-pkg-resources: /usr/lib/python2.5/site-packages/pkg_resources.py python-pkg-resources: /usr/lib/python2.6/dist-packages/pkg_resources.py HACL zooko-desktop:/sdb/zooko/ecometry2rabbit/ecometry2rabbit$ dpkg --list python-pkg-resources Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Description +++-=====================================================-=====================================================-========================================================================================================================== ii python-pkg-resources 0.6c9-0ubuntu5 Package Discovery and Resource Access using pkg_resources HACL zooko-desktop:/sdb/zooko/ecometry2rabbit/ecometry2rabbit$ dpkg --list python-setuptools Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Cfg-files/Unpacked/Failed-cfg/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Description +++-=====================================================-=====================================================-========================================================================================================================== un python-setuptools <none> (no description available) The following patch makes ez_setup.py treat this as a case where it needs to bootstrap setuptools itself: diff --git a/ez_setup.py b/ez_setup.py index 6dd60f9..8b2ad2a 100644 --- a/ez_setup.py +++ b/ez_setup.py @@ -86,6 +86,11 @@ def setuptools_is_new_enough(required_version): else: try: pkg_resources.require('setuptools >= %s' % (required_version,)) + except pkg_resources.DistributionNotFound: + # No setuptools installed (even though pkg_resources is installed)... + pkg_resources.__dict__.clear() # "If you want to be absolutely sure... before deleting it." --said PJE on IRC + del sys.modules['pkg_resources'] + return False except pkg_resources.VersionConflict: # An insufficiently new version is installed. pkg_resources.__dict__.clear() # "If you want to be absolutely sure... before deleting it." --said PJE on IRC ---------- messages: 462 nosy: zooko priority: bug status: unread title: [PATCH] ez_setup.py fails when pkg_resources is installed but setuptools is not _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue95> _______________________________________________

1 0

Fwd: People want CPAN :-)
by Antonio Cavallo 12 Nov '09

12 Nov '09

I might be the only one that doesn't need CPAN here but ... I'll try to explain the needs of a contractor working on some python projects in a commercial environment: 1. I don't need a replacement for rpm/yast/yum/synaptic: they provide file tracking and package resolution/dependency already and pepole are using them (and it has been for a very long time) 2. I use (and I'm failry happy) with bdist_wininst: is solves all my problems in windows 3. As part of my duty I must record and track about licenses and ownerships: I don't need a package build system that goes over internet fetching things 4. I might be working as administrator (as normal in windows) and doing the "CPAN" way will possibly overwrite packages upong which other components are depending I've used CPAN in the past and as system administrator I garantee you is the last thing you'd like to have around (and any system like that for that matter): there're are ALREADY platform standards, please don't ignore them. Regards, Antonio Cavallo PS. Why is that difficult a parallel install of packages under site-packages/mypackage/X.Y.Z enforcing the X.Y.Z numbering? Have I miss something? Am I too naive? On Fri, Nov 6, 2009 at 5:53 PM, Guido van Rossum <guido(a)python.org> wrote: > I just found this comment on my blog. People have told me this in > person too, so I believe it is real pain (even if the solution may be > elusive and the suggested solutions may not work). But I don't know > how to improve the world. Is the work on distutils-sig going to be > enough? Or do we need some other kind of work in addition? Do we need > more than PyPI?

4 3

Fwd: installing lxml ? - todays user experience from python-list@python.org
by David Lyon 11 Nov '09

11 Nov '09

-------- Original Message -------- Subject: installing lxml ? Date: Wed, 11 Nov 2009 04:49:48 -0800 (PST) From: 7stud <bbxx789_05ss(a)yahoo.com> To: python-list(a)python.org I'm trying to install lxml, but I can't figure out the installation instructions. Here: http://codespeak.net/lxml/installation.html it says: 1) Get the easy_install tool. Ok, I went to the easy_install website, downloaded, and installed it. The last two lines of the output during installation said this: Installing easy_install script to /Library/Frameworks/Python.framework/ Versions/2.6/bin Installing easy_install-2.6 script to /Library/Frameworks/ Python.framework/Versions/2.6/bin 2) ...run the following as super-user (or administrator): easy_install lxml On MS Windows, the above will install the binary builds that we provide. If there is no binary build of the latest release yet, please search PyPI for the last release that has them and pass that version to easy_install like this: easy_install lxml==2.2.2 On Linux (and most other well-behaved operating systems), easy_install will manage to build the source distribution as long as libxml2 and libxslt are properly installed, including development packages, i.e. header files, etc. Use your package management tool to look for packages like libxml2-dev or libxslt-devel if the build fails, and make sure they are installed. On MacOS-X, use the following to build the source distribution, and make sure you have a working Internet connection, as this will download libxml2 and libxslt in order to build them: STATIC_DEPS=true easy_install lxml ----------- My os is mac os x 10.4.11. But this: STATIC_DEPS=true easy_install lxml is not a valid command: $ sudo STATIC_DEPS=true easy_install lxml Password: sudo: STATIC_DEPS=true: command not found In any case, if I do this: $ sudo easy_install lxml sudo: easy_install: command not found In other words, when I installed easy_install it did not add anything to my PATH which points to the installation directory mentioned during installation: Installing easy_install script to /Library/Frameworks/Python.framework/ Versions/2.6/bin Ok, so I need to use the full path to the easy_install program (which is not mentioned ANYWHERE in the installation instructions), i.e. /Library/Frameworks/Python.framework/Versions/2.6/bin/easy_install ...but this still isn't going to work: $ sudo STATIC_DEPS=true /Library/Frameworks/Python.framework/Versions/ 2.6/bin/easy_install lxml Password: sudo: STATIC_DEPS=true: command not found So what the heck is going on?? Attention developers: you may be one of the best programmers in the world, but if you can't convey how to use your software to the average user, then you are the equivalent of one of the worst programmers on the planet. -- http://mail.python.org/mailman/listinfo/python-list

5 4

buildout/distribute recursion error fixed
by Reinout van Rees 11 Nov '09

11 Nov '09

Hi all, Several people have seen a hard-to-reproduce recursion error when running buildout in combination with distribute. You'd get 200 lines of "distribute". Good news: the problem has been identified. And I've fixed it on a zc.buildout branch (branches/reinout_distribute_upgrade_check). Tarek will test it and probably/hopefully release it real soon, wink, wink. Buildout's easy_install.py module detects the location of setuptools and buildout at import time and uses those paths when creating scripts. Setuptools's path is also used when calling easy_install to install .tgz files. Problem: the install() method that calls easy_install detects the setuptools location all over again at runtime. And there's a recursion in there. The method that searches for the available setuptools detects that there's a new version available and tries to install() it. And searches for the setuptools location. And tries to install() it. Repeat. I fixed it by simply using the pre-detected setuptools location. Note that setuptools did not have this recursion problem solely because it was packaged as an ``.egg``, which does not have to go through the easy_install step! So distribute's (necessary) packaging as a tgz exposed this bug. Reinout -- Reinout van Rees - reinout(a)vanrees.org - http://reinout.vanrees.org Software developer at http://www.thehealthagency.com "Military engineers build missiles. Civil engineers build targets"

2 4

Looking for portable what to determine directory where extensions are installed?
by Tom Epperly 11 Nov '09

11 Nov '09

[I am resending this because nobody responded to my first inquiry. I am hoping it just got overlooked in the discussion about CPAN because I am looking for the right solution for a bug in my tool that links Python to other languages.] I work on a language interoperability tool, Babel https://computation.llnl.gov/casc/components/components.html; and I need a portable way to determine the directory where Python extension modules are installed by distutils (i.e., lib or lib64) to resolve this issue: https://www.cca-forum.org/bugs/babel/issue670 For example, I will invoke "python setup.py install --prefix=/home/foo/_inst --exec-prefix=/home/foo/_inst", and it usually installs the extensions in either /home/foo/_inst/lib/python2.5/site-packages or /home/foo/_inst/lib64/python2.5/site-packages. Initially, I did the following to determine where the extensions actually got installed: # assume exec_prefix=/home/foo/_inst # assume python_version=`python -c 'import sys; print sys.version' | sed '1s/^$...$.*/\1/g;1q'` RUNTIME_PYTHON="$exec_prefix/lib/python$python_verbose/site-packages" This worked fine until I started running on 64-bit Python machines where the correct result was RUNTIME_PYTHON="$exec_prefix/lib64/python$python_verbose/site-packages" The first 64-bit machine I was using seemed to have this patch: http://bugs.python.org/file14726/Python-2.6.2-multilib.patch, so I amended my script to the following: pylib=`$PYTHON -c "import sys; print sys.__dict__.get('lib','lib')"` RUNTIME_PYTHON="$exec_prefix/$pylib/python$python_version/site-packages" However, this approach doesn't work with Fedora Core 12 prerelease or some other 64-bit Linux distributions. They don't define "sys.lib". I thought distutils.sysconfig.get_python_lib() might be helpful, but it returns "/usr/lib/python2.6/site-packages" even on the system where "/usr/lib64/python2.6/site-packages" is the right answer. Is there a canonical, Pythonic way to determine whether an installation of Python uses "lib" or "lib64"? Or is there a way to determine the full path of where distutils will install extensions relative to the specified exec prefix? Regards, Tom Epperly

3 2

A Python Packaging Tutorial
by Tarek Ziadé 10 Nov '09

10 Nov '09

As Ian suggested, and since others thinks its a good idea, I suggest that we start writing a "python packaging guide" using reStructuredText and DVCS, explaining how to handle packaging today. (with *all* existing tools) When it's done, we could add it in python's Doc/ with the help of Georg. I think Pauli provided a good starting list of what is not obvious for a new comer. I propose to start a bitbucket repo, and people can contribute some writing. Regards, Tarek

2 1

Distutils-SIG November 2009