Mailman 3 October 2009 - Distutils-SIG

Re: [Distutils] Package install failures in 2.6.3 - setuptools vs Distribute
by P.J. Eby 04 Oct '09

04 Oct '09

At 03:49 PM 10/3/2009 +0200, Tarek Ziadé wrote: >Notice that this has been fixed in Ubuntu already with a patched >version of setuptools Is the patch or an equivalent already in the setuptools tracker? And if not, can someone please post it there? Thanks.

2 1

[issue85] setuptools builds of extension modules fail with Python 2.6.3
by Ned Deily 04 Oct '09

04 Oct '09

New submission from Ned Deily <nad(a)acm.org>: A change in distutils distributed with Python 2.6.3 breaks setuptools. Builds of extension modules fail with n get_ext_filename ext = self.ext_map[fullname] KeyError: '<xx>'. An analysis and patch is documented in the Distribute tracker issue 41: http://bitbucket.org/tarek/distribute/issue/41/ ---------- messages: 357 nosy: nad priority: critical status: unread title: setuptools builds of extension modules fail with Python 2.6.3 _______________________________________________ Setuptools tracker <setuptools(a)bugs.python.org> <http://bugs.python.org/setuptools/issue85> _______________________________________________

1 0

Get install prefix for module at runtime
by Wolodja Wentland 04 Oct '09

04 Oct '09

Hi all, i am pretty new to distutils (since yesterday) and am struggling with accessing installed data_files. I want to ship a library that benefits from some data_files from another package. My wishes: * distribution should go in 'PREFIX/lib/foo' (site-packages?) * datafiles should go in 'PREFIX/share/foo' * Access to data files from modules My question is: "How to reliably access data files from a module in foo?" A thorough explanation follows. If you can answer the aforementioned question just think tl;dr and skip it ;-) Installation Schemes ==================== There seem to be three different installation schemes regarding setting of this PREFIX. The following examples assume installation of package 'foo' which contains module 'bar' and data file 'baz.data'. Default install (setup.py install) ----------------------------------------- * Packages installation path: 'sys.prefix/lib/pythonX.Y/site-packages/' * Data files installation path: 'sys.prefix/share/' That means that in the aforementioned example the following file system structure will be created upon installation: Package: /usr/lib/python2.5/site-packages/foo/bar.py Data: /usr/share/foo/baz.data Home scheme (setup.py install --home=HOME) --------------------------------------- * Packages installation path: 'HOME/lib/python' * Data files installation path: 'HOME/share/' Resulting structure: Package: HOME/lib/foo/bar.py Data HOME/share/foo/baz.data Prefix scheme (setup.py --prefix=PREFIX) * Package installation path: 'PREFIX/lib/pythonX.Y/site-packages' * Data installation path: 'PREFIX/share/' Resulting structure (for PREFIX==/usr/local) Package: /usr/local/lib/python2.5/site-packages/foo/bar.py Data: /usr/share/foo/baz.data Finding Data files ================== My problem is finding a reliable way to access the data files from within bar.py. Approach 1 - using module.__file__ ---------------------------------- bar_path_elements = bar.__file__.split(os.path.sep) prefix = os.path.join( os.path.sep, *itertools.takewhile(lambda el: el != 'lib')) data_file_path = os.path.join(prefix, 'share', 'foo') * Will this work reliably? * What if the user specifies a different data directory with '--install-data' ? Approach 2 - writing a build.py file at installation time --------------------------------------------------------- The last question made me think, that Approach 1 works for the three standard installation schemes, but fails miserably if the user decides to do anything fancy. As i do not comprehend all the options you can give to 'setup.py install' or know about them i am not sure if there are not many more problems hidden in there. The solution seems to be to create a build.py file that includes this information at installation time and access the data there. My problem is just: HOW? I tried with the following setup.py script, but am unsure how to access the correct options: --- snip --- # -*- coding: UTF-8 -*- #!/usr/bin/env python from __future__ import with_statement from distutils.core import setup import sys import getopt def create_build_file(): try: opts, args = getopt.getopt(sys.argv[1:], "", []) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ What to do here???? except getopt.GetoptError, err: pass with open('lib/mwdb/build.py', 'w') as build_fp: #build_fp.write('# -*- coding: UTF-8 -*-\n') #build_fp.write("LIB_PREFIX = '%s'\n"%(options.lib_prefix)) #build_fp.write("DATA_PREFIX = '%s'\n"%(options.data_prefix)) #build_fp.write("EXEC_PREFIX = '%s'\n"%(options.exec_prefix)) try: if 'install' == sys.argv[1]: create_build_file() except IndexError: pass setup(name='foo', version='0.1', packages=['foo'] package_dir = { '':'lib' }, ... ) --- snip --- The problem i am facing now is: How to parse the options? Do I really have to recreate the complete distutils option parsing logic - and probably fail? All that without optparse! How can i get the aforementioned options? Is the file creation way the correct way to handle this? I really don't know what to do now and am also wondering why nobody had that problem before! Are all other libraries shipped with external data buggy for some installation schemes? How is this normally solved? with kind regards and thanks for reading all this Wolodja Wentland

3 9

stdeb-0.3 error
by Gerry Reno 03 Oct '09

03 Oct '09

sudo easy_install stdeb # brought in stdeb 0.3 $ cd myappdir # where my setup.py is located $# following http://github.com/astraw/stdeb/ quickstart 1 $ python -c "import stdeb; execfile('setup.py')" sdist_dsc && \ > cd `find deb_dist -mindepth 1 -maxdepth 1 -type d` && \ > dpkg-buildpackage -rfakeroot -uc -us && \ > cd ../.. && > echo ".deb created successfully in deb_dist/" Traceback (most recent call last): File "<string>", line 1, in <module> File "setup.py", line 41, in <module> sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)), "bin")) NameError: name '__file__' is not defined ========================================================= The error is coming from the setup.py line: import imp import sys import os import glob from distutils.core import setup, Command from distutils.command.install import install if os.name == 'nt': import py2exe sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)), "bin")) =========================================================== I do not get this error when I run setup.py normally for other commands like 'sdist'. How can I fix this? Regards, Gerry

7 86

Problem using zc.recipe.cmmi to build Python
by Kent Tenney 03 Oct '09

03 Oct '09

Howdy, In pursuit of a buildout for my core environment, I try the following: [python-2.6.3] recipe = zc.recipe.cmmi url = http://www.python.org/ftp/python/2.6.3/Python-2.6.3rc1.tgz ... make fails with ImportError: No module named cStringIO because ./configure creates a Modules/Setup which says #cStringIO cStringIO.c if I uncomment that line the make is fine. I haven't found a way to tell $:./configure or $:make to do that, zc.recipe.cmmi doesn't seem to provide a hook between configure and make. If someone can tell me to fix this with options or environment I'm done. If not, I welcome advice on alternatives to zc.recipe.cmmi, (which I would consider the 'standard' solution.(darn you Shuttleworth for co-opting the word 'canonical' :-))) - comb through the many xx.xx.cmmi recipes (without a sense of associated quality or support level) - pick a xx.xx.command recipe and write a part which does what I want - write my own 'recipe.build-interp' Thanks, Kent

2 2

Clarification asked wrt FAQ "How does Distribute interacts with virtualenv?"
by Roger Erens 03 Oct '09

03 Oct '09

Hi Tarek e.a., thanks for the nice documentation for Distribute on Pypi. There are however a few more things that are not so clear to me in the virtualenv-FAQ: "How does Distribute interacts with virtualenv?" Typo: interacts -> interact "Everytime you create a virtualenv it will install setuptools, so you need to re-install Distribute in it right after. The Distribute project will not attempt to patch virtualenv so it uses it when globally installed." There are a few too many "it"s in those sentences to my taste. Is this meant: "Everytime you run virtualenv, it will install setuptools in the created virtualenv directory, so you need to install Distribute in that directory right after. (First activating the virtualenv)."? And "The Distribute project will not attempt to patch virtualenv, so [the virtual environment?] uses [the setuptools installed within the virtual environment?] when [Distribute? is [only]] globally installed."? "Once installed, your virtualenv will use Distribute transparently." explicitly this is: "Once Distribute is installed in your virtualenv, your virtualenv will use Distribute transparently."? "Although, if you have Setuptools installed in your system-wide Python, and if the virtualenv you are in was generated without the --no-site-packages option, the Distribute installation will stop." stop -> stop installing? "You need in this case to build a virtualenv with the --no-site-packages option or to install Distribute globally." Maybe it might also be noted that there's a virtualenv-distribute (by Florian Schulze) hanging around somewhere on the web. Hope this helps to improve the documentation, Roger

2 1

compiling psycopg2 with zc.recipe.egg: allowing custom build_ext options?
by Kevin Teague 01 Oct '09

01 Oct '09

Regarding Chris recent thread on doing some hair pulling while getting psycopg2 to install, I recently added this distribution to a buildout with the following configuration: [versions] pscopg2 = 2.0.8 [buildout] parts = app postgresql init-pgsql pgsql-symlinks psycopg2 find-links = http://initd.org/pub/software/psycopg/PSYCOPG-2-0/ versions = versions # app is a Grok-based application that requires pscopg2 [app] recipe = zc.recipe.egg eggs = ${psycopg2:egg} gsc.docsdb z3c.evalexception>=2.0 Paste PasteScript PasteDeploy interpreter = python-console # PostgreSQL parts [psycopg2] recipe = zc.recipe.egg:custom egg = psycopg2 include-dirs = ${postgresql:location}/include library-dirs = ${postgresql:location}/lib [postgresql] recipe = zc.recipe.cmmi url = http://wwwmaster.postgresql.org/redir/198/h/source/v8.3.7/postgresql-8.3.7.… extra_options = --with-readline [init-pgsql] recipe = iw.recipe.cmd on_install = true on_update = false cmds = ${postgresql:location}/bin/initdb -D ${postgresql:location}/var/data -E UNICODE [pgsql-symlinks] recipe = cns.recipe.symlink symlink_target = ${buildout:directory}/bin symlink_base = ${postgresql:location}/bin symlink = clusterdb createdb createlang createuser dropdb droplang dropuser ecpg initdb ipcclean pg_config pg_controldata pg_ctl pg_dump pg_dumpall pg_resetxlog pg_restore postgres postmaster psql reindexdb vacuumdb This builds a complete PostgreSQL install inside the project directory. Of course, if you've already got a preferred postgres install, you can simplify this quite a bit and just point the include- dirs and library-dirs to that location. I did find that I had to pin psycopg2 down to 2.0.8, since it didn't seem to work with 2.0.10. But looking a bit further into this issue, the preferred way to build psycopg2 is to supply the full path to pg_config as a build_ext. However, zc.recipe.egg:custom only supports a fixed list of build_ext options. Is this a feature or a bug? e.g. Is there any reason why it wouldn't be possible to support arbitrary build_ext options with this recipe? Or is there another egg-building recipe out there which supports this? By patching a local copy of zc.recipe.egg to allow the pg_config build_ext option, I was able to use this part to compile psycopg2: [psycopg2] recipe = zc.recipe.egg:custom egg = psycopg2 pg_config = ${buildout:bin-directory}/pg_config Which worked fine for the latest release, 2.0.10, of psycopg2. (also, as a bonus 'Arg!', note that it's necessary to supply a custom find-links URL to buildout, since the PyPI page for psycopg2 has bogus download links)

2 1

No announcement for 0.6.3?
by Roger Erens 01 Oct '09

01 Oct '09

Hi, distribute-0.6.3 was uploaded to Pypi on 09-27-09, but no announcement of its availability was made? Cheers, Roger

2 1

something else to throw into the static metadata mix
by Chris Withers 01 Oct '09

01 Oct '09

Hi All, Sorry to bring this up if it makes things more difficult, if it's been solved or is covered by existing proposals, please point me in the right way. I've had a bit of a torrid time building psycopg2 from source of late, and in my dream world, being able to specify psycopg2 as a dependent egg would "just work" and pull down the right statically linked binary egg. This raises a problem that I haven't seen covered anywhere: not only would these eggs have to be platform dependent, they'd also need to be dependent on the required version of postgres. Does setuptools/buildout/any other existing or proposed packaging solution cater for this kind of requirement? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

8 15