Mailman 3 September 2009 - Distutils-SIG

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

Re: [Distutils] how to easily consume just the parts of eggs that are good for you
by Stanley A. Klein 09 Oct '09

09 Oct '09

On Wed, April 9, 2008 10:05 pm, Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: > Message: 4 > Date: Thu, 10 Apr 2008 12:59:39 +1200 > From: Greg Ewing <greg.ewing(a)canterbury.ac.nz> > Subject: Re: [Distutils] how to easily consume just the parts of eggs > that are good for you > To: distutils-sig(a)python.org > > Paul Moore wrote: > >> I believe that Mac OS X goes for an even simpler structure - >> applications store *everything* in the one directory, so that >> install/uninstall is simply a directory copy/remove. > > Yep, and thereby cuts the whole gordian knot, throws the > pieces on the fire and burns them. :-) > > Package managers have always seemed to me to be an > excessively complex solution to a problem that needn't > have existed in the first place. > > I keep hoping that someday Linux will support something > like MacOSX application bundles and frameworks, but I > haven't seen any sign of it yet. > I think this discussion arises because the operating systems we are dealing with are based on very different concepts. Windows and Mac are fundamentally single user systems that have added capabilities for multiple users and are intended to be used with proprietary software. Those considerations lead to minimal dependencies among packages (each proprietary provider needs to control its own package, except for the OS), individual users serving as their own sysadmins, and similar factors. Any dependencies in the proprietary software are hidden from the user because the provider has compiled the dependencies into the binary code they supply. Unix/Linux are fundamentally multi-user systems with a distinct role for a sysadmin. Linux and the BSD's are intended to be used with Free/Open Source Software (FOSS), and Unix originated the Software Tools concept in which individual, relatively simple, separately-developed tools are combined to perform complex tasks. Both FOSS and the Software Tools concept encourage dependencies. The need for package managers arises out of the Unix FHS. If you look at the FHS, it is clearly designed to simplify the job of the sysadmin in a multi-user system that uses FOSS and Software Tools. For example, deciding what to backup and how often to do it in a Unix/Linux system is relatively easy for the sysadmin. All the installed software is in certain top-level directories. All the config files are in /etc. All the logs. caches, spools, web pages, process locks, and certain other data are in /var. All the user data is in /home or its sysadmin-determined equivalents. If the sysadmin needs space, deleting files in /tmp will not cause a problem. In summary, Python is being used on systems that have very different underlying OS use cases. To some extent, the natural use case for Python is closest to that of Linux/Unix. Running Python on Windows/Mac requires adapting for those platforms some of the kinds of tools that simplify operations on Linux/Unix systems. This discussion is essentially about how far that goes, how to accomplish it, and how to remain compatible with the existing tools on Linux/Unix. Stan Klein

4 4

Get install prefix for module at runtime
by Wolodja Wentland 04 Oct '09

04 Oct '09

Hi all, i am pretty new to distutils (since yesterday) and am struggling with accessing installed data_files. I want to ship a library that benefits from some data_files from another package. My wishes: * distribution should go in 'PREFIX/lib/foo' (site-packages?) * datafiles should go in 'PREFIX/share/foo' * Access to data files from modules My question is: "How to reliably access data files from a module in foo?" A thorough explanation follows. If you can answer the aforementioned question just think tl;dr and skip it ;-) Installation Schemes ==================== There seem to be three different installation schemes regarding setting of this PREFIX. The following examples assume installation of package 'foo' which contains module 'bar' and data file 'baz.data'. Default install (setup.py install) ----------------------------------------- * Packages installation path: 'sys.prefix/lib/pythonX.Y/site-packages/' * Data files installation path: 'sys.prefix/share/' That means that in the aforementioned example the following file system structure will be created upon installation: Package: /usr/lib/python2.5/site-packages/foo/bar.py Data: /usr/share/foo/baz.data Home scheme (setup.py install --home=HOME) --------------------------------------- * Packages installation path: 'HOME/lib/python' * Data files installation path: 'HOME/share/' Resulting structure: Package: HOME/lib/foo/bar.py Data HOME/share/foo/baz.data Prefix scheme (setup.py --prefix=PREFIX) * Package installation path: 'PREFIX/lib/pythonX.Y/site-packages' * Data installation path: 'PREFIX/share/' Resulting structure (for PREFIX==/usr/local) Package: /usr/local/lib/python2.5/site-packages/foo/bar.py Data: /usr/share/foo/baz.data Finding Data files ================== My problem is finding a reliable way to access the data files from within bar.py. Approach 1 - using module.__file__ ---------------------------------- bar_path_elements = bar.__file__.split(os.path.sep) prefix = os.path.join( os.path.sep, *itertools.takewhile(lambda el: el != 'lib')) data_file_path = os.path.join(prefix, 'share', 'foo') * Will this work reliably? * What if the user specifies a different data directory with '--install-data' ? Approach 2 - writing a build.py file at installation time --------------------------------------------------------- The last question made me think, that Approach 1 works for the three standard installation schemes, but fails miserably if the user decides to do anything fancy. As i do not comprehend all the options you can give to 'setup.py install' or know about them i am not sure if there are not many more problems hidden in there. The solution seems to be to create a build.py file that includes this information at installation time and access the data there. My problem is just: HOW? I tried with the following setup.py script, but am unsure how to access the correct options: --- snip --- # -*- coding: UTF-8 -*- #!/usr/bin/env python from __future__ import with_statement from distutils.core import setup import sys import getopt def create_build_file(): try: opts, args = getopt.getopt(sys.argv[1:], "", []) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ What to do here???? except getopt.GetoptError, err: pass with open('lib/mwdb/build.py', 'w') as build_fp: #build_fp.write('# -*- coding: UTF-8 -*-\n') #build_fp.write("LIB_PREFIX = '%s'\n"%(options.lib_prefix)) #build_fp.write("DATA_PREFIX = '%s'\n"%(options.data_prefix)) #build_fp.write("EXEC_PREFIX = '%s'\n"%(options.exec_prefix)) try: if 'install' == sys.argv[1]: create_build_file() except IndexError: pass setup(name='foo', version='0.1', packages=['foo'] package_dir = { '':'lib' }, ... ) --- snip --- The problem i am facing now is: How to parse the options? Do I really have to recreate the complete distutils option parsing logic - and probably fail? All that without optparse! How can i get the aforementioned options? Is the file creation way the correct way to handle this? I really don't know what to do now and am also wondering why nobody had that problem before! Are all other libraries shipped with external data buggy for some installation schemes? How is this normally solved? with kind regards and thanks for reading all this Wolodja Wentland

3 9

stdeb-0.3 error
by Gerry Reno 03 Oct '09

03 Oct '09

sudo easy_install stdeb # brought in stdeb 0.3 $ cd myappdir # where my setup.py is located $# following http://github.com/astraw/stdeb/ quickstart 1 $ python -c "import stdeb; execfile('setup.py')" sdist_dsc && \ > cd `find deb_dist -mindepth 1 -maxdepth 1 -type d` && \ > dpkg-buildpackage -rfakeroot -uc -us && \ > cd ../.. && > echo ".deb created successfully in deb_dist/" Traceback (most recent call last): File "<string>", line 1, in <module> File "setup.py", line 41, in <module> sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)), "bin")) NameError: name '__file__' is not defined ========================================================= The error is coming from the setup.py line: import imp import sys import os import glob from distutils.core import setup, Command from distutils.command.install import install if os.name == 'nt': import py2exe sys.path.append(os.path.join(os.path.abspath(os.path.dirname(__file__)), "bin")) =========================================================== I do not get this error when I run setup.py normally for other commands like 'sdist'. How can I fix this? Regards, Gerry

7 86

compiling psycopg2 with zc.recipe.egg: allowing custom build_ext options?
by Kevin Teague 01 Oct '09

01 Oct '09

Regarding Chris recent thread on doing some hair pulling while getting psycopg2 to install, I recently added this distribution to a buildout with the following configuration: [versions] pscopg2 = 2.0.8 [buildout] parts = app postgresql init-pgsql pgsql-symlinks psycopg2 find-links = http://initd.org/pub/software/psycopg/PSYCOPG-2-0/ versions = versions # app is a Grok-based application that requires pscopg2 [app] recipe = zc.recipe.egg eggs = ${psycopg2:egg} gsc.docsdb z3c.evalexception>=2.0 Paste PasteScript PasteDeploy interpreter = python-console # PostgreSQL parts [psycopg2] recipe = zc.recipe.egg:custom egg = psycopg2 include-dirs = ${postgresql:location}/include library-dirs = ${postgresql:location}/lib [postgresql] recipe = zc.recipe.cmmi url = http://wwwmaster.postgresql.org/redir/198/h/source/v8.3.7/postgresql-8.3.7.… extra_options = --with-readline [init-pgsql] recipe = iw.recipe.cmd on_install = true on_update = false cmds = ${postgresql:location}/bin/initdb -D ${postgresql:location}/var/data -E UNICODE [pgsql-symlinks] recipe = cns.recipe.symlink symlink_target = ${buildout:directory}/bin symlink_base = ${postgresql:location}/bin symlink = clusterdb createdb createlang createuser dropdb droplang dropuser ecpg initdb ipcclean pg_config pg_controldata pg_ctl pg_dump pg_dumpall pg_resetxlog pg_restore postgres postmaster psql reindexdb vacuumdb This builds a complete PostgreSQL install inside the project directory. Of course, if you've already got a preferred postgres install, you can simplify this quite a bit and just point the include- dirs and library-dirs to that location. I did find that I had to pin psycopg2 down to 2.0.8, since it didn't seem to work with 2.0.10. But looking a bit further into this issue, the preferred way to build psycopg2 is to supply the full path to pg_config as a build_ext. However, zc.recipe.egg:custom only supports a fixed list of build_ext options. Is this a feature or a bug? e.g. Is there any reason why it wouldn't be possible to support arbitrary build_ext options with this recipe? Or is there another egg-building recipe out there which supports this? By patching a local copy of zc.recipe.egg to allow the pg_config build_ext option, I was able to use this part to compile psycopg2: [psycopg2] recipe = zc.recipe.egg:custom egg = psycopg2 pg_config = ${buildout:bin-directory}/pg_config Which worked fine for the latest release, 2.0.10, of psycopg2. (also, as a bonus 'Arg!', note that it's necessary to supply a custom find-links URL to buildout, since the PyPI page for psycopg2 has bogus download links)

2 1

something else to throw into the static metadata mix
by Chris Withers 01 Oct '09

01 Oct '09

Hi All, Sorry to bring this up if it makes things more difficult, if it's been solved or is covered by existing proposals, please point me in the right way. I've had a bit of a torrid time building psycopg2 from source of late, and in my dream world, being able to specify psycopg2 as a dependent egg would "just work" and pull down the right statically linked binary egg. This raises a problem that I haven't seen covered anywhere: not only would these eggs have to be platform dependent, they'd also need to be dependent on the required version of postgres. Does setuptools/buildout/any other existing or proposed packaging solution cater for this kind of requirement? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

8 15

[buildout] is there a way to force re-install of a part?
by Chris Withers 30 Sep '09

30 Sep '09

Hi All, Is there any way to force a part to re-install? I have a part using the djangorecipe part that stubbornly refuses to notice that I've changed an eggs dependencies or changed the stuff in the develop line of the [buildout] section. I've tried "bin/buildout install djangopart" but this doesn't have any effect. The only way I've found to get the part to re-install is to change the options in the part, install, and then put the options back like I want them before re-installing :-( cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

3 3

Bug in setuptool's findall
by Aaron Staley 30 Sep '09

30 Sep '09

Hello, setuptools replaces distutl's findall() with: for base, dirs, files in os.walk(dir): print 'walk', base, dirs, files if base==os.curdir or base.startswith(os.curdir+os.sep): base = base[2:] if base: files = [os.path.join(base, f) for f in files] all_files.extend(filter(os.path.isfile, files)) This unfortunately will not follow directory symlinks (as followlinks is set to false in os.walk by default). This is NOT the behavior of the original findall; oddly enough, the new findall actually produces bugs relative to distutil's. -Aaron

2 1