Mailman 3 August 2009 - Distutils-SIG

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

Re: [Distutils] how to easily consume just the parts of eggs that are good for you
by Stanley A. Klein 09 Oct '09

09 Oct '09

On Wed, April 9, 2008 10:05 pm, Greg Ewing <greg.ewing(a)canterbury.ac.nz> wrote: > Message: 4 > Date: Thu, 10 Apr 2008 12:59:39 +1200 > From: Greg Ewing <greg.ewing(a)canterbury.ac.nz> > Subject: Re: [Distutils] how to easily consume just the parts of eggs > that are good for you > To: distutils-sig(a)python.org > > Paul Moore wrote: > >> I believe that Mac OS X goes for an even simpler structure - >> applications store *everything* in the one directory, so that >> install/uninstall is simply a directory copy/remove. > > Yep, and thereby cuts the whole gordian knot, throws the > pieces on the fire and burns them. :-) > > Package managers have always seemed to me to be an > excessively complex solution to a problem that needn't > have existed in the first place. > > I keep hoping that someday Linux will support something > like MacOSX application bundles and frameworks, but I > haven't seen any sign of it yet. > I think this discussion arises because the operating systems we are dealing with are based on very different concepts. Windows and Mac are fundamentally single user systems that have added capabilities for multiple users and are intended to be used with proprietary software. Those considerations lead to minimal dependencies among packages (each proprietary provider needs to control its own package, except for the OS), individual users serving as their own sysadmins, and similar factors. Any dependencies in the proprietary software are hidden from the user because the provider has compiled the dependencies into the binary code they supply. Unix/Linux are fundamentally multi-user systems with a distinct role for a sysadmin. Linux and the BSD's are intended to be used with Free/Open Source Software (FOSS), and Unix originated the Software Tools concept in which individual, relatively simple, separately-developed tools are combined to perform complex tasks. Both FOSS and the Software Tools concept encourage dependencies. The need for package managers arises out of the Unix FHS. If you look at the FHS, it is clearly designed to simplify the job of the sysadmin in a multi-user system that uses FOSS and Software Tools. For example, deciding what to backup and how often to do it in a Unix/Linux system is relatively easy for the sysadmin. All the installed software is in certain top-level directories. All the config files are in /etc. All the logs. caches, spools, web pages, process locks, and certain other data are in /var. All the user data is in /home or its sysadmin-determined equivalents. If the sysadmin needs space, deleting files in /tmp will not cause a problem. In summary, Python is being used on systems that have very different underlying OS use cases. To some extent, the natural use case for Python is closest to that of Linux/Unix. Running Python on Windows/Mac requires adapting for those platforms some of the kinds of tools that simplify operations on Linux/Unix systems. This discussion is essentially about how far that goes, how to accomplish it, and how to remain compatible with the existing tools on Linux/Unix. Stan Klein

4 4

Static metadata using setup.cfg
by Tarek Ziadé 18 Sep '09

18 Sep '09

Hello Following the discussion on the format of the static metadata file, it's evident for me that it has to be a ConfigParser file. Now for the name of the file, I wouldn't want yet another file since we already have setup.cfg and since it can be extended w/o any problem or bacward compatibility issue. (That's from Tres and Matthias idea at Pycon) I am also realizing that we can use it to describe "static metadata" almost with the existing distutils code. We can use the [global] section of the setup.cfg file to describe them. If we want to remove all metadata expressed as setup() arguments, we can just move them to the setup.cfg. example of setup.cfg file: """ [global] name: foo version: 1.0 author: tarek long_description: some long description here url: http://example.com """ What's left in setup.py : """ from distutils.core import setup setup() """ Right now the behavior of the code is: Distutils will take the setup.cfg options and apply them to the Distribution class, overriding any argument passed to setup(), then they will be in turn overriden by the command line options if any. This behavior seems fine. Now there's a very small change to make in distutils to make this work, wich consists of applying these values to the DistutilsMetadata object (the metadata attribute in the dist instance) I've changed this in my working trunk to give a try, and it works fine. the long_description field is expressed as a multi-line field following the config parser convention so no problem for it (see my example) Although there's another change we need to apply and decide : being able to express a list of values, for fields like "keywords" or "classifiers". It can be a "," separated list of values, or a "\n" separated one. I am proposing the current scheme (applying it in this precise order): 1/ if the value contains '\n' signs, it's a list and values are splitted using \n, then each element is left stripped (not sure yet if we want to right strip them too) 2/ if the value contains ',' signs, it's a list and values are splitted using ',' (not sure yet if we want to strip them) 3/ else, it's a value This allows us to use "," for small lists like keywords we know they will never contain any "," sign, and several lines when the values might contain "," signs. So for example: """ [global] name: foo version: 1.0 author: tarek long_description: some long description here url: http://example.com keywords: one, two, three classifiers = Classifier1 Classifier2 Classifier3 """ Any thoughts ? Cheers Tarek -- Tarek Ziadé | http://ziade.org

15 66

Distribute 0.6 released
by Tarek Ziadé 08 Sep '09

08 Sep '09

Hi all, I am pleased to announce the release of Distribute 0.6. Distribute is a friendly fork of the Setuptools project. You can get more info, download it and install it using the instructions from its PyPI page : http://pypi.python.org/pypi/distribute --- CHANGES (compared to setuptools 0.6c9) : setuptools * Packages required at build time where not fully present at install time. This closes http://bitbucket.org/tarek/distribute/issue/12. * Protected against failures in tarfile extraction. This closes http://bitbucket.org/tarek/distribute/issue/10. * Made Jython api_tests.txt doctest compatible. This closes http://bitbucket.org/tarek/distribute/issue/7. * sandbox.py replaced builtin type file with builtin function open. This closes http://bitbucket.org/tarek/distribute/issue/6. * Immediately close all file handles. This closes http://bitbucket.org/tarek/distribute/issue/3. * Added compatibility with Subversion 1.6. This references http://bitbucket.org/tarek/distribute/issue/1. pkg_resources * Avoid a call to /usr/bin/sw_vers on OSX and use the official platform API instead. Based on a patch from ronaldoussoren. This closes http://bitbucket.org/tarek/distribute/issue/5. * Fixed a SandboxViolation for mkdir that could occur in certain cases. This closes http://bitbucket.org/tarek/distribute/issue/13. * Allow to find_on_path on systems with tight permissions to fail gracefully. This closes http://bitbucket.org/tarek/distribute/issue/9. * Corrected inconsistency between documentation and code of add_entry. This closes http://bitbucket.org/tarek/distribute/issue/8. * Immediately close all file handles. This closes http://bitbucket.org/tarek/distribute/issue/3. easy_install * Immediately close all file handles. This closes http://bitbucket.org/tarek/distribute/issue/3. Cheers Tarek -- Tarek Ziadé | http://ziade.org

2 7

virtualenv and buildout integration ?
by Yonsy Solis 07 Sep '09

07 Sep '09

Hi o/ we recently began tu use virtualenv and buildout for development and deployments in some projects (more one internal effort to convince the ppl heads to use this for the next projects) by now i have two ways to work. 1) create a venv and inside this create the buildout dir project. 2) create a venv and in the same venv directory create the buildout tree. by now i see that maybe the second option is little dangerous but we dont use an custom interpreter and like we are a django shop, we use djangorecipe and do bin/django shell (shell_plus with django_extensions) to have a shell to the project. by now both options require to do two operations, first create the venv, activate this and after this create the buildout and run this. i am thinking about develop an recipe/extension for buildout to: - create a virtualenv - activate the venv - rerun the buildout using the ptyhon venv (bypassing the venv part) my questions are: 1) someone think and do this first? 2) do you consider this alternative dangerous for some operation? 3) exist any plan for buildout to obtain the same isolation level than virtualenv ? when ? 4) do you see any other alternative to obtain the same effect (we need to test apps between python 2.5 and python 2.6, we see than the buildout dont ignore the global site-package, some dev machines are really messed in their python interpreters used and the eggs/modules/installed in their systems, and we love to deploy the apps inside venvs with mod_wsgi, no more hell with pythonpath) well that's all by now ^^ -- Yonsy Solis Aureal Systems (mov) 989-124-141 (www) http://www.aureal.com.pe

8 17

[buildout] zc.recipe.egg interpretters don't support -m
by Chris Withers 02 Sep '09

02 Sep '09

With a buildout.cfg section that looks like: [py] recipe = zc.recipe.egg eggs = ... interpreter = py If I try and do: bin/py -m some.moldule ...I get: Traceback (most recent call last): File "bin/py", line 28, in <module> _options, _args = getopt.getopt(sys.argv[1:], 'ic:') File "/usr/local/lib/python2.6/getopt.py", line 91, in getopt opts, args = do_shorts(opts, args[0][1:], shortopts, args[1:]) File "/usr/local/lib/python2.6/getopt.py", line 191, in do_shorts if short_has_arg(opt, shortopts): File "/usr/local/lib/python2.6/getopt.py", line 207, in short_has_arg raise GetoptError('option -%s not recognized' % opt, opt) How should whatever generates bin/py be changed to enable -m to work? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

3 12

Regression in zc.buildout 1.4.0?
by Jonathan Ballet 28 Aug '09

28 Aug '09

Hi, We often use the 'extends' feature of Buildout to create customized version of configuration files per-developer or per-buildbot. One nice feature of Buildout was that I can easily extend another configuration file and add part to those specified in this 'base' file. Then, if I launched the Buildout, I got all the parts from the base file plus the one I defined in *my* configuration file. I was very happy. Unfortunately, it seems that the new release 1.4.0 of Buildbot has broke this feature and the new behavior seems to be: install only parts which are *strictly* defined in the configuration file used by the Buildout, and not the 'inherited' ones anymore. I wrote a quick doctest which illustrate this problem (merely adapted from other doctests): =========================================================== Create a test recipe, which only prints itself >>> mkdir('recipes') >>> write('recipes', 'testrecipe.py', ... """ ... import logging, os, zc.buildout ... ... class TestRecipe: ... ... def __init__(self, buildout, name, options): ... self.name, self.options = name, options ... ... def install(self): ... logging.getLogger(self.name).info( ... 'This is the test recipe which is installing!') ... return '' ... ... def update(self): ... logging.getLogger(self.name).info( ... 'This is the test recipe which is updateting!') ... """) And create a dummy package to use it: >>> write('recipes', 'setup.py', ... """ ... from setuptools import setup ... ... setup( ... name = "recipes", ... entry_points = {'zc.buildout': ... ['testrecipe = testrecipe:TestRecipe']}, ... ) ... """) Now, this is a base buildout configuration file, which defines and uses by default one part: >>> write('base.cfg', """\ ... [buildout] ... develop = recipes ... parts = base-part ... [base-part] ... recipe = recipes:testrecipe ... """) And another buildout configuration file, which extends the base configuration file, and add to it another part: >>> write('buildout.cfg', """\ ... [buildout] ... extends = base.cfg ... parts += buildout-part ... [buildout-part] ... recipe = recipes:testrecipe ... """) If we run the buildout, then we should install first the part from the base configuration file, then the part from the other buildout configuration file: >>> print system(buildout) Develop: '/sample-buildout/recipes' Installing base-part. base-part: This is the test recipe which is installing! Installing buildout-part. buildout-part: This is the test recipe which is installing! <BLANKLINE> =========================================================== ...which fails with latest versions of Buildout with: =========================================================== File "..." line 52, in extends.txt Failed example: print system(buildout) Expected: Develop: '/sample-buildout/recipes' Installing base-part. base-part: This is the test recipe which is installing! Installing buildout-part. buildout-part: This is the test recipe which is installing! <BLANKLINE> Got: Develop: '/sample-buildout/recipes' Installing buildout-part. buildout-part: This is the test recipe which is installing! <BLANKLINE> =========================================================== As you can see, the 'base-part' is not installed anymore. I bisected the problem down to the merge of the 'tlotze-download-api' branch (the bug didn't not occured before). I'm not sure the behavior I expect was actually really documented (I don't find it in the documentation) but this was nice! Should I report a bug for this or was it a necessary change for the new version of Buildout? Thanks! -Jonathan

2 1

Distribute split final shape
by Tarek Ziadé 26 Aug '09

26 Aug '09

Hello I've been working on bug fixes for 0.6.1, if you have some bug fixes you'd like to see urgently fixed in this release please let me know. I'll also split Distribute 0.7 sometimes today and change the names, here's the splitting I'll do (Distribution Name / Role / Content ) - Distribution Name = "distribute" - Role: the core package - Content: setuptools package renamed to distribute (minus some elements). root modules in setuptools will be moved to a "distribute.core" package so the root namespace stays empty. (the version comparison engine is included here) - Distribution Name : "distribute.installer" - Role: everything needed to install - Content: the part of pkg_resources.py that handles version comparison + package_index, that gets files at pypi + easy_install.py + easy_install command extracted from setuptools package + renamed into a "distribute.installer" package - Distribution Name = "distribute.resources" - Role: everything needed to query installed packages - Content: the part of pkg_resources.py that handles installed distribution querying + renamed to "distribute.resources" - Distribution Name = "distribute.entrypoints" - Role: the plugin system - Content: the part of pkg_resources.py that handles entry points + renamed to "distribute.entrypoints" package. I'll drop then the bootstraping part we had to have for 0.6 since the names will not conflict anymore with the Setuptools project. Cheers Tarek -- Tarek Ziadé | http://ziade.org

1 0