Mailman 3 September 2012 - Distutils-SIG

Two fixes (at last)
by Greg Ward April 29, 2018

April 29, 2018

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker April 29, 2018

April 29, 2018

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller April 29, 2018

April 29, 2018

3 2

error in windows 7 installation
by Manuel López May 13, 2014

May 13, 2014

Hello Everybody: I am trying to install distribute-0.6.25 in a windows 7 machine. I have python 2.7.3 in 32 bits, although the machine is 64 bits. I installed the 32 bit python version because I want to install ipython and there is no 64 bit builds of the Windows installer for ipython. In any case, upon installing disuitils with python setup.py install I get the following error error: C:\manuel\prog_red\python\distribute-0.6.25\distribute-0.6.25\setuptools\cli-32.exe: No such file or directory Any help will be greatly appeciated. Thanks, Manuel -- Manuel López Mariscal Depto. de Oceanografía Física/CICESE malope(a)cicese.mx

2 1

Some clarifications and/or corrections to PEP 376
by Paul Moore Jan. 7, 2013

Jan. 7, 2013

I've been looking at the details of PEP 376 (Database of Installed Python Distributions) and there are a couple of aspects that I don't think work properly alongside PEP 370 (Per-user site-packages directory).. 1. The dist-info directory for a distribution is stated as being "located in the site-packages directory". It's not clear how this is intended to work in a PEP 370 world with multiple site-packages. What I propose is that the description be changed to be worded in terms of sysconfig-style locations: the dist-info directory is located in whichever of purelib or platlib is used by the distribution. When the distribution uses both, purelib is preferred, when it uses neither (!) purelib is used. In nearly all cases, this is the same as currently. The exceptions are posix_home (where the directory name isn't "site-packages" but its function is the same), posix_prefix (where purelib and platlib differ, and PEP 376 is currently ambiguous as to which is implied), and any custom schemes that might be created (where PEP 376 is silent, and this proposal has the benefit of at least being specific). I do not believe this changes any actual practice - as far as I have been able to determine any code using dist-info at the moment follows this proposal in the corner cases where it differs from PEP 376. 2. File paths in RECORD are stated as being relative to the "base location" as long as the files sit under that location, or under the "install prefix" where that is above the base location. That's a messy but precise way of saying you can go up a bit as long as you stay within sys.prefix. (Other paths are platform format absolute paths, which is an ugly discrepancy but the only really viable option). Again, this doesn't work as I'd expect it should in a PEP 370 user site-packages. In this case, the answer isn't quite as clear. I would suggest, looking at the schemes defined in sysconfig, allowing any files located in one of the defined sysconfig paths (for the current scheme) to be recorded as relative paths. The paths should be relative to whichever of purelib and platlib was chosen in (1) above. The exception is that where pureXXX and platXXX do not use the same base (posix_prefix and possibly custom schemas) it should not be allowed to have a relative path from a pureXXX to a platXXX or vice versa. That equates to something slightly stricter than the current scheme (although not in a way that's likely to be used in practice), but using the appropriate one of {base} or {userbase} rather than sys.prefix. As noted, the two exceptions are custom schemes (where PEP 376 is silent and would probably end up effectively mandating platform absolute paths throughout) and posix_prefix, where platlib/platinclude/platstdlib are located under {platbase} rather than {base} and the current proposal would probably allow relative paths to be used a little more often that PEP 376 allows. Current practice that I've seen here is not as clear cut, but mostly in the sense that code doesn't consider the corner cases at all, and probably only follows PEP 376 by luck, if at all... Comments would be appreciated. I'm pretty happy that my proposal for (1) is an improvement. With (2) I'm concerned about the posix_prefix case, particularly as I'm a Windows developer and my understanding of Python's layout on POSIX is limited. But I honestly don't think it's possible to do much better than my proposal, and I do think that PEP 376 as it stands now is distinctly worse because it doesn't consider PEP 370 or the posix_prefix case. Also, while these proposals are not 100% backward compatible, I don't believe they change behaviour in the most common scenarios, and given that PEP 376 currently has limited adoption (mainly in distutils2, I believe) I'd suggest that the minor compatibility break now is better than keeping the current system until it's more widely adopted and there are more users to impact. Obviously I'd be looking for views from Tarek and Éric in particular here, as it's their code in distutils2 that would be directly impacted. Paul.

4 10

distribute buildbot
by Adam GROSZER Oct. 22, 2012

Oct. 22, 2012

Hello, Are there any buildbots for distribute? -- Best regards, Adam GROSZER -- Quote of the day: Men do not reject the Bible because it contradicts itself but because it contradicts them. - The Defender

5 4

distlib updated with resources API
by Vinay Sajip Oct. 4, 2012

Oct. 4, 2012

I've updated distlib[1] with a resources API - functionality for accessing data files co-located with code in Python packages. This is missing from the stdlib and currently people use pkg_resources to achieve this. The design and implementation allows for accessing resources from packages imported from the file system or from .zip files, and is intended to be extensible to other PEP 302-compliant import systems. The update includes: A short tutorial showing how to use the API. [2] A discussion of the design of the API. [3] Reference documentation for the API. [4] I'd be interested in any and all feedback on this - be it on design, code, docs. Please regard it as very much a first draft - it would appear to cover pretty much the same functionality as the corresponding part of pkg_resources [5]. Regards, Vinay Sajip [1] https://bitbucket.org/vinay.sajip/distlib/ [2] http://distlib.readthedocs.org/en/latest/tutorial.html#using-the-resource-a… [3] http://distlib.readthedocs.org/en/latest/internals.html#the-resources-api [4] http://distlib.readthedocs.org/en/latest/reference.html#the-distlib-resourc… [5] http://packages.python.org/distribute/pkg_resources.html#basic-resource-acc…

9 61

Struggling with a modular buildout configuration
by Lele Gaifax Oct. 2, 2012

Oct. 2, 2012

Hi all, I'm trying to setup a modular buildout configuration, but I'm hitting my head on an invisible wall... Basically, I'd like to have a top-level buildout.cfg that "collects" settings from various "subprojects"; in particular, each subproject may "augment" a set of eggs, some of which may be defined as "develop" (in buildout parlance). Finally, with a single "zc.recipe.egg" I wanna install a custom interpreter with all those eggs inside. So, I have the top-level configuration that uses "extends" to include a "projects.cfg": [buildout] extends = base.cfg projects.cfg that in turn brings in the various subprojects snippets thru "extends": [buildout] extends = src/this-project.cfg src/that-project.cfg these snippets typically contain a section like [projects] eggs += this-project-egg some-other-dependency-egg develop += src/this-project parts += this-project-specific-part [this-project-specific-part] ... The problem I'm having is that for incomprensible (to me, of course :) reason some of those settings do not "emerge" at the top level (others do!), as if a later "extends" is clobbering a previous one. I built a minimalistic example [1] that shows the unexpected behaviour; doing a "buildout annotate" I get something like the following: Annotated sections ================== [buildout] ... develop= ${projects:develop} /tmp/tb/buildout.cfg parts= ${projects:parts} /tmp/tb/buildout.cfg ... [extra-parts-from-proj2] [projects] develop= src/proj2 += /tmp/tb/src/proj2/devel.cfg eggs= extra-needed-by-proj1 proj1 proj2 += /tmp/tb/src/proj1/devel.cfg += /tmp/tb/src/proj1/base.cfg += /tmp/tb/src/proj2/base.cfg parts= extra-parts-from-proj2 += /tmp/tb/src/proj2/devel.cfg [python] eggs= ${projects:eggs} /tmp/tb/base.cfg interpreter= python /tmp/tb/base.cfg recipe= zc.recipe.egg /tmp/tb/base.cfg If you look at the [projects] section, you can see that while the "eggs" setting carries the three expected values (one for each subproject plus one extra), the "develop" setting contains just a single value, the one injected by proj2, and the one that should be coming from proj1 is missing. If I omit "src/proj2" from the extends option, then the "develop" setting correctly contains just "src/proj1"... Of course I tried hard to understand what's going, even walking into buildout 1.5.2 code with pdb. I tried also with the latest development version (the upcoming 1.6) as well as the github branch (IIUC it derivates from 1.4), but all exhibit the same behaviour. The 1.6 code is slightly different, and at first it seemed doing the right thing, but at a closer look it really moved the problem from the "develop" setting "eggs". As far as I can tell, there is something strange happening in the _update_section() function when the _open() function handles the "extends" setting: at some point it "loses" the cognition that a certain assignment is an expansion (tecnically, that the key is actually "develop +", instead of "develop"), so the final "s1.update(s2)" overrides it, instead of augmenting it... Of course I may be doing the wrong thing or missing the obvious, and if so I'll be happy if you can shed some light on me. Thanks in advance for any help and sorry for the long post, ciao, lele. [1] http://artiemestieri.tn.it/~lele/test-buildout.tar.gz -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. lele(a)metapensiero.it | -- Fortunato Depero, 1929.

3 6

Differences in PEP386 and setuptools
by Donald Stufft Oct. 1, 2012

Oct. 1, 2012

I've been comparing how PEP386/distutils2.version treats versions and how pkg_resources from setuptools treats versions and it confirmed a worry of me with the way that dev is treated in PEP386. In PEP386 you have several kinds of "top level" releases, these are alpha, beta, candidate, and final and they are sorted as you would expect. On top of that it has pre and post releases for any of those top level releases. The pre releases are tagged with a ".dev" and post releases are tagged with a ".post". They are sorted immediately before/after the "main" release that they are pre/post for. In pkg_resources dev is treated as it's own "main" release and sorts it in front of an alpha. This means that given the versions: ["0.1.dev1", "0.1a1", "0.1b1", "0.1c1", "0.1"] PEP386 will sort them as: ["0.1a1", "0.1b1", "0.1c1", "0.1.dev1", "0.1"] and pkg_resources will sort them as: ["0.1.dev1", "0.1a1", "0.1b1", "0.1c1", "0.1"] To further complicate things the most common usage I've personally seen in the wild is that of "0.1dev" or "0.1dev1" which the author expects to sort before an alpha (In this case distutils2.version throws an error, but the suggest function is able to turn it into 0.1.dev1). I think this difference is going to cause confusion, especially during the the transition period when you're going to have people using both pkg_resources and the new PEP386 functions. Since PEP386 is only in the "Accepted" stage, and isn't part of the official implementation yet, is it at all possible to revise it? Ideally I think to follow with the prior art, and people's expectations "dev" should be moved to a "main" release type sorted before an alpha, and to take it's place as a pre release modifier perhaps something like "pre" can be used instead (e.g. "0.1.pre1").

9 24

the 'wheel' binary package format
by Daniel Holth Sept. 30, 2012

Sept. 30, 2012

I got tired of waiting for lxml to compile over and over again, so I invented a binary packaging format called 'wheel' (of cheese) that uses Metadata 1.2 and .dist-info directories instead of .egg-info, patched pkg_resources.py to be able to load .dist-info directories, implemented "python setup.py bdist_wheel", and patched pip to be able to install .whl files. The gist of the spec is that it is a zip file with the .whl extension containing the contents of 'purelib' for a distribution, plus a Name-1.0.dist-info/ directory with the metadata files, plus Name-1.0.data/subdir directories for scripts, platlib, packaging's "categories", ... My specification so far is at https://docs.google.com/document/d/1mWPyvoeiqCrAy4UPNnvaz7Cgrqm4s_cfaTauAeJ… and an lxml compiled for linux-x86-64 is at https://docs.google.com/open?id=0BxHz5bC4iN5TN0VWTFNrZGtCbWs http://bitbucket.org/dholth/distribute http://github.com/dholth/pip http://pypi.python.org/pypi/wheel Perhaps it will be useful. The implementation is still pretty rough, for example it does not check the architecture while installing, but it could be a handy way to speed up repeated virtualenv builds. Daniel Holth

5 21