Mailman 3 February 2013 - Distutils-SIG

Two fixes (at last)
by Greg Ward 29 Apr '18

29 Apr '18

Hi all -- at long last, I have fixed two problems that a couple people noticed a while ago: * I folded in Amos Latteier's NT patches almost verbatim -- just changed an `os.path.sep == "/"' to `os.name == "posix"' and added some comments bitching about the inadequacy of the current library installation model (I think this is Python's fault, but for now Distutils is slavishly aping the situation in Python 1.5.x) * I fixed the problem whereby running "setup.py install" without doing anything else caused a crash (because 'build' hadn't yet been run). Now, the 'install' command automatically runs 'build' before doing anything; to make this bearable, I added a 'have_run' dictionary to the Distribution class to keep track of which commands have been run. So now not only are command classes singletons, but their 'run' method can only be invoked once -- both restrictions enforced by Distribution. The code is checked into CVS, or you can download a snapshot at http://www.python.org/sigs/distutils-sig/distutils-19990607.tar.gz Hope someone (Amos?) can try the new version under NT. Any takers for Mac OS? BTW, all parties involved in the Great "Where Do We Install Stuff?" Debate should take a good, hard look at the 'set_final_options()' method of the Install class in distutils/install.py; this is where all the policy decisions about where to install files are made. Currently it apes the Python 1.5 situation as closely as I could figure it out. Obviously, this is subject to change -- I just don't know to *what* it will change! Greg -- Greg Ward - software developer gward(a)cnri.reston.va.us Corporation for National Research Initiatives 1895 Preston White Drive voice: +1-703-620-8990 Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913

2 1

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

error in windows 7 installation
by Manuel López 13 May '14

13 May '14

Hello Everybody: I am trying to install distribute-0.6.25 in a windows 7 machine. I have python 2.7.3 in 32 bits, although the machine is 64 bits. I installed the 32 bit python version because I want to install ipython and there is no 64 bit builds of the Windows installer for ipython. In any case, upon installing disuitils with python setup.py install I get the following error error: C:\manuel\prog_red\python\distribute-0.6.25\distribute-0.6.25\setuptools\cli-32.exe: No such file or directory Any help will be greatly appeciated. Thanks, Manuel -- Manuel López Mariscal Depto. de Oceanografía Física/CICESE malope(a)cicese.mx

2 1

Packaging & Distribution Mini-Summit at PyCon US
by Nick Coghlan 14 Mar '13

14 Mar '13

As folks may be aware, I am moderating a panel called "Directions in Packaging" on the Saturday afternoon at PyCon US. Before that though, I am also organising what I am calling a "Packaging & Distribution Mini-Summit" as an open space on the Friday night (we have one of the larger open space rooms reserved, so we should have a fair bit of space if a decent crowd turns up). An overview of what I'm hoping we can achieve at the session is at https://us.pycon.org/2013/community/openspaces/packaginganddistributionmini… (that page should be editable by anyone that has registered for PyCon US). We're certainly not going to resolve all our disagreements and come up with a grand plan to "fix Python packaging" in a couple of hours on a Friday night. My hopes are far more modest: that we can get an idea of the different things people are worried about and trying to resolve, and start pondering ways we can work towards an cooperative ecosystem of interoperable tools, rather than the "one tool to rule them all" model of distutils. I fully expect that discussions on the Friday night will continue as hallway track discussions during the conference, development efforts at the sprints, and, of course, requests for feedback on the mailing lists (since there will likely be quite a few interested people that won't be present at PyCon US). This is not a new conversation - it is one that has been going on for years, and while some improvements have been made, we still have a long way to go. For those that are able to make it, I look forward to meeting you in person in March :) Regards, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

4 4

distribute-0.6.35 fails on python 3.3?
by Chris Withers 06 Mar '13

06 Mar '13

Hi All, Can anyone else reproduce this? I have a Python 3.3 virtualenv and I wanted to upgrade one of the packages as follows: $ bin/pip install -U manuel Requirement already up-to-date: manuel in ./lib/python3.3/site-packages Downloading/unpacking distribute from http://pypi.python.org/packages/source/d/distribute/distribute-0.6.35.tar.g… (from manuel) Downloading distribute-0.6.35.tar.gz (644kB): 644kB downloaded Running setup.py egg_info for package distribute Traceback (most recent call last): File "<string>", line 3, in <module> File "./setuptools/__init__.py", line 2, in <module> from setuptools.extension import Extension, Library File "./setuptools/extension.py", line 5, in <module> from setuptools.dist import _get_unpatched File "./setuptools/dist.py", line 103 except ValueError, e: ^ SyntaxError: invalid syntax Complete output from command python setup.py egg_info: Traceback (most recent call last): File "<string>", line 3, in <module> File "./setuptools/__init__.py", line 2, in <module> from setuptools.extension import Extension, Library File "./setuptools/extension.py", line 5, in <module> from setuptools.dist import _get_unpatched File "./setuptools/dist.py", line 103 except ValueError, e: ^ SyntaxError: invalid syntax Can other people reproduce this? When was that Python 3 incompatible except clause introduced? cheers. Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

9 12

Library instability on PyPI and impact on OpenStack
by Mark McLoughlin 05 Mar '13

05 Mar '13

Hey, (I sent this to the wrong list and was directed here by Nick. I wasn't aware of the very promising sounding PEP426 and haven't read it yet, so apologies on that. Just wanted to resend my mail here ASAP to prevent the discussion happening on the wrong list. Thanks!) Generally speaking, when a project has a large list of dependencies on libraries outside of its control, it can take one of two approaches to those dependencies: 1) specify the minimum required version of each library and assume new releases of all your dependencies will be backwards compatible with previous versions. You feel safe that if an incompatible version of the library is released, it will be a completely new stream and you can adopt the new stream at your leisure. I'm much more familiar with C libraries than any other language. Somehow, C library maintainers seem to understand the need for this approach and so you've got mechanisms like libtool managed sonames and parallel installable libraries like gtk, gtk2, gtk3. 2) specify exactly what version of each library to use, because you assume all of your dependencies are constantly changing their APIs and breaking your app This is what you see in the Ruby (bundler) and Java (maven) worlds. For distribution packagers, it presents horrific problems - either you accept that you're going to be packaging (potentially many) different versions of the same library and that any time a security issue comes up you need to patch each version. Personally, I tend to pour scorn on Ruby and Java folks for the chaotic nature that pushes app developers down this "I need to control the whole stack of dependencies because they're so unstable" path. OTOH, I hear the Perl community are pretty good about taking the first approach. I always felt that the Python community tended more towards the former approach, but there always exceptions to the rule - to unfairly pick one one project, sqlalchemy seems to have an API that often changes incompatibly. However, OpenStack is starting to get burned more often and some are advocating taking the second approach to managing our dependencies: http://lists.openstack.org/pipermail/openstack-dev/2013-February/thread.htm… http://lists.openstack.org/pipermail/openstack-dev/2013-February/thread.htm… http://lists.openstack.org/pipermail/openstack-dev/2012-November/002075.html It's probably not worthwhile for everyone to try and read the nuances of those threads. The tl;dr is we're hurting and hurting bad. Is this a problem the OpenStack and Python communities want to solve together? Or does the Python community fundamentally seem themselves as taking the same approach as the Ruby and Java communities? Maybe it sounds like I'm trolling, but this is an honest question. What I'd really like to hear back is "please, please OpenStack keep using the first approach and we can work through the issues you're seeing and together make Python better". We can totally do that IMHO. If people want an example of the kind of stuff things we need to work on: http://lists.openstack.org/pipermail/openstack-dev/2013-February/006048.html Thanks, Mark.

9 33

buildout and build-time dependencies
by Ben Acland 01 Mar '13

01 Mar '13

Hey folks, <butt_covering> I've googled around, hit IRC, and tried a bunch of stuff to fix this. Even so, maybe I missed something obvious. Sorry if this question is redundant. </butt_covering> I'm using buildout to install a series of python modules, each of which is dependent on the others *at build time.* For example, numpy, then scipy. If I try to build scipy without numpy around, the scipy build fails complaining about not being able to find "numpy.distutils...". I can include the full text of the error here, but get the feeling that you either know what I'm talking about or not by this point. There doesn't seem to be a way to do this cleanly using buildout. Instead, I have had to put .tar.gz versions of each of the libraries in my repo along with bootstrap.py, and am installing them using a step like this: [numpy] recipe = collective.recipe.cmd cmds = mkdir tmp_numpy cd tmp_numpy tar -xzvf ${buildout:packages}/numpy-1.7.0.tar.gz ${buildout:directory}/bin/buildout setup numpy-1.7.0 install cd ../ rm -rf tmp_numpy on_install = true on_update = true ... obviously, this won't fly if I don't have sudo access, so I'm having to run buildout inside of a virtualenv. Blah blah blah, I can go into the gory details if you want, but the bottom line things that I dislike about this are: 1) I have to include the packages in the repo. 2) I'm going to have to teach my labmates how to use virtualenv AND buildout. 3) After I install them this way, I have to install them again as follows in order to get a linked interpreter: [pyscripts] recipe = zc.recipe.egg:scripts dependent-scripts = true eggs = numpy scipy networkx pydicom traits wsgiref nibabel nipype SO. Please, tell me that I'm stupid and that I've missed something obvious. tl;dr: how to handle build time dependencies between python modules using buildout, without looking stupid or including .tar.gz files in my repo. Thanks for your time and good work, Ben

3 2

time for packaging summit at pycon
by Doug Hellmann 28 Feb '13

28 Feb '13

The Packaging Summit page [1] lists "Friday evening" but doesn't specify a time or location, as far as I can tell. Have those been set, yet? Doug [1] https://us.pycon.org/2013/community/openspaces/packaginganddistributionmini…

4 4

Setuptools/Distribute - using from a "non-installed" location
by Paul Moore 28 Feb '13

28 Feb '13

For some scripts I am writing, I want to be able to run a package's setup.py using subprocess.check_call(). However, the python installation I'm using may not have setuptools/distribute installed, and yet I want to correctly process setup.py files which rely on setuptools. Is there any way of setting things up so that the Python subprocess can use a local, non-installed, copy of setuptools or distribute for the setup.py run? I've tried setting PYTHONPATH, but that doesn't seem to work - probably because directories on PYTHONPATH do not have .pth files processed. To clarify, what I'm doing is: here = os.path.abspath(os.path.dirname(__file__)) # setuptools is available in os.path.join(here, 'setuptools') # setuptools.pth and easy-install.pth are in here os.environ[PYTHONPATH] = here # this doesn't work setup_py = <the setup.py file> setup_dir = os.path.abspath(os.path.dirname(setup_py)) subprocess.check_call([sys.executable, 'setup.py', 'install'] + special_install_options, cwd=setup_dir) The point of this is that I want to write a script to build wheels from sdists, but I don't want to require the user to install setuptools just for the build step. Does anyone have any suggestions? Paul

3 5