Distutils-SIG October 2006

distutils-sig@python.org

42 participants
76 discussions

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

eggs and Python Unicode variants (UCS2, UCS4)
by Martijn Faassen 29 May '09

29 May '09

Hi there, In the lxml project (http://codespeak.net/lxml), we've just noticed the following problem with lxml eggs: you can easy_install an egg that won't work for your Python. This is because Python can be compiled with either 2 or 4 bytes unicode as its internal representation. Any egg that contains compiled C code that uses unicode such as lxml will run into trouble: if it's compiled with a 4 bytes unicode Python, it won't work on a 2 bytes unicode Python, and vice versa. This problem is fairly common in Linux. Many distributions such as Ubuntu and Fedora compile their python with 4 bytes unicode internal representation. If you compile a Python interpreter by hand it defaults to 2 bytes unicode, however. Hand-building a Python interpreter is done fairly commonly by Linux sysadmins for various reasons. It would therefore be very nice if it became possible to make eggs for the different unicode compilation options of Python. This configuration dimension is a real world issue for any binary Python module that does anything with unicode text.. In an earlier mail to this list: http://mail.python.org/pipermail/distutils-sig/2005-October/005222.html M.-A. Lemburg and Phillip Eby had the following discussion: [MAL] >>Please make sure that your eggs catch all possible >>Python binary build dimensions: >> >>* Python version >>* Python Unicode variant (UCS2, UCS4) >>* OS name >>* OS version >>* Platform architecture (e.g. 32-bit vs. 64-bit) [PJE] >As far as I know, all of this except the Unicode variant is captured in >distutils' get_platform(). And if it's not, it should be, since it >affects any other kind of bdist mechanism. I'm not sure whether this means this needs to be escalated from setuptools to the Python interpreter level itself. With this mail, I've done the job escalating this lxml problem to what appears to be the right place, though. :) Thanks, Martijn

5 5

bdist_rpm fails due to bytecompiled modules
by Christopher Blunck 05 Dec '06

05 Dec '06

hello, i have a very basic pure-python package and script that i would like to distribute as a noarch rpm using distutils. i am using FC5 as my development platform and have prepared a setup.py that properly builds and installs my application. the first problem i have is that the bdist_rpm command results in an %install section as follows: %install python setup.py install --single-version-externally-managed --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES the trouble here with that %install section is that the python modules get bytecompiled and thus turned into native code. i'd prefer for that not to happen, so i have written my own install.sh script and i provide that to my python setup.py call as follows: $ python setup.py bdist_rpm --spec-only --install-script /path/to/install.sh my install.sh passes the --optimize 0 and --no-compile arguments to distutils, resulting in no .pyc or .pyo files being installed in my RPM_BUILD_ROOT. the contents of my install.sh are as follows: python setup.py install --single-version-externally-managed --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES --optimize 0 --no-compile i thought that would result in no byte compiled files but unfortunately i was wrong. i'm not sure what's causing it, but rpmbuild ends up calling brp-python-bytecompile to compile the .py files. i noticed in the ChangeLog for rpm the following: - force *.py->*.pyo byte code compilation with brp-python-bytecompile. so it appears to me as tho there is no way to assemble a noarch RPM using distutils. does that sound accurate to you all? or are those folks out there using distutils and an old version of rpm (prior to 2005) that are able to assemble noarch RPMs? -c -- WeatherNet Observations for station: home Temperature: 54.80F Pressure: 30.02in; Dew Point: 30.76F (40%) Wind: 248 at 5 mph Recorded: 17:54:59 10/25/06 (http://wsdl.wxnet.org/inquiry/binding.wsdl)

5 8

Setuptools with custom path configuration
by Jean-Paul Calderone 30 Oct '06

30 Oct '06

I revisited setuptools again tonight hoping to learn a bit more about it. Unfortunately I'm still stuck where I was last time I looked. I'm using Combinator to manage my Python libraries. This means that I have a fair amount of software installed at ~/.local/lib/pythonX.Y/site-packages, a PYTHONPATH which includes only ~/Projects/Divmod/trunk/Combinator, at which location resides a sitecustomize.py which sets up the rest of my path. >From the documentation, and various experimentation, it seems like this configuration is still not supported. What needs to happen for to change this? Jean-Paul

2 7

IBM Developerworks setuptools article - example doesnt work ?
by Michael Bayer 30 Oct '06

30 Oct '06

I just read an IBM Developerworks article on setuptools today, its linked at: http://www-128.ibm.com/developerworks/linux/library/l-cppeak3.html? ca=dgr-lnxw07PythonEggWithSetuptools it refers to the ability to use pkg_resources.require() in a Python script to specify any particular version of a package that happens to exist in site-packages, overriding the currently active package...something i used to think setuptools could do, but then figured it couldnt (even though it seems like obviously useful functionality), and it seems that it still cannot. from the article's example: # install Gnosis_Utils (latest version is 1.2.1) easy_install -f http://gnosis.cx/download/Gnosis_Utils.More/ Gnosis_Utils # install version 1.2.0 of Gnosis_Utils easy_install -f http://gnosis.cx/download/Gnosis_Utils.More/ "Gnosis_Utils==1.2.0" # re-activate version 1.2.1 easy_install "Gnosis_Utils==1.2.1" # run a script that requires version 1.2.0 from pkg_resources import require require("Gnosis_Utils==1.2.0") Traceback (most recent call last): File "<stdin>", line 1, in ? File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6c3-py2.4.egg/pkg_resources.py", line 585, in require needed = self.resolve(parse_requirements(requirements)) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/ python2.4/site-packages/setuptools-0.6c3-py2.4.egg/pkg_resources.py", line 487, in resolve raise VersionConflict(dist,req) # XXX put more info here pkg_resources.VersionConflict: (Gnosis-Utils 1.2.1 (/Library/ Frameworks/Python.framework/Versions/2.4/lib/python2.4/site-packages/ Gnosis_Utils-1.2.1-py2.4.egg), Requirement.parse('Gnosis-Utils==1.2.0')) bug ? or feature ?

3 6

Import module failure with an egg(sometimes)
by John Green 27 Oct '06

27 Oct '06

Would appreciate some help with where to start trying to find out what's going on here. We're using eggs as the format for extensions to our app and I'm having the odd bit of strangeness on one machine. I have another with identical software setup which behaves as expected. Here is a snippet of our code: pkg_resources.working_set.add_entry(fullName) dist_generator = pkg_resources.find_distributions(fullName) for dist in dist_generator: try: extension_class = dist.load_entry_point("jokosher.extensions", "extension") extension = extension_class() print("...done.") except Exception, e : print("...failed.") print(str(e)) and usually it works. Now because can run with an experimental version of the gstreamer libraries we can run a script before we run our program which sets various ENV variables (including PYTHONPATH) to point us to the development libraries instead of the stock ones. If I run with the stock setup everything is OK. On one machine (and one machine only) if I run the setup script in order to pick up the experimental libraries then I get ImportError. This is a section of the output by running the program with 'python -vv'. First - with the stock libs (no pre-run script): importing extension... AddInstrumentType-0.1-py2.4.egg # zipimport: found 10 names in /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg # zipimport: zlib available # trying /home/john/code/jokosher/trunk/Jokosher/AddInstrumentType.so # trying /home/john/code/jokosher/trunk/Jokosher/AddInstrumentTypemodule.so # trying /home/john/code/jokosher/trunk/Jokosher/AddInstrumentType.py # trying /home/john/code/jokosher/trunk/Jokosher/AddInstrumentType.pyc # trying /usr/lib/python2.4/site-packages/gst-0.10/AddInstrumentType.so # trying /usr/lib/python2.4/site-packages/gst-0.10/AddInstrumentTypemodule.so # trying /usr/lib/python2.4/site-packages/gst-0.10/AddInstrumentType.py # trying /usr/lib/python2.4/site-packages/gst-0.10/AddInstrumentType.pyc # trying /home/john/code/jokosher/trunk/Jokosher/AddInstrumentType.so # trying /home/john/code/jokosher/trunk/Jokosher/AddInstrumentTypemodule.so # trying /home/john/code/jokosher/trunk/Jokosher/AddInstrumentType.py # trying /home/john/code/jokosher/trunk/Jokosher/AddInstrumentType.pyc # trying /usr/lib/python24.zip/AddInstrumentType.so ... ... ... # trying /home/john/code/jokosher/trunk/AddInstrumentType.so # trying /home/john/code/jokosher/trunk/AddInstrumentTypemodule.so # trying /home/john/code/jokosher/trunk/AddInstrumentType.py # trying /home/john/code/jokosher/trunk/AddInstrumentType.pyc # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/__init __.pyc # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/AddIns trumentType/__init__.pyc # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/AddIns trumentType/__init__.pyo # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/AddIns trumentType/__init__.py # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/AddIns trumentType.pyc # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/Jokosh er.so # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/Jokosh ermodule.so # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/Jokosh er.py # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/Jokosh er.pyc # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/gtk.so # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/gtkmod ule.so # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/gtk.py # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/gtk.py c # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/os.so # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/osmodu le.so # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/os.py # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/os.pyc # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/pkg_re sources.so # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/pkg_re sourcesmodule.so # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/pkg_re sources.py # trying /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg/AddInstrumentType/pkg_re sources.pyc import AddInstrumentType.AddInstrumentType # loaded from Zip /home/john/.jokosher/extensions/AddI nstrumentType-0.1-py2.4.egg/AddInstrumentType/AddInstrumentType.pyc import AddInstrumentType # loaded from Zip /home/john/.jokosher/extensions/AddInstrumentType-0.1- py2.4.egg/AddInstrumentType/__init__.pyc ...done. No problems there. But after running the env-altering script all we get out is: importing extension... AddInstrumentType-0.1-py2.4.egg # zipimport: found 10 names in /home/john/.jokosher/extensions/AddInstrumentType-0.1-py2.4.egg # zipimport: zlib available ...failed. No module named AddInstrumentType It doesn't even seem to begin to look for the module. As I said this only happens on one machine and I know that it used to work on that one too. If you could give us a clue on what to look for next then that would be appreciated. Thanks -- John Green

2 1

How to get list of package requirements from PyPI without downloading egg?
by Tim Cera 26 Oct '06

26 Oct '06

Hello, I want the list of package requirements, which I hoped would be real simple, but have not been able to find a good solution. I could download the egg and look in the EGG_INFO directory, but is there any way to have setuptools develop the list for me? And hopefully not have to download the egg. If the above idea doesn't go anywhere - is it possible that the package requirements could be added to CheeseShop's XMLRPC? Kindest regards, Tim Cera

3 13

permission / umask on install?
by Georg 26 Oct '06

26 Oct '06

Hi, I was somewhat surprised to find that distutils on install does honor the current umask. If run as root and root's umask is 027 (a common case nowadays), stuff will be installed as non world readable / executable. Compare this to install(1)'s behavior on Unix. Shouldn't the default be to DTRT on Unix? Or shouldn't there be at least an option to specify the target permissions for data and executable files? -- Regards, Georg.

1 0

disappointed by setupTools
by Pierre Imbaud 25 Oct '06

25 Oct '06

Im a little bit disappointed by setupTools. Not that much the tool itself, than the learning curve. Maybe the problem at hand is far from simple, and thats what makes the solution so intricate. http://peak.telecommunity.com/DevCenter/setuptools: this document is 100 pages long! (by page I mean: full screen). easyInstall is 40 pages. I spent about 3 days to figure out what goes where. And I realize Im not done. Yet the documentation is remarkable, great presentation, text is precise and clear, when U read it. Makes U feel guilty U dont find what U lookin for, but U dont find it. I just discovered scripts dont get installed directly (in /usr/local/bin), instead a wrapper in installed, that calls the script. So an extra file is involved, and extra code, every time the script is called. Makes some overhead, and it breaks a simple mechanism I use: my script is able to perform many actions, based on its name (much like cp, mv and ln are the same executable). at install, I need to make symlinks to the script, one per action, and this was automated: this is broken by the wrapping. No big deal, just makes things less simple. Good tools have 2 qualities: - When U need simple things, its easy to use. - When U need more sophisticated things, U dont need to give up the tool for a more powerful one: just have to use extra features. Python is a wonderful illustration of this. While setupTools obviously provides the latter, I consider it fails on the former. I decided to use setupTools after reading, and failing to understand, distutils. setupTools seemed both more powerful and simpler. Probably this choice will save me some trouble later. Im not sure it helped so far.

4 5

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

Distutils-SIG October 2006