Mailman 3 March 2008 - Distutils-SIG

Package versioning
by Timothy Docker 29 Apr '18

29 Apr '18

Hi all, I've been aware that the distutils sig has been simmerring away, but until recently it has not been directly relevant to what I do. I like the look of the proposed api, but have one question. Will this support an installed system that has multiple versions of the same package installed simultaneously? If not, then this would seem to be a significant limitation, especially when dependencies between packages are considered. Assuming it does, then how will this be achieved? I am presently managing this with a messy arrangement of symlinks. A package is installed with its version number in it's name, and a separate directory is created for an application with links from the unversioned package name to the versioned one. Then I just set the pythonpath to this directory. A sample of what the directory looks like is shown below. I'm sure there is a better solution that this, and I'm not sure that this would work under windows anyway (does windows have symlinks?). So, has this SIG considered such versioning issues yet? Cheers, Tim -------------------------------------------------------------- Tim Docker timd(a)macquarie.com.au Quantative Applications Division Macquarie Bank -------------------------------------------------------------- qad16:qad $ ls -l lib/python/ total 110 drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1 -r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1 dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8 drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1 dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1 dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3 drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7 dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1 qad16:qad $ ls -l lib/python/1.1 total 30 lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1 lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8 lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1 lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1 lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3 lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7 lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1

3 2

[Distutils-sig] distutils charter and interscript
by John Skaller 29 Apr '18

29 Apr '18

3 2

eggs and Python Unicode variants (UCS2, UCS4)
by Martijn Faassen 29 May '09

29 May '09

Hi there, In the lxml project (http://codespeak.net/lxml), we've just noticed the following problem with lxml eggs: you can easy_install an egg that won't work for your Python. This is because Python can be compiled with either 2 or 4 bytes unicode as its internal representation. Any egg that contains compiled C code that uses unicode such as lxml will run into trouble: if it's compiled with a 4 bytes unicode Python, it won't work on a 2 bytes unicode Python, and vice versa. This problem is fairly common in Linux. Many distributions such as Ubuntu and Fedora compile their python with 4 bytes unicode internal representation. If you compile a Python interpreter by hand it defaults to 2 bytes unicode, however. Hand-building a Python interpreter is done fairly commonly by Linux sysadmins for various reasons. It would therefore be very nice if it became possible to make eggs for the different unicode compilation options of Python. This configuration dimension is a real world issue for any binary Python module that does anything with unicode text.. In an earlier mail to this list: http://mail.python.org/pipermail/distutils-sig/2005-October/005222.html M.-A. Lemburg and Phillip Eby had the following discussion: [MAL] >>Please make sure that your eggs catch all possible >>Python binary build dimensions: >> >>* Python version >>* Python Unicode variant (UCS2, UCS4) >>* OS name >>* OS version >>* Platform architecture (e.g. 32-bit vs. 64-bit) [PJE] >As far as I know, all of this except the Unicode variant is captured in >distutils' get_platform(). And if it's not, it should be, since it >affects any other kind of bdist mechanism. I'm not sure whether this means this needs to be escalated from setuptools to the Python interpreter level itself. With this mail, I've done the job escalating this lxml problem to what appears to be the right place, though. :) Thanks, Martijn

5 5

unicode bug in distutils
by Tarek Ziadé 05 Apr '09

05 Apr '09

Hello, I have created a setup.py file for distirbution and I bumped into a small bug when i tried to set my name in the contact field (Tarek Ziadé) Using string (utf8 file): setup( maintainer="Tarek Ziadé" ) leads to: File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/command/register.py", line 162, in send_metadata auth) File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/command/register.py", line 257, in post_to_server value = unicode(value).encode("utf-8") UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128) Using unicode: setup( maintainer=u"Tarek Ziadé" ) leads to: File "/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/distutils/dist.py", line 1094, in write_pkg_file file.write('Author: %s\n' % self.get_contact() ) UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 18: ordinal not in range(128) I would propose a patch for this problem but i don't know what would be the best input (i guess unicode for names) Regards Tarek -- Tarek Ziadé | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/

6 9

buildout: setting envirionment variables for zc.recipe.egg:custom?
by Christian Zagrodnick 24 Jun '08

24 Jun '08

Hi, building lxml requires additional configuration so it finds xslt-config correctly. One possibility is to set the PATH environment variable. The other is to call setup.py --xslt-config=... Using setup.py --xslt-config doesn't fit at all into buildout i think. Setting environment variables before installing an egg on the other hand might be useful. Would it make sense to put this into zc.recipe.egg or would it be better to create some unrelated recipe which only sets envirionment variables? Regards, -- Christian Zagrodnick gocept gmbh & co. kg · forsterstrasse 29 · 06112 halle/saale www.gocept.com · fon. +49 345 12298894 · fax. +49 345 12298891

3 5

Re: [Distutils] [Python-Dev] PEP 365 (Adding the pkg_resources module)
by Phillip J. Eby 15 Apr '08

15 Apr '08

At 10:53 AM 3/17/2008 -0500, Guido van Rossum wrote: >I don't think this should play games with scripts being overridden or >whatever. If a bootstrap script is to be installed it should have a >separate name. I'm not sure what the advantage is of a bootstrap >script over "python -m bootstrap_module ..." though. And -m also makes explicit: 1. that it's a Python-specific tool 2. which Python version it will apply to >The PEP suggests that other package managers also benefit. How do they >benefit if the bootstrap script installs setuptools? Because those other package managers depend, in fact, on setuptools, or at least pkg_resources... which was why the original proposal was to just include pkg_resources in the first place. :) >I'd also like to avoid the specific name "easy_install" for any of >this. That's a "brand name" (and a misleading one if you ask me, but >that's politics again :-). Ok, so if someone will propose a name and API for the thing, I'll implement it. (Assuming the proposed API is sane and reasonably implementable, of course.)

9 20

how to easily consume just the parts of eggs that are good for you
by zooko 14 Apr '08

14 Apr '08

Folks: Here is a simple proposal: make the standard Python "import" mechanism notice eggs on the PYTHONPATH and insert them (into the *same* location) on the sys.path. This eliminates the #1 problem with eggs -- that they don't easily work when installing them into places other than your site-packages and that if you allow any of them to be installed on your system then they take precedence over your non-egg packages even you explicitly put those other packages earlier in your PYTHONPATH. (That latter behavior is very disagreeable to more than a few prorgammers.) This also preserves most of the value of eggs for many use cases. This is backward-compatible with most current use cases that rely on eggs. This is very likely forward-compatible with new schemes that are currently being cooked up and will be deployed in the future. Regards, Zooko

9 11

A Modest Proposal for "A Database of Installed Packages"
by Alexander Michael 09 Apr '08

09 Apr '08

I'll continue my fool hearty effort [1] to build a concrete proposal for "a database of installed packages" by offering up a sketch of a possible straw-man "solution". I realize that this is likely oversimplified to a fault, but I hope it will help us move forward. Apologies if the equivalent of this has been proposed and rejected before. My proposal is basically to make PKG-INFO functional and usable by: * Fixing the technical issues with requirements (i.e. dependencies) and naming as specified by PEP 314/345. * Modifying distutils to install PKG-INFO alongside each module file or package directory as a side-car file of the same name but with a special extension (.pyi or whatever). These files would be the place to include the optional list of installed files as well as the optional md5sums, if desired by the installer. Files in the package will be listed using relative paths, while far flung files (bin, shared, etc) will get full paths so that there is full allowance for relocating simple (nothing in bin or shared) modules and packages. Although optional, "python setup.py install" will include the installed file list by default. That's it. The intent is to provide just enough information to allow the development of tools to use it, for those that are interested, while being minimally invasive to developers that are not interested in such tools. To determine the current state of your python environment, walk sys.path looking for modules and packages, collecting PKG-INFO when available. No standard centralized database. Some of us will choose to opt-in to a particular installation management tool that might maintain a cache (centralized or per-directory) for efficiency, but that would be considered a performance optimization for that particular tool. We can also bootstrap older python installations by creating an online database (that can of course be downloaded by security conscious individuals for offline querying) that maps (module-file/package-directory name, md5sum) pairs to their respective PKG-INFO contents (no list of installed files) which can be queried by an automated sys.path walker to fill-in missing side-car files. Thus, I can opt-in to this scheme for python 2.5 by installing a distutils patch and meta-data side-car bootstrapper that does its best to identify what's on my sys.path. It would be quite tractable to maintain this for the python standard library and perhaps the official installations of a few major OS versions. Such a database could even be used for the community to provide metadata for packages that the developer didn't (again, furthering an opt-in mentality). Of course, even though it worked for CDDB, it would likely be too much to expect this level of coverage through user submitted entries. [1] http://mail.python.org/pipermail/distutils-sig/2008-March/009108.html

3 7

setuptools bug report: _get_unpatched() is way too clever (and also returns the wrong module)
by zooko 05 Apr '08

05 Apr '08

Bug report #1: _get_unpatched() returns a module from the system site-packages instead of the module from the PYTHONPATH which ought to have precedence, leading to setuptools-0.6c8 invoking setuptools-0.6a9 in the following stack trace: PYTHONPATH="/home/buildslave/slave-tahoe/dapper/build/support/lib/ python2.4/site-packages" python ./setup.py develop --prefix="/home/ buildslave/slave-tahoe/dapper/build/support" ... Running Nevow-0.9.18/setup.py -q bdist_egg --dist-dir /tmp/ easy_install-rk83KQ/Nevow-0.9.18/egg-dist-tmp-nJ8GSy Traceback (most recent call last): ... File "/home/buildslave/slave-tahoe/dapper/build/ setuptools-0.6c8.egg/setuptools/command/develop.py", line 102, in install_for_development File "/usr/lib/python2.4/site-packages/setuptools-0.6a9-py2.4.egg/ easy_install.py", line 518, in process_distribution Full details: https://dev.allmydata.com/buildbot-tahoe/builders/dapper/builds/1427/ steps/compile/logs/stdio Bug report #2: _get_unpatched() is way too clever for me to spend time trying to understand and debug right now: def _get_unpatched(cls): """Protect against re-patching the distutils if reloaded Also ensures that no other distutils extension monkeypatched the distutils first. """ What the heck? I don't want to think about that, so instead we're delisting Ubuntu Dapper as a supported platform, since it is the only one of our supported platforms on which we have this problem. Generalized complaint: There are a lot of things that are too clever in setuptools/ easy_install/eggs. Cleverness makes it work in more situations, which is good, but it also means that it is harder to know when it will fail or to understand why it failed. A simpler system that satisfied 90% of the use cases while failing in more obvious ways would be appreciated. (Provided, of course, that those 90% of use cases include the ones that I really care about.) See my earlier post which proposes such a simplification: http://mail.python.org/pipermail/python-dev/2008-March/078243.html Bug report #3: setuptools/easy_install/eggs needs a bug tracker. (Also, of course, a buildbot and a rich suite of automated unit tests.) Some of this message may have been inappropriate for distutils-sig, but what else am I going to do with this issue before I delist Dapper and forget all about it? I would be willing to host a trac instance to serve as a wiki and issue tracker for setuptools, and a buildbot. (But I don't want to have anything to do with subversion -- I much prefer darcs.) Regards, Zooko

4 9

FW: [issue2513] 64bit cross compilation on windows
by Mark Hammond 03 Apr '08

03 Apr '08

FYI, I've uploaded a patch that provides for cross-compilation on Windows between 32 and 64 bit platforms - all comments invited! Thanks, Mark -----Original Message----- From: Mark Hammond [mailto:report@bugs.python.org] Sent: Sunday, 30 March 2008 6:01 PM To: mhammond(a)users.sourceforge.net Subject: [issue2513] 64bit cross compilation on windows New submission from Mark Hammond <mhammond(a)users.sourceforge.net>: I've taken the liberty of adding Trent, Christian and Martin to the nosy list as I know they are actively, if reluctantly interested in this. This patch allows the distutils to cross-compile on Windows. It has been tested on x86 and amd64 platforms, with both platforms successfully able to natively and cross-compile extension modules and create binary distributions. To cross-compile, specify '--plat-name' to the build command (valid values are 'win32', 'win-amd64' and 'win-ia64'). This option name was chosen to be consistent with the bdist_dumb command. I've included the docs I added below (which are also in the patch), but note that as with native compilation using distutils, it's not necessary to set any environment variables or do anything else special with your environment to make this work. The patch also adds a x64 target for the 'bdist_wininst' target, which it creates as distutils/command/wininst-9.0-amd64.exe. This executable is necessary even for bdist_wininst to work natively on x64, but is still included here for simplicity. To assist with testing, I've also added a distutils setup.py script to the PC/example_nt directory. This is capable of creating bdist_wininst executables for both native and cross platforms; 'setup.py build --platname=win-amd64 bdist_wininst' will create an amd64 installer on an x86 machine. The patch has not been tested with a Visual Studio environment without cross-compile tools installed - it will obviously fail, but its not clear how ugly this failure will be. Below is the text I added to docs/distutils/builtdist.rst: Cross-compiling on Windows ===================== Starting with Python 2.6, distutils is capable of cross-compiling between Windows platforms. In practice, this means that with the correct tools installed, you can use a 32bit version of Windows to create 64bit extensions and vice-versa. To build for an alternate platform, specify the :option:`--plat-name` option to the build command. Valid values are currently 'win32', 'win-amd64' and 'win-ia64'. For example, on a 32bit version of Windows, you could execute:: python setup.py build --plat-name=win-amd64 to build a 64bit version of your extension. The Windows Installers also support this option, so the command:: python setup.py build --plat-name=win-amd64 bdist_wininst would create a 64bit installation executable on your 32bit version of Windows. Note that by default, Visual Studio 2008 does not install 64bit compilers or tools. You may need to reexecute the Visual Studio setup process and select these tools. ---------- assignee: mhammond components: Distutils files: windows-cross-compile.patch keywords: 64bit, patch messages: 64743 nosy: Trent.Nelson, ctheune, loewis, mhammond severity: normal status: open title: 64bit cross compilation on windows type: feature request versions: Python 2.6 Added file: http://bugs.python.org/file9900/windows-cross-compile.patch __________________________________ Tracker <report(a)bugs.python.org> <http://bugs.python.org/issue2513> __________________________________

3 7