I've been aware that the distutils sig has been simmerring away, but
until recently it has not been directly relevant to what I do.
I like the look of the proposed api, but have one question. Will this
support an installed system that has multiple versions of the same
package installed simultaneously? If not, then this would seem to be a
significant limitation, especially when dependencies between packages
Assuming it does, then how will this be achieved? I am presently
managing this with a messy arrangement of symlinks. A package is
installed with its version number in it's name, and a separate
directory is created for an application with links from the
unversioned package name to the versioned one. Then I just set the
pythonpath to this directory.
A sample of what the directory looks like is shown below.
I'm sure there is a better solution that this, and I'm not sure that
this would work under windows anyway (does windows have symlinks?).
So, has this SIG considered such versioning issues yet?
Tim Docker timd(a)macquarie.com.au
Quantative Applications Division
qad16:qad $ ls -l lib/python/
drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1
-r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so
drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1
drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1
dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8
drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1
dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1
dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3
drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7
dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1
qad16:qad $ ls -l lib/python/1.1
lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so
lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so
lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so
lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so
lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so
lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so
lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1
lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8
lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1
lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1
lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so
lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so
lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3
lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7
lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1
In the lxml project (http://codespeak.net/lxml), we've just noticed the
following problem with lxml eggs: you can easy_install an egg that won't
work for your Python.
This is because Python can be compiled with either 2 or 4 bytes unicode
as its internal representation. Any egg that contains compiled C code
that uses unicode such as lxml will run into trouble: if it's compiled
with a 4 bytes unicode Python, it won't work on a 2 bytes unicode
Python, and vice versa.
This problem is fairly common in Linux. Many distributions such as
Ubuntu and Fedora compile their python with 4 bytes unicode internal
representation. If you compile a Python interpreter by hand it defaults
to 2 bytes unicode, however. Hand-building a Python interpreter is done
fairly commonly by Linux sysadmins for various reasons.
It would therefore be very nice if it became possible to make eggs for
the different unicode compilation options of Python. This configuration
dimension is a real world issue for any binary Python module that does
anything with unicode text..
In an earlier mail to this list:
M.-A. Lemburg and Phillip Eby had the following discussion:
>>Please make sure that your eggs catch all possible
>>Python binary build dimensions:
>>* Python version
>>* Python Unicode variant (UCS2, UCS4)
>>* OS name
>>* OS version
>>* Platform architecture (e.g. 32-bit vs. 64-bit)
>As far as I know, all of this except the Unicode variant is captured in
>distutils' get_platform(). And if it's not, it should be, since it
>affects any other kind of bdist mechanism.
I'm not sure whether this means this needs to be escalated from
setuptools to the Python interpreter level itself. With this mail, I've
done the job escalating this lxml problem to what appears to be the
right place, though. :)
I have created a setup.py file for distirbution and I bumped into
a small bug when i tried to set my name in the contact field (Tarek Ziadé)
Using string (utf8 file):
line 162, in send_metadata
line 257, in post_to_server
value = unicode(value).encode("utf-8")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10:
ordinal not in range(128)
line 1094, in write_pkg_file
file.write('Author: %s\n' % self.get_contact() )
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position
18: ordinal not in range(128)
I would propose a patch for this problem but i don't know what would be the
best input (i guess unicode
Tarek Ziadé | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
building lxml requires additional configuration so it finds xslt-config
correctly. One possibility is to set the PATH environment variable. The
other is to call setup.py --xslt-config=...
Using setup.py --xslt-config doesn't fit at all into buildout i think.
Setting environment variables before installing an egg on the other
hand might be useful. Would it make sense to put this into
zc.recipe.egg or would it be better to create some unrelated recipe
which only sets envirionment variables?
gocept gmbh & co. kg · forsterstrasse 29 · 06112 halle/saale
www.gocept.com · fon. +49 345 12298894 · fax. +49 345 12298891
I hope the cross-post is appropriate.
I've started playing with getting the pywin32 extensions building under
the AMD64 architecture. I started building with Visual Studio 8 (it was
what I had handy) and I struck a few issues relating to the compiler version
that I thought worth sharing.
* In trying to build x64 from a 32bit VS7 (ie, cross-compiling via the
PCBuild directory), the python.exe project fails with:
pythoncore fatal error LNK1112: module machine type 'X86' conflicts with
target machine type 'AMD64'
is this a known issue, or am I doing something wrong?
* The PCBuild8 project files appear to work without modification (I only
tried native compilation here though, not a cross-compile) - however, unlike
the PCBuild directory, they place all binaries in a 'PCBuild8/x64'
directory. While this means that its possible to build for multiple
architectures from the same source tree, it makes life harder for tools like
'distutils' - eg, distutils already knows about the 'PCBuild' directory, but
it knows nothing about either PCBuild8 or PCBuild8/x64.
A number of other build processes also know to look inside a PCBuild
directory (eg, Mozilla), so instead of formalizing PCBuild8, I think we
should merge PCBuild8 into PCBuild. This could mean PCBuild/vs7 and
PCBuild/vs8 directories with the "project" files, but binaries would still
be generated in the 'PCBuild' (or PCBuild/x64) directory. This would mean
the same tree isn't capable of hosting 2 builds from different VS compilers,
but I think that is reasonable (if it's a problem, just have multiple source
directories). I understand that PCBuild8 is not "official", but in the
assumption that future versions of Python will use a compiler later than
VS7, it makes sense to me to clean this up now - what are others opinions on
* Re the x64 directory used by the PCBuild8 process. IMO, it makes sense to
generate x64 binaries to their own directory - my expectation is that
cross-compiling between platforms is a reasonable use-case, and we should
support multiple achitectures for the same compiler version. This would
mean formalizing the x64 directory in both 'PCBuild' and distutils, and
leaving other external build processes to update as they support x64 builds.
Does this make sense? Would this fatally break other scripts used for
packaging (eg, the MSI framework)?
* Wide characters in VS8: PC/pyconfig.h defines PY_UNICODE_TYPE as 'unsigned
short', which corresponds with both 'WCHAR' and 'wchar' in previous compiler
versions. VS8 defines this as wchar_t, which I'm struggling to find a
formal definition for beyond being 2 bytes. My problem is that code which
assumes a 'Py_UNICODE *' could be used in place of a 'WCHAR *' now fails. I
believe the intent on Windows has always been "PyUNICODE == 'native
unicode'" - should PC/pyconfig.h reflect this (ie, should pyconfig.h grow a
version specific definition of PyUNICODE as wchar_t)?
* Finally, as something from left-field which may well take 12 months or
more to pull off - but would there be any interest to moving the Windows
build process to a cygwin environment based on the existing autoconf
scripts? I know a couple of projects are doing this successfully, including
Mozilla, so it has precendent. It does impose a greater burden on people
trying to build on Windows, but I'd suggest that in recent times, many
people who are likely to want to build Python on Windows are already likely
to have a cygwin environment. Simpler mingw builds and nuking MSVC specific
build stuff are among the advantages this would bring. It is not worth
adding this as "yet another windows build option" - so IMO it is only worth
progressing with if it became the "blessed" build process for windows - if
there is support for this, I'll work on it as the opportunity presents
I'm (obviously) only suggesting we do this on the trunk and am happy to make
all agreed changes - but I welcome all suggestions or critisisms of this
The pywin32 extensions require (well, prefer) administrative access during
installation - certain files are copied to the System32 directory and the
registry at HKEY_LOCAL_MACHINE is written to. Also, if I understand
correctly, if Python happened to be installed into "\Program Files", admin
access would be required to create any files in that directory tree - I'm
not sure what permissions the \PythonXX directory are created with, but its
not unreasable to assume that some shops might choose to secure that
directory similarly to "\Program Files".
The simplest way to achieve this for bdist_wininst installations is to
include some magic in a "manifest". I've confirmed that once this magic is
added, programs created by bdist_wininst get the little "shield" icon
overlay and prompt for elevation before starting the executable. A problem
here is that not all installations will require admin access - eg, a user
who installed Python just for themselves will not need elevation to install
an extension. A solution here would be for the installer to *not* be marked
as requiring elevation, then sniffing the registry to make an educated guess
(eg, HKLM\Software\Python\PythonCore\2.5 existing could indicate admin
access is required). If it finds elevation is required, it would spawn
another copy of itself with elevation requested, and terminate. This will
have a side-effect of meaning the installer never gets the "shield" overlay,
meaning the user will not be expecting to be prompted - but that is
something we can live with.
However, there is a complication here. Any "pure python" packages are not
tied to a particular Python version, so the user can choose what installed
Python version to use. Hence, in the general case, we can only determine if
admin is required after the UI has opened and the user has selected the
Python version. Arranging for the new "elevated" child process at this
point will be (a) ugly, as the UI will vanish before the child process
displays its GUI and (b) would require additional command-line processing
logic - ie, passing the target directory to the child process. If we could
make the determination *before* the GUI opens, it would appear seamless and
would not require special command-line handling (the child process just does
So I see a few alternatives, but none are very desirable:
* Only make this "admin required" check if a specific version of Python is
necessary (ie, the package contains extension modules). This would leave
pure-python packages out in the cold.
* Live with the ugly UI that would result in performing that check after the
Python version has been selected, and add the command-line processing
necessary to make this work.
* Ignore the issue and try to educate people that they must explicitly use
"Run as Administrator" for such packages on Vista.
I'm wondering if anyone has any opinions or thoughts on how we should handle
The setuptools plugins architecture may eventually supplant Makefiles
in my projects. It's very nice that people can re-use other people's
development hacks as packages containing setuptools plugins instead
of by cutting and pasting snippets of code into their Makefiles. It
would be nice if I could find a listing of such plugins that I might
want to use in my own development.
Here is a list of setuptools plugins that I just found by browsing
the Python Package Index:
setuptools_bzr, setuptools_darcs, setuptools_mtn, setuptools_qt4,
darcsver, eggchecker, iw.releaser, apydia, bbfreeze (oh I wish I had
known about bbfreeze a month ago!), py2app, mkcode, eggtestinfo,
easydeb, buildutils, hg.setuptools, setuptools-git
Here is one that I wish someone would convert into a plugin (it runs
unit tests using Twisted's "trial" unit test framework):
So my question is: how can we make it easier for people to find
these? I was unaware of most of these until just now and there are
probably other useful plugins that I'm still unaware of.
One idea that I had was to run a query on the Python Package Index
database to find all packages whose setup.py's define entry points in
the "distutils.commands" group name.
Another was to put the word out (i.e. in the setuptools
documentation) that if your package is a setuptools plugin, that you
should name your package "setuptools_blah_blah" and register it on
the Python Package Index.
P.S. Here is the trac ticket where the allmydata.org project is
discussing moving more of our build infrastructure into setuptools
http://allmydata.org/trac/tahoe/ticket/179 -- "rewrite our Makefile
in Python instead of GNUmake"
With Subversion 1.5 entering alpha, I've been using it for more than
just a few projects, and I've noticed that setuptools fails to run in
a 1.5 working copy. For the short term, I've found that I can just
modify the setuptools source to trust that version 9 WCs will work the
same as version 8 WCs. The diff for that is attached.
Based on what I've read of subversion's README on the format of the
entries file, the modification I've made should be just fine - the
overall structure of the file is the same, there's just some new tags
possible which will already not get stuck in the regex already in
place. In my local testing, using setuptools for installing files
works just fine with this patch in place.
In preparing for a tutorial on Python eggs, I'm wrestling with the *precise*
definition of a "Python egg". It appears to be a slippery term from the
"There are several binary formats that embody eggs, but the most
common is '.egg' zipfile format, because it's a convenient one
for distributing projects. All of the formats support including
package-specific data, project-wide metadata, C extensions, and
Ok, so what binary formats -other than- .egg files are actually eggs? I
figure RPMs, .debs, etc. are eggs, as long as they have an EGG-INFO
subdirectory and conform to certain basic structural rules.
"Eggs are pluggable distributions in one of the three formats
currently supported by ``pkg_resources``. There are built
eggs, development eggs, and egg links."
Hmmm, so it's not just binary distributions that can be eggs, but also
egg-links and development eggs. The concept of a "built egg" egg still
includes an RPM or .deb though.
"Python Eggs are the preferred binary distribution format for
EasyInstall, because they are cross-platform (for "pure"
packages), directly importable, and contain project metadata
including scripts and information about the project's
dependencies. They can be simply downloaded and added to
sys.path directly, or they can be placed in a directory on
sys.path and then automatically discovered by the egg
Well drat, so RPMs and .debs are -not- eggs, because you don't just place them
on sys.path directly to use them. Egg-links don't quite fit that definition
either, unless you look at them funny. So what -other- binary format could
that first paragraph be talking about that can be dropped onto sys.path and be
picked up by Python?
Python eggs are so very zen; you can only understand them by already