I've been aware that the distutils sig has been simmerring away, but
until recently it has not been directly relevant to what I do.
I like the look of the proposed api, but have one question. Will this
support an installed system that has multiple versions of the same
package installed simultaneously? If not, then this would seem to be a
significant limitation, especially when dependencies between packages
Assuming it does, then how will this be achieved? I am presently
managing this with a messy arrangement of symlinks. A package is
installed with its version number in it's name, and a separate
directory is created for an application with links from the
unversioned package name to the versioned one. Then I just set the
pythonpath to this directory.
A sample of what the directory looks like is shown below.
I'm sure there is a better solution that this, and I'm not sure that
this would work under windows anyway (does windows have symlinks?).
So, has this SIG considered such versioning issues yet?
Tim Docker timd(a)macquarie.com.au
Quantative Applications Division
qad16:qad $ ls -l lib/python/
drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1
-r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so
drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1
drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1
dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8
drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1
dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1
dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3
drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7
dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1
qad16:qad $ ls -l lib/python/1.1
lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so
lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so
lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so
lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so
lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so
lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so
lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1
lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8
lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1
lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1
lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so
lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so
lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3
lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7
lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1
In the lxml project (http://codespeak.net/lxml), we've just noticed the
following problem with lxml eggs: you can easy_install an egg that won't
work for your Python.
This is because Python can be compiled with either 2 or 4 bytes unicode
as its internal representation. Any egg that contains compiled C code
that uses unicode such as lxml will run into trouble: if it's compiled
with a 4 bytes unicode Python, it won't work on a 2 bytes unicode
Python, and vice versa.
This problem is fairly common in Linux. Many distributions such as
Ubuntu and Fedora compile their python with 4 bytes unicode internal
representation. If you compile a Python interpreter by hand it defaults
to 2 bytes unicode, however. Hand-building a Python interpreter is done
fairly commonly by Linux sysadmins for various reasons.
It would therefore be very nice if it became possible to make eggs for
the different unicode compilation options of Python. This configuration
dimension is a real world issue for any binary Python module that does
anything with unicode text..
In an earlier mail to this list:
M.-A. Lemburg and Phillip Eby had the following discussion:
>>Please make sure that your eggs catch all possible
>>Python binary build dimensions:
>>* Python version
>>* Python Unicode variant (UCS2, UCS4)
>>* OS name
>>* OS version
>>* Platform architecture (e.g. 32-bit vs. 64-bit)
>As far as I know, all of this except the Unicode variant is captured in
>distutils' get_platform(). And if it's not, it should be, since it
>affects any other kind of bdist mechanism.
I'm not sure whether this means this needs to be escalated from
setuptools to the Python interpreter level itself. With this mail, I've
done the job escalating this lxml problem to what appears to be the
right place, though. :)
I have created a setup.py file for distirbution and I bumped into
a small bug when i tried to set my name in the contact field (Tarek Ziadé)
Using string (utf8 file):
line 162, in send_metadata
line 257, in post_to_server
value = unicode(value).encode("utf-8")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10:
ordinal not in range(128)
line 1094, in write_pkg_file
file.write('Author: %s\n' % self.get_contact() )
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position
18: ordinal not in range(128)
I would propose a patch for this problem but i don't know what would be the
best input (i guess unicode
Tarek Ziadé | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
I hope the cross-post is appropriate.
I've started playing with getting the pywin32 extensions building under
the AMD64 architecture. I started building with Visual Studio 8 (it was
what I had handy) and I struck a few issues relating to the compiler version
that I thought worth sharing.
* In trying to build x64 from a 32bit VS7 (ie, cross-compiling via the
PCBuild directory), the python.exe project fails with:
pythoncore fatal error LNK1112: module machine type 'X86' conflicts with
target machine type 'AMD64'
is this a known issue, or am I doing something wrong?
* The PCBuild8 project files appear to work without modification (I only
tried native compilation here though, not a cross-compile) - however, unlike
the PCBuild directory, they place all binaries in a 'PCBuild8/x64'
directory. While this means that its possible to build for multiple
architectures from the same source tree, it makes life harder for tools like
'distutils' - eg, distutils already knows about the 'PCBuild' directory, but
it knows nothing about either PCBuild8 or PCBuild8/x64.
A number of other build processes also know to look inside a PCBuild
directory (eg, Mozilla), so instead of formalizing PCBuild8, I think we
should merge PCBuild8 into PCBuild. This could mean PCBuild/vs7 and
PCBuild/vs8 directories with the "project" files, but binaries would still
be generated in the 'PCBuild' (or PCBuild/x64) directory. This would mean
the same tree isn't capable of hosting 2 builds from different VS compilers,
but I think that is reasonable (if it's a problem, just have multiple source
directories). I understand that PCBuild8 is not "official", but in the
assumption that future versions of Python will use a compiler later than
VS7, it makes sense to me to clean this up now - what are others opinions on
* Re the x64 directory used by the PCBuild8 process. IMO, it makes sense to
generate x64 binaries to their own directory - my expectation is that
cross-compiling between platforms is a reasonable use-case, and we should
support multiple achitectures for the same compiler version. This would
mean formalizing the x64 directory in both 'PCBuild' and distutils, and
leaving other external build processes to update as they support x64 builds.
Does this make sense? Would this fatally break other scripts used for
packaging (eg, the MSI framework)?
* Wide characters in VS8: PC/pyconfig.h defines PY_UNICODE_TYPE as 'unsigned
short', which corresponds with both 'WCHAR' and 'wchar' in previous compiler
versions. VS8 defines this as wchar_t, which I'm struggling to find a
formal definition for beyond being 2 bytes. My problem is that code which
assumes a 'Py_UNICODE *' could be used in place of a 'WCHAR *' now fails. I
believe the intent on Windows has always been "PyUNICODE == 'native
unicode'" - should PC/pyconfig.h reflect this (ie, should pyconfig.h grow a
version specific definition of PyUNICODE as wchar_t)?
* Finally, as something from left-field which may well take 12 months or
more to pull off - but would there be any interest to moving the Windows
build process to a cygwin environment based on the existing autoconf
scripts? I know a couple of projects are doing this successfully, including
Mozilla, so it has precendent. It does impose a greater burden on people
trying to build on Windows, but I'd suggest that in recent times, many
people who are likely to want to build Python on Windows are already likely
to have a cygwin environment. Simpler mingw builds and nuking MSVC specific
build stuff are among the advantages this would bring. It is not worth
adding this as "yet another windows build option" - so IMO it is only worth
progressing with if it became the "blessed" build process for windows - if
there is support for this, I'll work on it as the opportunity presents
I'm (obviously) only suggesting we do this on the trunk and am happy to make
all agreed changes - but I welcome all suggestions or critisisms of this
The pywin32 extensions require (well, prefer) administrative access during
installation - certain files are copied to the System32 directory and the
registry at HKEY_LOCAL_MACHINE is written to. Also, if I understand
correctly, if Python happened to be installed into "\Program Files", admin
access would be required to create any files in that directory tree - I'm
not sure what permissions the \PythonXX directory are created with, but its
not unreasable to assume that some shops might choose to secure that
directory similarly to "\Program Files".
The simplest way to achieve this for bdist_wininst installations is to
include some magic in a "manifest". I've confirmed that once this magic is
added, programs created by bdist_wininst get the little "shield" icon
overlay and prompt for elevation before starting the executable. A problem
here is that not all installations will require admin access - eg, a user
who installed Python just for themselves will not need elevation to install
an extension. A solution here would be for the installer to *not* be marked
as requiring elevation, then sniffing the registry to make an educated guess
(eg, HKLM\Software\Python\PythonCore\2.5 existing could indicate admin
access is required). If it finds elevation is required, it would spawn
another copy of itself with elevation requested, and terminate. This will
have a side-effect of meaning the installer never gets the "shield" overlay,
meaning the user will not be expecting to be prompted - but that is
something we can live with.
However, there is a complication here. Any "pure python" packages are not
tied to a particular Python version, so the user can choose what installed
Python version to use. Hence, in the general case, we can only determine if
admin is required after the UI has opened and the user has selected the
Python version. Arranging for the new "elevated" child process at this
point will be (a) ugly, as the UI will vanish before the child process
displays its GUI and (b) would require additional command-line processing
logic - ie, passing the target directory to the child process. If we could
make the determination *before* the GUI opens, it would appear seamless and
would not require special command-line handling (the child process just does
So I see a few alternatives, but none are very desirable:
* Only make this "admin required" check if a specific version of Python is
necessary (ie, the package contains extension modules). This would leave
pure-python packages out in the cold.
* Live with the ugly UI that would result in performing that check after the
Python version has been selected, and add the command-line processing
necessary to make this work.
* Ignore the issue and try to educate people that they must explicitly use
"Run as Administrator" for such packages on Vista.
I'm wondering if anyone has any opinions or thoughts on how we should handle
I'm using zc,buildout to manage the development of a very simple web
application. I'd like to have a simple test runner that runs my
doctests, but have run into a slight problem. I'm using
wsgi_intercept to aid in testing, but unfortunately wsgi_intercept
isn't a proper Python package ("yet", according to Titus). So I have
a package-i-fied version in my svn repository and in my buildout.cfg I
develop = . ./wsgi_intercept_
I initially was using the "interpreter" option to get a Python
interpreter with my eggs on the path, and running "./bin/python
setup.py test". That, however, required me to list wsgi_intercept as
a package requirement, when I really want to list it as a testing
requirement. Moving it to a testing requirement caused it's develop
egg not to be on the PYTHONPATH, so setuptools tries to find it.
Which it couldn't.
I tried using zc.recipe.testbrowser, thinking that maybe it'd look at
the tests_require for the target eggs, but no such luck. The
testrunner also seems pretty promiscuous in looking for things to test
(it tries to import my eggs directory and test them which predictably
doesn't work), but that's another story.
So... any suggestions on using zc.buildout along with testing
dependencies (generally or with zc.recipe.testrunner)?
I've been trying to build rpms of enthought system components. Some of
them use namespace packages. Those packages have the required __init__.py
According to the Setuptools Guide, these __init__.py files are not to be
packaged when using "system" packaging (such as bdist_rpm) and when I run
bdist_rpm they are not included in the rpms. However, there are egg-info
directories and nspkg.pth files produced, included in the rpms, and
installed in site-packages. When I tried to run the enthought example
applications, I had various kinds of import failures. I found that fixing
them required putting the unpackaged __init__.py files into the relevant
directories in the namespace packages.
The various rpms were being placed in the proper locations where they
should have been found for importing. There were some strange errors,
such as in one case when I tried to test the imports interactively an
"import a.b.c" worked, but an "import a.b.c as c" failed.
Before putting in the __init__.py files, I tried removing the egg-info and
nspkg.pth files from site_packages. That did not improve things. I also
noticed that some other packages had pth files in site-packages, so that
may be what is needed if the egg-info and nspkg.pth are removed, although
I don't know why or what the pth files are supposed to contain.
I think setuptools is somehow not properly processing bdist_rpm if there
are namespace packages involved. Also, the setuptools don't seem to have
an option to not create the egg info and nspkg.pth files and/or to not
package them if bdist_rpm is being used.
At 12:56 AM 7/23/2007 -0400, Fred Drake wrote:
>On 7/22/07, Phillip J. Eby <pje(a)telecommunity.com> wrote:
>>Setuptools has lots of features that are targeted at different
>>audiences. There are plenty of features targeted at the group you're
>>in, don't begrudge the other groups their features. :)
>Actually, I suspect this is a substantial contributor to setuptools
>being considered controversial: it encompasses to many different
That's precisely my point, though: "too many" always means "has
features I don't use", even though the features one person uses are
considered superfluous by other parties, who consider the first
person's "superfluous" features essential.
However, I'm entirely reconciled to it being controversial. It is in
fact impossible to have a significant impact on *anything* without
creating controversy, as anyone involved with the history of Zope
should be well aware. ;-)
Usually, people think they can get away with making "simpler"
versions of Zope because they don't understand the full range of
requirements which it is intended to meet. The situation with
setuptools is much the same, except that sometimes now it's the Zope
folks making the accusations of too many features, excess complexity,
evil, etc. :)
Robert Kern wrote:
Stanley A. Klein wrote:
>> I tried to do something to fix the numpy distutils bdist_rpm.py (by
>> trying to follow what was done in install.py) but it didn't work and I
>> got an error message I didn't understand.
>I'd like to help, but if you don't copy the exact error message here, I
OK, here is my revised numpy/distutils/commands/bdist_rpm.py (trying --
obviously not well -- to follow what was done in
if 'setuptools' in sys.modules:
import setuptools.command.bdist_rpm as old_bdist_rpm
from distutils.command.bdist_rpm import bdist_rpm as old_bdist_rpm
spec_file = old_bdist_rpm._make_spec_file(self)
# Replace hardcoded setup.py script name
# with the real setup script name.
setup_py = os.path.basename(sys.argv)
if setup_py == 'setup.py':
new_spec_file = 
for line in spec_file:
line = line.replace('setup.py',setup_py)
And here is the error message I got (using the kiva setup.py):
[stan@localhost enthought.kiva_2.0]$ python setup.py bdist_rpm
Traceback (most recent call last):
File "setup.py", line 2, in ?
from numpy.distutils.core import setup
File "/usr/lib/python2.4/site-packages/numpy/distutils/core.py", line
32, in ? from numpy.distutils.command import bdist_rpm
line 6, in ?
TypeError: Error when calling the metaclass bases
module.__init__() takes at most 2 arguments (3 given)
BTW, below is what the unpackaged files error looks like (with a traceback
generated by having the DISTUTILS_DEBUG environment variable set). In a
much earlier thread I learned that this can usually be resolved by setting
[install] optimize=1 in setup.cfg.
Installed (but unpackaged) file(s) found:
Traceback (most recent call last):
File "setup.py", line 108, in ?
version = '2.0b1',
File "/usr/lib/python2.4/site-packages/numpy/distutils/core.py", line
174, in setup
File "/usr/lib/python2.4/distutils/core.py", line 149, in setup
File "/usr/lib/python2.4/distutils/dist.py", line 946, in run_commands
File "/usr/lib/python2.4/distutils/dist.py", line 966, in run_command
File "/usr/lib/python2.4/distutils/command/bdist_rpm.py", line 365, in run
File "/usr/lib/python2.4/distutils/cmd.py", line 398, in spawn
spawn(cmd, search_path, dry_run= self.dry_run)
File "/usr/lib/python2.4/distutils/spawn.py", line 37, in spawn
_spawn_posix(cmd, search_path, dry_run=dry_run)
File "/usr/lib/python2.4/distutils/spawn.py", line 165, in _spawn_posix
raise DistutilsExecError, \
distutils.errors.DistutilsExecError: command 'rpmbuild' failed with exit