Hi all --
at long last, I have fixed two problems that a couple people noticed a
* I folded in Amos Latteier's NT patches almost verbatim -- just
changed an `os.path.sep == "/"' to `os.name == "posix"' and added
some comments bitching about the inadequacy of the current library
installation model (I think this is Python's fault, but for now
Distutils is slavishly aping the situation in Python 1.5.x)
* I fixed the problem whereby running "setup.py install" without
doing anything else caused a crash (because 'build' hadn't yet
been run). Now, the 'install' command automatically runs 'build'
before doing anything; to make this bearable, I added a 'have_run'
dictionary to the Distribution class to keep track of which commands
have been run. So now not only are command classes singletons,
but their 'run' method can only be invoked once -- both restrictions
enforced by Distribution.
The code is checked into CVS, or you can download a snapshot at
Hope someone (Amos?) can try the new version under NT. Any takers for
BTW, all parties involved in the Great "Where Do We Install Stuff?"
Debate should take a good, hard look at the 'set_final_options()' method
of the Install class in distutils/install.py; this is where all the
policy decisions about where to install files are made. Currently it
apes the Python 1.5 situation as closely as I could figure it out.
Obviously, this is subject to change -- I just don't know to *what* it
Greg Ward - software developer gward(a)cnri.reston.va.us
Corporation for National Research Initiatives
1895 Preston White Drive voice: +1-703-620-8990
Reston, Virginia, USA 20191-5434 fax: +1-703-620-0913
I've been aware that the distutils sig has been simmerring away, but
until recently it has not been directly relevant to what I do.
I like the look of the proposed api, but have one question. Will this
support an installed system that has multiple versions of the same
package installed simultaneously? If not, then this would seem to be a
significant limitation, especially when dependencies between packages
Assuming it does, then how will this be achieved? I am presently
managing this with a messy arrangement of symlinks. A package is
installed with its version number in it's name, and a separate
directory is created for an application with links from the
unversioned package name to the versioned one. Then I just set the
pythonpath to this directory.
A sample of what the directory looks like is shown below.
I'm sure there is a better solution that this, and I'm not sure that
this would work under windows anyway (does windows have symlinks?).
So, has this SIG considered such versioning issues yet?
Tim Docker timd(a)macquarie.com.au
Quantative Applications Division
qad16:qad $ ls -l lib/python/
drwxr-xr-x 2 mts mts 512 Nov 11 11:23 1.1
-r--r----- 1 root mts 45172 Sep 1 1998 cdrmodule_0_7_1.so
drwxr-xr-x 2 mts mts 512 Sep 1 1998 chart_1_1
drwxr-xr-x 3 mts mts 512 Sep 1 1998 Fnorb_0_7_1
dr-xr-x--- 3 mts mts 512 Nov 11 11:21 Fnorb_0_8
drwxr-xr-x 3 mts mts 1536 Mar 3 12:45 mts_1_1
dr-xr-x--- 7 mts mts 512 Nov 11 11:22 OpenGL_1_5_1
dr-xr-x--- 2 mts mts 1024 Nov 11 11:23 PIL_0_3
drwxr-xr-x 3 mts mts 512 Sep 1 1998 Pmw_0_7
dr-xr-x--- 2 mts mts 512 Nov 11 11:21 v3d_1_1
qad16:qad $ ls -l lib/python/1.1
lrwxrwxrwx 1 root other 29 Apr 10 10:43 _glumodule.so -> ../OpenGL_1_5_1/_glumodule.so
lrwxrwxrwx 1 root other 30 Apr 10 10:43 _glutmodule.so -> ../OpenGL_1_5_1/_glutmodule.so
lrwxrwxrwx 1 root other 22 Apr 10 10:43 _imaging.so -> ../PIL_0_3/_imaging.so
lrwxrwxrwx 1 root other 36 Apr 10 10:43 _opengl_nummodule.so -> ../OpenGL_1_5_1/_opengl_nummodule.so
lrwxrwxrwx 1 root other 27 Apr 10 10:43 _tkinter.so -> ../OpenGL_1_5_1/_tkinter.so
lrwxrwxrwx 1 mts mts 21 Apr 10 10:43 cdrmodule.so -> ../cdrmodule_0_7_1.so
lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 chart -> ../chart_1_1
lrwxrwxrwx 1 root other 12 Apr 10 10:43 Fnorb -> ../Fnorb_0_8
lrwxrwxrwx 1 mts mts 12 Apr 10 10:43 mts -> ../mts_1_1
lrwxrwxrwx 1 root other 15 Apr 10 10:43 OpenGL -> ../OpenGL_1_5_1
lrwxrwxrwx 1 root other 33 Apr 10 10:43 opengltrmodule.so -> ../OpenGL_1_5_1/opengltrmodule.so
lrwxrwxrwx 1 root other 33 Apr 10 10:43 openglutil_num.so -> ../OpenGL_1_5_1/openglutil_num.so
lrwxrwxrwx 1 root other 10 Apr 10 10:43 PIL -> ../PIL_0_3
lrwxrwxrwx 1 mts mts 10 Apr 10 10:43 Pmw -> ../Pmw_0_7
lrwxrwxrwx 1 root other 10 Apr 10 10:43 v3d -> ../v3d_1_1
I'm wondering what the state of play is with the following branches:
What more needs to happen for these to get merged to trunk and a release
I was wondering if there is a way in distutils to
include/indicate a program icon that will be used in the program
menus / desktop shortcuts.
I'm actualy packaging for debian with `stdeb` and I was
expecting some stdeb-specific way to include the icon in the pacakge
but to the best of my knowledge there isn't.
I'm presently including in my setup.py file:
but this - while working - seems the wrong thing to do, as it
insert in the general python package something that is
Any ideas or primers on the subject?
Thanks in advance,
I'm pleased to announce a new release of buildout-versions.
This release allows you to specify the version of python that a buildout
should be run with, for example:
extensions = buildout-versions
versions = versions
python = 2.6
buildout-versions = 1.2
setuptools = 0.6c11
zc.buildout = 1.5.0
If you attempt to run the buildout with the wrong version of python,
you'll get an error such as:
Error: python version specified not found in sys.version: 2.6
The PyPI page is here:
The documentation is here:
If you have any problems or suggestions, please let me know on this list
or the Simplistix open source google group!
I have sent that to the PSF list because there's a PSF project about PyPI infra.
But someone complained, saying that I was doing this discussion
"behind closed doors"
SInce this is not my goal, I am now spamming more lists...
---------- Forwarded message ----------
From: Tarek Ziadé <ziade.tarek(a)gmail.com>
Date: Tue, Sep 27, 2011 at 10:37 AM
Subject: The state of PyPI
To: PSF Members List <psf-members(a)python.org>
Cc: Richard Jones <r1chardj0n3s(a)gmail.com>, Steve Holden <steve(a)holdenweb.com>
This is just a mail that summarizes the current state of PyPI, the
existing features, and what can be done next to improve stuff.
I am sending this in the PSF members list because we had a project of
an infrastructure going on, and I want to make sure all involved
parties are in the same page.
1/ stability and high availability
2/ private mirrors
3/ private projects
4/ tutorial ?
= stability and high availability =
we went in two directions to improve PyPI :
1/ add the mirroring protocol
2/ make the PyPI server more reliable by pushing its storage in a
== mirroring ==
The mirroring protocol (PEP 381) is implemented on server-side, I've
worked with Martin on this, and we have mirrors now:
Look at http://pypi.python.org/mirrors
Also, there's a client that anyone can use to set up a mirror:
The idea is that anyone in the community willing to maintain a mirror
can do so. We add the mirror in the CND, and make it available for
client tools to use. What's really missing right now is more
integration on client-side.
- Pip supports the mirroring protocol, and can fall back to a mirror,
but I am not 100% sure this is a default behavior. (please correct me
if it is now)
- Buildout knows how to use *another server* than the main PyPI, so
can manually switch to a mirror, but I don't think it's transparent.
- Distribute/Setuptools does not do anything for this, and should.
- everything is already implemented in packaging/distutils2
The effect of the mirrors is that PyPI being down should not impact
the community. This will be true once all tool are transparently using
== better infra ==
I think the project is staled right now.
= private mirrors =
Having a private mirror makes a lot of sense, when companies need to
make sure their build systems are not relying on external services
like PyPI or a mirror. It's also a good way to dramatically reduce the
load for the community servers.
The idea is that a Jenkins server that builds hundreds of Python apps
every hour should not hammer PyPI.
We have everything needed these days to set up this kind of system,
with zc.buildout or pip good practices.
What we need is a good tutorial or a guide [*]
= private projects =
The part that we do not address in the community is private projects:
since we don't have any permissions/group/roles system in PyPI,
everything is public.
One way to solve this is to have a local repository for private
packages, that is looked by tools like pip or easy_install, with the
What we need is a good tutorial or a guide [*]
= tutorial =
[*] If this helps, I am willing to work on a tutorial day for Pycon
US, that goes through all of this, to help people set up their dev.
environment the best way possible.
The material could then be published at python.org/pypi to help out.
I know Richard has some material already, so maybe this could be a
joint tutorial ?
Tarek Ziadé | http://ziade.org
Tarek Ziadé | http://ziade.org
At 04:20 PM 9/21/2011 +0200, Tristan Seligmann wrote:
>If you include "twisted.plugins" in your setup.py, then this works
>fine with distutils "setup.py install" as well as "pip install";
>setuptools "setup.py install" will install everything into an egg,
>which will also work due to the way __path__ is set. However, since
>"twisted" ends up in top_files.txt in the egg-info, "pip uninstall"
>will blow away your whole Twisted install when uninstalling a
>project shipping Twisted plugins that was installed with "pip install".
This really sounds like a bug in pip; top_level.txt is not a
replacement for a proper uninstall log.
>So, how should Twisted and Twisted-related projects be packaged in
>order to avoid these issues? Please bear in mind that the current
>plugin system in Twisted was first introduced around March 2005
>(replacing the even older plugin system in use at the time, I
>believe), thus there are quite a number of users relying on this
>code; any changes would need to be backwards-compatible to avoid
>causing problems for all of the existing projects and users relying
>on the functionality.
I think you've answered your own question here: there *isn't* any way
to package Twisted-related projects in a way that avoids the issue,
due to the bug in pip. It's not Twisted's fault that pip takes shortcuts here.
This conversation originally started in a bug report against pip,
but I'm moving it here since it seems this might be a better venue;
you may wish to read the bug log for interest, but I'll summarize the
issue from scratch anyway.
Twisted has a plugin system (twisted.plugin) which is used by
Twisted itself, as well as other projects (such as Axiom and
Dosage) to allow for pluggable extensibility. The implementation
looks for modules in a plugins package and then looks in those modules
for objects providing (in the zope.interface sense) the IPlugin
interface. For the sake of simplifying this explanation, I will only
refer to twisted.plugins in the rest of this mail, but assume that
everything applies similarly to plugin packages in other projects (eg.
axiom.plugins and dosage.plugins).
twisted/plugins/__init__.py sets __path__ as follows:
from twisted.plugin import pluginPackagePaths
The implementation of pluginPackagePaths loops through the
directories on sys.path, and includes DIR/twisted/plugins/ for every
DIR on sys.path, *except* when DIR/twisted/plugins/__init__.py exists.
This means that if you drop a project dir into PYTHONPATH while
developing, the plugins from your project will be picked up, even
though they're not installed into the copy of Twisted you're using;
you can also have a project shipping plugins installed in
~/.local/lib/pythonX.Y/site-packages even though you're using a
site-wide install of Twisted in /usr/lib/pythonX.Y/site-packages.
However, if you have multiple versions of Twisted on sys.path (because
you have a newer version installed locally overriding the site-wide
one, for example), plugins from a different version of Twisted won't
be accidentally picked up.
Given the way __path__ is set, if you have a project that ships
Twisted plugins, it would be possible to install it to
then include this in a .../site-packages/MyProject.pth. However, the
usual way to install your project is to install it to
.../site-packages/twisted/plugins/myproject_plugin.py; in other words,
directly into the existing Twisted installation. If you include
"twisted.plugins" in your setup.py, then this works fine with
distutils "setup.py install" as well as "pip install"; setuptools
"setup.py install" will install everything into an egg, which will
also work due to the way __path__ is set. However, since "twisted"
ends up in top_files.txt in the egg-info, "pip uninstall" will blow
away your whole Twisted install when uninstalling a project shipping
Twisted plugins that was installed with "pip install". Ironically, if
the project was installed with setuptools "setup.py install", there is
no problem, since pip just removes the egg that was installed, and
In the pip bug report, Carl Meyer suggests using the setuptools
"namespace packages" feature; this works around the immediate problem
(pip uninstall doesn't blow away your Twisted install anymore), but
causes somer othe problems of its own. First of all, code in
__init__.py is not functional for a namespace package, so the __path__
functionality described above ceases to function. Additionally,
twisted.plugins cannot be declared as a namespace package without also
declaring twisted as a namespace package, which thus also affects
twisted/__init__.py; the code in this module exports an __version__
attribute (identifying the version of Twisted) as well as importing
twisted.python.compat to ensure that various backwards compatibility
monkeypatches are installed.
So, how should Twisted and Twisted-related projects be packaged in
order to avoid these issues? Please bear in mind that the current
plugin system in Twisted was first introduced around March 2005
(replacing the even older plugin system in use at the time, I
believe), thus there are quite a number of users relying on this code;
any changes would need to be backwards-compatible to avoid causing
problems for all of the existing projects and users relying on the
mithrandi, i Ainil en-Balandor, a faer Ambar
Buildout and system packages are apparently hard. Buildout 1.5.x tries
hard to get it working, but I cannot get it to work (reliably), for
Perhaps the problem is that we're trying too hard. Both buildout 1.5.x
and the osc.recipe.sysegg recipe that I'm using on 1.4.x are trying to
look up packages and trying to include them in the path.
What you normally want is just to use the system's PIL and mapnik and
numpy. All stuff that you really don't want to build from source. So you
assume it is installed with a clicky-clicky installer on windows and
with apt-get on ubuntu.
Why not use this assumption? Add an "assumed-available" option to
buildout that lists the eggs that are assumed to be available locally?
No need to actually search them (which is the thing that breaks 1.5.x
for me). Just make sure the regular python path is in place as the last
item in sys.path.
Possible addition: allow a dotted path as a second parameter after the
"assumed-available" eggs: buildout tries to import that path to check
the availability of the egg that's assumed to be available.
So: if you need system eggs, you won't run buildout without system
paths. Only thing left is to tell buildout that certain eggs are
available locally so that it doesn't search for it. Just assume they're
there and leave it at that.
Reinout van Rees http://reinout.vanrees.org/
"If you're not sure what to do, make something. -- Paul Graham"