Gael Varoquaux wrote:
> On Sat, Apr 12, 2008 at 09:41:08PM -0400, Stephen Waterbury wrote:
>>> These are all broken and you should report bugs on them. I have
>>> reported many for Ubuntu. A system application should only ever
>>> depend on the system Python (or interpreter), never on the whims of
>>> your $PATH.
>> I agree with Barry here -- I would even go further and say that
>> the Python used by system apps (i.e., apps that the OS depends on)
>> should be separate and should not be on the default PATH (so the user
>> can't use it accidentally), and no non-system apps should use the
>> system Python unless specifically installed by the user or sysadmin
>> into the system Python's area.
> I agree with you, and not with Barry that all these are broken. If you
> look at the apps that use "#!/usr/bin/env", they are all pretty
> non-system apps, like for instance "f2py". I think it is good that these
> apps use the default python: I don't want to have f2py to be running for
> another python than my default one. I am actually quite surprised to see
> that ipython is installed with a hardcoded path to /usr/bin/python.
> Anyway, I am not going to bug report anything, as my system works just
> fine, thank you.
Yes, your luck holds -- as does mine, for now ... but since Barry
works for Canonical (besides being Python's new Release Manager),
he has a civic responsibility to report Ubuntu Python bugs even if
his own system works ... and since that can be quite time-consuming,
I can't blame him for trying to recruit some help ... ;)
On Fri, April 11, 2008 6:00 am, Dave Peterson <dpeterson(a)enthought.com>
> Message: 5
> Date: Thu, 10 Apr 2008 18:16:03 -0500
> From: Dave Peterson <dpeterson(a)enthought.com>
> Subject: Re: [Distutils] how to easily consume just the parts of eggs
> that are good for you
> To: "Phillip J. Eby" <pje(a)telecommunity.com>
> Cc: distutils-sig <distutils-sig(a)python.org>
> Phillip J. Eby wrote:
>> At 03:48 PM 4/10/2008 -0500, Dave Peterson wrote:
>>> Stanley A. Klein wrote:
>>>> On Wed, 2008-04-09 at 18:17 -0500, Dave Peterson wrote:
>>>>> I think I can sum up any further points by simply asking: "Should it
>>>>> be safe to assume I can distribute my application via eggs /
>>>>> easy_install just because it is written in Python?"
>>>> I think that based on this discussion the bottom line answer to this
>>>> question is "No".
>>> I agree that it seems like that's where things are headed in the
>>> discussion. But the discussion doesn't always coincide with the
>>> reality, right? I guess I'm trolling more for a statement from the
>>> setuptools maintainer here.
>>> Particularly since I'm looking for an answer to my question about
>>> should Enthought continue to invest time into a setuptools patch that
>>> lets developers include docs, config files, etc. in eggs for
>>> installation in a FHS-approved location at install time?
>> I think it's more than reasonable to define a standard for including
>> such data. I'm somewhat more skeptical about doing that installation
>> automatically within easy_install. Likewise, I'm skeptical about
>> doing other sorts of non-package, non-script installation. I'd like
>> to see proposals that show due care to cross-platformness,
>> uninstallability, etc.
>> In other words, when it comes to a "patch" -- the documentation is
>> going to count for a lot more than the code, and I'd rather see a
>> concrete proposal well in advance of the patch.
>> Sooner would be better than later, too, because it's likely that the
>> plan for "non-egg installs" is going to be affected by the plan as well.
> I believe I understand, and agree, with your concerns. Let me be clear
> on the status of where we are in our work: we've internally talked
> through a number of design possibilities, and are now trying out (via
> hacking on setuptools) how the one we thought was "best" worked out. In
> particular, we're concerned about the difficulty of use in terms of even
> just the use-cases we have for ETS projects. Also, since we do a bit
> of cross-platform deployment, we're also investigating those effects on
> the design as well. That being said, I don't think we're ready to put
> forward a proposal that would withstand too much public scrutiny quite
> yet - at least if the resulting discussion implied a significant time or
> effort commitment on our part to carry the conversation forward. If it
> sounded like we'd already figured it all out, I apologize for getting
> people's hopes up! I just wanted to make sure further pursuit in this
> direction on our part wasn't completely wasted.
> The above not withstanding, if anyone is interested in talking about it
> / helping us, we certainly wouldn't ignore you. I just can't promise
> immediate responses due to pretty pressing customer commitments on our
> Regarding the mention of 'uninstallability' above, I assume it would be
> sufficient if the installed files were simply included in the generated
> list of files provided by the "--record" option since there is currently
> no uninstall command at all? Or is there something else you'd like to
> see as an intermediate measure? I'd love to add an uninstall option to
> easy_install as well (it's something we get hit about alot by our user
> community) but there's only so many hours in a day. ;-)
Here are a few principles I think I've gleaned from this discussion:
1. Eggs are not intended to serve as substitutes for rpms, debs, and
other platform package distribution systems. They are primarily intended
as a convenience for developers during development. They can be used in
distribution packaging in a manner analogous to .jar files in java.
2. Every platform wants to put the files somewhere different.
Accordingly, Setuptools and Distutils need to support mapping the files to
the preferred locations for the various platforms.
3. Setuptools and Distutils support some platform packaging systems, such
as rpm. Packagers should distribute using those systems. If eggs are
included in a package, they should be treated as any other file for that
packaging system and not as a form of distribution. For example, an rpm
should reasonably be able to contain egg files along with other
distribution files. (Note that java-related rpms often contain .jar
files.) Consideration should be given to adding features to existing
support for platform packaging systems. Also, if egg files are to be
included in a distribution package, it needs to be a two-step process
(first make the eggs, then make the package).
4. It would probably help to have a style guide or automated converter
for naming and placement of files in a source distribution to facilitate
file location mappings and platform naming issues. For example, Windows
accepts file names with spaces in them but Linux generally doesn't (and
bdist_rpm produces errors if it sees them). For a multiplatform
distribution either a style guide or a lint-type tool needs to address
issues like this.
At 06:07 PM 4/11/2008 +0200, Tarek Ziadé wrote:
>On Fri, Apr 11, 2008 at 6:01 PM, Phillip J. Eby <pje(a)telecommunity.com> wrote:
> > >
> > >was the patch satisfactory or is it still lacking?
> > >Or do I need to do anything more for it to be applied?
> > It's sitting in my folder of patches to eventually be applied. :)
>What about this one ?
>Do you have a public tracker where we can submit patches ?
No -- but I heard it's being worked on. My first preference would be
a python.org hosted Roundup instance, especially if I can use my
existing login. If that can't be arranged, I'll probably look into
throwing up a Trac or Roundup myself -- eventually.
I need to install a big bunch of python extentions in a couple of offline
windows XPcomputer. Is there any standard way to do that on the win32
The best options I thought of is installing ez_install manually on every
computer, and then install all the egg-files in a certain directory. Is
there any better way to do that? Is there a standard way to distribute
Example extentions: scipy, pyserial, wxpython
At 11:37 PM 4/10/2008 +0200, Jeroen Ruigrok van der Werven wrote:
>I am not sure (haven't verified) if my email ever got through to the list,
>but with FreeBSD's Perl installation an additional module gets installed
>that registers every CPAN installed module as a pkg in FreeBSD's pkg system.
>Given the recent bruhaha about the setuptools and eggs (hey, from a
>programmer and sysadmin point of view I like 'm) I wonder if this would be
>easier for the discussion if you could dump a prototype on the table.
>If it would help I can see if I can cook up something.
At this point, it has become clear to me that the package database
isn't really necessary for other systems. easy_install could simply
not uninstall things that it doesn't find listed in its *own* metadata.
Really, the only technical hurdle remaining to solving egg-free
installation is setting up a namespace package's __path__ correctly,
in a situation where:
1. The user has system packages contributing to the namespace in a
2. There is no __init__.py in the namespace under that site directory,
3. There are non-egg packages on PYTHONPATH that contribute to the
same directory, and
4. Two versions of the same module within that namespace have been
installed, one in site-packages and the other on PYTHONPATH
Actually, there are various other criteria involved, such as whether
pkg_resources is imported or not, and whether the setuptools site.py
hack is employed in the PYTHONPATH directory. For example, if
pkg_resources were imported (which normally happens when
setuptools-installed scripts are run), it would be possible to have
it do the sorting, and thus ensure that there's never a problem like that.
What it would not fix, is the circumstance where you just pop into
the Python interpreter with your PYTHONPATH set, if you have the
above situation. And the above situation is going to occur anytime
you're using PYTHONPATH to get a later version of something that uses
a namespace package.
So, the other alternative is to keep the 'site.py' hack in effect,
but either change the contents of that site.py or add extra
information to a .pth file so that the namespaces get sorted at that
point. There would still be a hacked site module at that point, but
at least upgrading the site.py would let us get rid of the need to
load any .pth files from the directory, put eggs ahead of other things, etc.
For that matter, the site.py wouldn't be needed unless you were
installing things with namespace packages in it. (Unfortunately,
looking at site-packages to see if the hack is needed won't help,
since you might install a conflicting system package *later*.)
NOTE: cc'ing distutils-sig for help debugging potential problem with
setuptools on gentoo.
Hans Meine wrote:
> Am Donnerstag, 10. April 2008 09:19:31 schrieb Hans Meine:
>> The same errors occur without virtualenv, so you're obviously right that
>> this is setuptools-related. I could try again and really deinstall
>> Gentoo's setuptools (instead of just locally installing it the official
>> way), but I really doubt now that this would make a difference.
> So, I have even tried this - deinstalling setuptools altogether, and
> installing everything from scratch. I used "script" to create a log file of
> the process with all input and resulting output + errors:
Thanks for providing that full log! From looking at that, I'm really
starting to suspect a bug in setuptools 0.6c8 as I can't make any sense
of the following sequence of events. While it's possible we've done
something wrong in our setup.py scripts, I have my doubts as this
release appears to install fine for users on Windows, OSX, and RH. (For
distutils-sig readers, Hans is using Gentoo.)
1) During the first install attempt of "ets", the log shows two lines
reporting success with installing
enthought.tvtk-2.0.2-py2.4-linux-i686.egg. One right after downloading
and building enthought.tvtk from the source tarball, and one in the
middle of the enthought.kiva build.
2) Regarding that enthought.kiva build, the log indicates that a
dependency caused a search to be made for
"enthought.kiva>=2.0.3.dev,<=2.0.3" and a match was found
(enthought.kiva-2.0.3.tar.gz) The log then shows this building,
apparently a successful build (no error messages, just warnings) up
until the normally reported "Installed /path/to/egg" line where it
mentions success with installing enthought.tvtk rather enthought.kiva.
After that, it reports being unable to find the enthought.kiva distribution.
3) You then re-run the "easy_install -f ... ets" command which has no
problem building enthought.kiva this time using the exact same source
I have no idea what might be causing this sequence of events. Has
anyone seen anything similar with setuptools?
At 09:10 AM 4/10/2008 +0100, Paul Moore wrote:
>For me, on Windows, the biggest negative aspect of eggs is when people
>provide binary installations (ones with compiled C extensions) in eggs
>only. I prefer bdist_wininst/bdist_msi installers, and with compiled
>code, I can't always build from source (complex library dependencies,
>So, given that an egg already has everything that is needed at
>runtime, how easy would it be to create a tool which converted an egg
>into a bdist_wininst installer? With that, I could happily forget
Basically what you need to do is rewrite the zipfile index such that
everything is prefixed with PURELIB/ or PLATLIB/, except for the
EGG-INFO directory which must be renamed to match the original egg's
filename, plus '-info' -- i.e., foo-1.2-py2.3-win32.egg should have
its EGG-INFO renamed to PURELIB/foo-1.2-py2.3-win32.egg-info or
Once the egg is reformatted in this way, it's a perfectly valid
zipfile that the existing bdist_wininst code could be reused to turn
into an .exe.
For me, on Windows, the biggest negative aspect of eggs is when people
provide binary installations (ones with compiled C extensions) in eggs
only. I prefer bdist_wininst/bdist_msi installers, and with compiled
code, I can't always build from source (complex library dependencies,
So, given that an egg already has everything that is needed at
runtime, how easy would it be to create a tool which converted an egg
into a bdist_wininst installer? With that, I could happily forget
PS I am assuming that "eggs are only a packaging format" - and that
all of the setuptools goodies like namespace support, entry points
etc, that some projects benefit from, would still work whether the
package was an egg or a conventionally-installed bdist_wininst...
At 05:05 PM 4/7/2008 -0400, Alexander Michael wrote:
>a. I believe that having side-car files that sit alongside
>packages because they have the same base name makes the database more
>transparent to the uninitiated.
I'm not aware that this was ever a stated design goal, nor why it
should have any priority. OTOH, files named by distribution would be
at least as, if not even *more* transparent than package names, so I
don't see any particular benefit to this.
>Just browsing a directory of python
>packages will allow you to see what's going on. Moving like-names
>files around manually maintains the integrity and availability of the
Moving anything manually, other than the *entire* directory, will be
unlikely to retain any form of integrity, so it's best not to give
the false impression that it would.
>I think that having magic entries in an essentially "hidden"
>directory somewhere will cause all sorts of trouble that could be
>avoiding at the cost of a small bit of duplication.
> b. I assume, perhaps incorrectly, that most distributions contain
>only a single package.
Very incorrectly, unless you mean a single top-level package. Odds
are fairly good that if there's a package, there's probably at least
a subpackage, too, like perhaps a tests subpackage.
>That said, I do agree that if you are primarily interested in a
>database of *distributions* (as opposed to *packages*) then something
>like is proposed in PEP 262 makes more sense (but it would have to be
>per directory and not site-wide due to the dynamic nature of the
That's exactly what I want. The only reason I didn't just implement
easy_install using a per-directory form of PEP 262 is that I wanted
something done rather more immediately. That was years ago, so I can
afford to be more patient now. :)
>This is a trade-off between putting the metadata up
>front in an obvious and easy to understand way so that it will
>hopefully have a better chance of being noticed and maintained, versus
>tucking it away hidden someplace so that even though it is broken, it
>doesn't bother anyone until they care enough to fix it. *It is this
>trade-off that I am exploring with this strawman "counter" proposal to
Someone would have to be crazy to maintain this information by
hand. So I'd actually consider it an advantage if the file format
made this fact plain, by using something that's difficult for a human
being to maintain, like say a pickle. ;-) OTOH, it's possible that
some system packagers will not wish to use Python to generate the
files, so using something a bit less complex would be a good
idea. The format proposed by PEP 262 isn't really that bad of a
trade-off in those terms.
>2. The strawman proposal did not explicitly address how optional
>add-on tools (like setuptools) might manage namespace packages.
I think there's some mistunderstanding here about the proposal's
goals. If the proposal doesn't work for setuptools, it doesn't work, period.
The entire point is to allow setuptools to do its work without
annoying the people who don't want to use it.
>I agree with Floris that the best way to avoid magic is to actually
>have the sub-packages in a namespace share the same parent directory
I agree with this also. The issue is that an __init__.py must exist
for this to happen, but most system packaging tools (e.g. RPM)
require that a given file be owned by at most one system package
(i.e., distribution), whereas the contents of a namespace package are
assembled from multiple distributions.
That's the problem that needs solving, not runtime support for the
>4. Concerns were raised about the performance penalty for using the
>side-car style files without version numbers possibly not all of which
>were located at the top-most level of the directory listed in the
>Any add-on tool that actually used the data would very likely need to
>build a cache of the data using a more efficient representation,
This is a misunderstanding of the point I raised. Floris merely
asked why there were version numbers in .egg-info files, and I
answered him. That doesn't actually have much, if anything, to do
with the package database proposal. It's merely how installed
distributions' versions can be recognized quickly at runtime, not
anything to do with how potential installation conflicts are handled
at installation time.
easy_install uses eggs for installation simply because it need never
worry about file ownership conflicts. There is a direct mapping from
a distribution to its files: the contents of a zipfile or
subdirectory. This also allows for (relatively) straightforward
The goal of the proposal, then, is to have a way for easy_install to
have another way to map from a distribution to its owned files (and
vice versa), so that eggs are not necessary for normal,
At 11:48 PM 4/9/2008 +0100, Paul Moore wrote:
>On 09/04/2008, Phillip J. Eby <pje(a)telecommunity.com> wrote:
> > It would be, if .eggs were a packaging format, rather than a binary
> > distribution/runtime format.
> > Remember "eggs are to Python as jars are to Java" -- a Java .jar
> > doesn't contain documentation either, unless it's needed at
> > runtime. Same for configuration files.
>And yet, Java doesn't have an equivalent of easy_install for jar
>files, to my knowledge.
Actually, OSGi and Eclipse plugins and "feature sites" come quite
close, and setuptools rips off many of its features from them. OSGi
is basically a standard for additional .jar metadata to encompass
dependencies and other info.