<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Phillip J. Eby wrote:
<blockquote
cite="mid:20080318004718.56E173A40FF@sparrow.telecommunity.com"
type="cite">
<pre wrap="">At 05:10 PM 3/17/2008 -0500, Jeff Rush wrote:
</pre>
<br>
<blockquote type="cite">
<pre wrap="">1. Many felt the existing dependency resolver was not correct. They wanted a
full tree traversal resulting in an intersection of all restrictions,
instead of a first-acceptable-solution approach taking now, which can
result in top-level dependencies not being enforced upon lower
levels. The
latter is faster however. One solution would be to make the resolver
pluggable.
</pre>
</blockquote>
<pre wrap=""><!---->
Patches welcome, on both counts. Personally, Bob and I originally
wanted a full-tree intersection, too, but it turned out to be hairier
to implement than it seems at first. My guess is that none of the
people who want it, have actually tried to implement it without a
factorial or exponential O(). But that doesn't mean I'll be unhappy
if somebody succeeds. :)
</pre>
</blockquote>
<br>
I think we'd make significant progress by just intersecting the
dependencies we know about as we progress through the dependency tree.
For example, if A requires B==2 and C==3, and if B requires
C>=2,<=4, then at the time we install A we'd pick C==3 and also
at the time we install B we'd pick C==3. As opposed to the current
scheme that would choose C==4 for the latter case. This would allow
dependent projects (think applications here) to better control the
versions of the full set of libraries they use. Things would still
fail (like they do now) if you ran across dependencies that had no
intersection or if you encountered a new requirement after the target
projected was already installed.<br>
<br>
<br>
If you really wanted to do a full-tree intersection, it seems to me
that the problem is detecting all the dependencies without having to
spend significant time downloading/building in order to find them
out. This could be solved by simply extending the cheeseshop
interface to export the set of requirements outside of the egg /
tarball / etc. We've done this for our own egg repository by
extracting the appropriate meta-data files out of EGG-INFO and putting
it into a separate file. This info is also useful for users as it
gives them an idea of how much *new* stuff is going to be installed (a
la yum, apt-get, etc.)<br>
<br>
<br>
<blockquote
cite="mid:20080318004718.56E173A40FF@sparrow.telecommunity.com"
type="cite">
<pre wrap="">In other words, we attempt to achieve heuristically what's being
proposed to do algorithmically. And my guess is that whatever cases
the heuristic is failing at, would probably not be helped by an
algorithmic approach either. But I would welcome some actual data, either way.
</pre>
</blockquote>
<br>
With our ETS projects, we've run into problems with the current
heuristic. Perhaps we just don't know how to make it work like we
want? <br>
<br>
We have a set of projects that we want to be individually installable
(to the extent that we limit cross-project dependencies) but we also
want to make it easy to install the complete set. We use a meta-egg
for the latter. It's purpose is only to specify the exact versions of
each project that have been explicitly tested to work together -- you
could almost think of it as a source control system tag. Whereas on
the individual projects, we explicitly want to ensure that people get
the latest possible release of each required API so the version
requirements are wider here. This setup causes problems whenever we
release new versions of projects because it seems easy_install ignores
the meta-egg exact versions when it gets down into a project and comes
across a wider cross-project dependency. We ended up having to give
up on the ranges in the cross-project dependencies and synchronize them
to the same values in the meta-egg dependencies. There are numerous
side-effects of this that we don't like but we haven't found a way
around it.<br>
<br>
<blockquote
cite="mid:20080318004718.56E173A40FF@sparrow.telecommunity.com"
type="cite">
<pre wrap="">Again, though, patches are welcome. :) (Specifically, for the
trunk; I don't see a resolver overhaul as being suitable for the 0.6
stable branch.)
</pre>
</blockquote>
<br>
We're planning to pursue this (for the above mentioned strategy) as
soon as we work ourselves out of a bit of a backlog of other things to
do.<br>
<br>
<br>
<br>
<blockquote
cite="mid:20080318004718.56E173A40FF@sparrow.telecommunity.com"
type="cite">
<blockquote type="cite">
<pre wrap="">2. People want a solution for the handling of documentation. The distutils
module has had commented out sections related to this for several years.
</pre>
</blockquote>
<pre wrap=""><!---->
As with so many other things, this gets tossed around the
distutils-sig every now and then. A couple of times I've thrown out
some options for how this might be done, but then the conversation
peters out around the time anybody would have to actually do some
work on it. (Me included, since I don't have an itch that needs
scratching in this area.)
In particular, if somebody wants to come up with a metadata standard
for including documentation in eggs, we've got a boatload of hooks by
which it could be done. Nothing's stopping anybody from proposing a
standard and building a tool, here. (e.g. using the setuptools
command hook, .egg-info writer hook, etc.)</pre>
</blockquote>
<br>
Enthought has started an effort (it's currently one of two things in
our ETSProjectTools project at
<a class="moz-txt-link-freetext" href="https://svn.enthought.com/svn/enthought/ETSProjectTools/trunk">https://svn.enthought.com/svn/enthought/ETSProjectTools/trunk</a>) and
we're experimenting with our solution before proposing it as a patch.
We'd love some more help if anyone wants to participate.<br>
<br>
<br>
<blockquote
cite="mid:20080318004718.56E173A40FF@sparrow.telecommunity.com"
type="cite">
<blockquote type="cite">
<pre wrap="">3. A more flexible internal handing of the different types of files is needed.
Currently the code, data, lib, etc. files are aggregated at
build time and
people would like them to be kept separate until install/packaging time.
</pre>
</blockquote>
<pre wrap=""><!---->
I don't know what this means, exactly.
</pre>
</blockquote>
<br>
A number of projects want to provide various types of files besides
code in their distributable, and they'd like these to end up in
standard locations for that type of file. Think documentation, sample
data, web templates, configuration settings, etc. Each of these
should be treated differently at installation time depending on
platform. On *nix, docs should go in /usr/share/doc whereas we might
need to create a C:\Python2.5\docs on Windows. With sample data and
templates, you probably just want it accessible outside of the zipped
egg so users can easily look at it, add to it, edit it, etc.
Configuration settings should be installed with some defaults into a
standard configuration directory like /etc on *nix, etc.<br>
<br>
Basically the issue is that it needs to be easier to include different
sets of files into an egg for different actions to be taken during
installation or packaging into an OS-specific distribution format.<br>
<br>
<br>
<blockquote
cite="mid:20080318004718.56E173A40FF@sparrow.telecommunity.com"
type="cite">
<blockquote type="cite">
<pre wrap=""> The other is the use of a single .pth file to control the list
of activated
packages. Those who produce distributions would prefer a magic directory
into which links to distributions could be dropped, similar to
the current
best practices for Linux, with /etc/conf.d/, /etc/profile.d/,
/etc/xinetd.d/ and so forth.
</pre>
</blockquote>
<pre wrap=""><!---->
site-packages is that directory, and has been since long before
setuptools. Just drop uniquely-named .pth files there, and you're good to go.
</pre>
</blockquote>
<br>
But the docs for easy_install claim that the list of active eggs is
maintained in easy-install.pth. Also, if I create my own .pth file,
and the user tries to update my version to a new one, will the
easy_install tool modify my .pth file to remove the mention of the old
version from my sys.path and put the new version in the same .pth
file? Or will it now be listed in both places? Or will it only in
easy-install.pth? <br>
<br>
<br>
<blockquote
cite="mid:20080318004718.56E173A40FF@sparrow.telecommunity.com"
type="cite">
<blockquote type="cite">
<pre wrap="">7. Many wanted to ability to install files anywhere in the install tree and
not just under the Python package. Under distutils this was possible but
it was removed in setuptools for security reasons.
</pre>
</blockquote>
<pre wrap=""><!---->
It wasn't security, it was manageability. Egg-based installation
means containment, (analagous to GNU stow) and therefore portability
and disposability of plugins. (Which again is what eggs were really
developed for in the first place.)
</pre>
</blockquote>
<br>
Yes, but as you've already pointed out, they've escaped into a larger
ecosystem and this restriction is a severe limitation -- leading to
significant frustration. Especially as projects evolve and want to do
something more complex than simply install pure Python code. Here at
Enthought, we use and ship a number of projects that have extensions
and thus dynamic libraries that need to either be modified during
installation to work from the user's installed location, or copied
elsewhere on the system to avoid the need to modify (which we also
can't do via an egg install) env variables, registries, etc. We'd
also love to be able to ship end-user enterprise-scale applications via
eggs so that bug fixes and updates don't require downloading a
monolithic 100MB+ installer. But doing that requires the ability to
update desktop icons, menus, etc. which we also can't do automatically
via an egg.<br>
<br>
If you don't want the burden on setuptools to support, much less track,
all these options, then perhaps it could just support automatic
execution of a post-install script (and pre-uninstall script if
uninstallation ever happens) that allows individual project developers
to do what they need to do? Let the burden of describing how those
things happen and how to uninstall/relocate/update them fall to the
provider of the projects that do them.<br>
<br>
Also, IIUC, stow only tries to "contain" the hard files. It puts links
in multiple standard locations (for man pages, executables, libraries,
etc.) If setuptools supported these options, I don't think there'd be
any discussion here except for things like "how do I extend the set of
things the tool supports so that my foo-type files get linked into the
standard /os/path/to/foo for the X os?"<br>
<br>
<br>
<blockquote
cite="mid:20080318004718.56E173A40FF@sparrow.telecommunity.com"
type="cite">
<blockquote type="cite">
<pre wrap=""> Custom code can still
be written to do this explicitly but this is not popular.
</pre>
</blockquote>
<pre wrap=""><!---->
No kidding. :) Current best practice is to include a script or
module in the package that can install other files to a designated
location. Personally, though, I tend to view applications and
libraries that target specific install locations to be overreaching
their bounds, and stepping into sysadmin territory. Give me the
tools to install the data, don't just dump it somewhere on my system
where *you* think it should go, in other words.
</pre>
</blockquote>
<br>
I should have read ahead. This sounds close to what I've been
describing except that this leads me to picture a script that prompts
for install locations and allows the user to customize the destinations
rather than one that assumes everything goes in a standard place. I'm
all for this, and the continuation of the ability to install an egg
into a user-environment vs. a system-environment. <br>
<br>
The only thing missing here is the ability for the installer to
automatically run that script so that installation isn't a disjointed,
two-step manual process that a user is prone to forgot to complete. <br>
<br>
One of the features of Enthought's Enstaller extension to easy_install
was that it looks for a post_install.py script in EGG-INFO and if one
is found, runs it. I would think that getting this into setuptools
would be a significant step forward but I believe you previously
rejected that idea. We'll take a stab at creating a patch for you if
you're more receptive to that idea now. Just let me know.<br>
<br>
<br>
<blockquote
cite="mid:20080318004718.56E173A40FF@sparrow.telecommunity.com"
type="cite">
<pre wrap="">On the other hand, I've been puzzling over how to handle legitimate
post-install features. On Windows, both wx and pywin32 have a real
need to do some actuall "install" operations. Some is just copying
files, but pywin32 also has to do some registry stuff. I don't know
how to allow just what's sensible, without opening up a huge can of
worms, though.
</pre>
</blockquote>
<br>
I think there are lots of situations that are legitimate (projects with
extensions, projects that want to put icons on the desktop or in menus,
projects that need to interact with a registry, projects that want to
put configuration information somewhere other than in a zip file in a
site-packages dir, etc.) I think we should worry less about
preventing developers from shooting themselves in the foot and more
about ensuring that they can hunt for food for their survival. We can
always tighten things down after seeing the usecases that develop,
right?<br>
<br>
<br>
-- Dave<br>
</body>
</html>