[Distutils] formencode as .egg in Debian ??

[Phillip J. Eby]

At 09:04 PM 11/25/2005 +1100, David Arnold wrote:
>-->"Phillip" == Phillip J Eby <pje at telecommunity.com> writes:
>   Phillip> I would like to clarify the phrase "shipped as an egg",
>   Phillip> though.  To me, that would mean that the developer is
>   Phillip> distributing a binary .egg file, and I'm assuming that Debian
>   Phillip> is primarily interested in *source* packages, being a Free
>   Phillip> Software distribution.
>
>Debian ships both source packages and binary packages.  It's my
>impression (although I could be wrong) that most users use only the
>binary packages.

I was saying I assume Debian packages (source or binary) are built from 
upstream source packages, but you were asking about upstream packages 
"shipped as an egg".  .egg files do not include the full source of a 
disutils package, except perhaps the .py files.  C source would not be 
included in a distributed .egg.  Anyway, I assume that Debian Python 
packages are currently built from upstream source, not an upstream binary, 
and .egg files would be considered a binary.


>Assume a developer grabs some setuptools-using Python app, and tries to
>run it on a Debian system.  It will look in sys.path for its
>dependencies.  If it doesn't find the egg info, as I understand it, the
>app will then go out and grab those dependencies via PyPI until it's
>able to run.

No, the default behavior will be to throw a DependencyNotFound exception 
describing the problem.  A particular application can in principle catch it 
and do something about it, or pass in an installer hook that will be tried 
before throwing the exception, but that's up to the application.  The 
setuptools runtime by default is just going to give an exception.  Indeed, 
if it's a required dependency (as opposed to a dependency needed for an 
optional feature of the application), then you're not going to be able to 
start the application at all; you'll just get the DependencyNotFound 
exception currently.


>But, if compatible versions of those dependencies are already installed
>as Debian packages *without* egg metadata, will these be ignored?

That's correct; the wrapper scripts presently have no way to tell such 
packages exist, and no reliable uniform way to check their version.  One 
possible feature that has been discussed is allowing package authors to 
specify fallback hooks in their application to try to verify the presence 
of a package by importing it and testing for the existence of certain 
symbols.  A couple of developers have suggested that for some dependencies, 
they would be willing to specify such fallbacks in order to take advantage 
of non-egg system packages.  At this point, the means by which this could 
happen is incompletely specified, and it would be tricky to implement 
because it means importing code from partially processed dependencies in 
order to process *their* dependencies, which means the dependency 
resolution process can't be safely unwound and retried.

The other thought I'd had about this (aside from adding .egg-info files to 
every Python package), is to provide an entry point group that could be 
registered by a module provided by a system packager that would use the 
packager's native API to test for the presence of a dependency, and supply 
a mock Distribution object with the metadata.  Of course, this hypothetical 
packager API would have to be able to map PyPI names and versions to native 
package names and versions and obtain the metadata, so I'm not sure that in 
the general case such a thing would be any more useful.  (Aside from the 
fact that such a feature would allow you to put the metadata in /usr/share 
without adding that directory to sys.path.)


>Even if it was possible for Debian to extend this downloading mechanism
>so that it looked for dependencies via apt-get before trying to install
>from the raw source or egg or whatever,

Again, this is not the default behavior.  It's the "easy_install" tool that 
does such downloading and installing, or "setup.py install".  Downloading 
and running things from the internet by default didn't seem to me like a 
safe thing to do.  :)  There are various "setup.py" commands that will do 
it (such as "develop" and "test"), but by running a setup.py you're 
trusting the author to install arbitrary stuff on your machine anyway.  The 
installed application, unless it provides some explicit user functionality 
to say, download plugins, will not have this download-and-install behavior.

I'm assuming that a Debian user will not be using "easy_install" except to 
get and run bleeding edge stuff, and that in that case they will be 
directing it to install in /usr/local or a personal directory.


>it would usually be the case
>that the user running the newly downloaded Python application would not
>have permission to install system packages.
>
>Ultimately, I suppose this isn't a major problem -- if a user (or
>developer) runs such an application, they might end up pulling down and
>installing a bunch of dependencies in their home directory (that's where
>they'd go by default, right?)

No.  The default is determined by the standard distutils configuration file 
chain.  If an install location isn't specified in the user's 
~/.pydistutils.cfg, the systemwide distutils.cfg will be checked, and 
finally the default will be /usr/lib/.../site-packages.  Debian, by the 
way, apparently ships with this default unchanged; it should presumably 
point to /usr/local/.../site-packages instead, so that locally-installed 
packages would go to the right place.  Or you could set a default location 
such that easy_install will install to a home-relative location by default, 
and include that directory in your .pth processing (as is done by Mac OS 
Python with the ~/Library/.../site-packages directory).


>  But once the app is packaged, both it and
>its dependencies will be available for all users, so it'll just be early
>adopters who will encounter this.

Correct, and only if they are using easy_install or a setup.py to install 
the package in the first place.  And a Debian user using Debian packages 
should have no reason to touch easy_install, unless they lack permission to 
install packages.