[Distutils] setuptools in a cross-compilation packaging environment

[M.-A. Lemburg]

Phillip J. Eby wrote:
>>I must admit that I haven't followed the discussions about
>>these .egg-info directories. Is there a good reason not to
>>use the already existing PKG-INFO files that distutils builds
>>and which are used by PyPI (aka cheeseshop) ?
> 
> I don't know if there's such a reason or not, but in any case that's what 
> we use as part of the egg-info directories.  However, we *also* allow for 
> unlimited metadata resources to be provided in egg-info, as this is what 
> allows us to carry things like plugin metadata and scripts in the 
> egg.  There are other metadata files listing the C extensions in the 
> package, the "namespace packages" that the egg participates in, and so on.
> 
>>Hmm, you seem to be making things unnecessarily complicated.
> 
> That probably just means you're not familiar with the requirements. 

I did read your posting, but still don't understand why you
need a multitude of meta-data files in a special directory.

PKG-INFO is general and extensible enough to hold all that information,
IMHO.

>>Why not just rely on the import mechanism and put all
>>eggs into a common package, e.g. pythoneggs ?!
>>Your EasyInstall script could then modify a file in that
>>package called e.g. database.py which includes all the
>>necessary information about all the installed packages
>>in form of a dictionary.
> 
> 
> You completely lost me.  A major feature of eggs is that for an application 
> needing plugins, it can simply scan a directory of downloaded eggs and plug 
> them into itself.  Having a required installation mechanism other than 
> "download the egg and put it here" breaks that.

While I don't find a non-managed Python installation
mechanism a particularly useful goal to have, you could still
have the same thing by using and scanning a sub-directory of
the pythoneggs package directory or directories listed in
an environment variable PYTHONEGGS as fallback solution (if the
egg was not found in the database.py module).

> What's more, putting them in a single package makes it impossible to have 
> eggs installed in more than one directory, since packages can't span 
> directories, at least not without using setuptools' namespace package 
> facility.  And using that facility would mean the runtime would have to 
> always get imported whenever you used an egg - which is *not* required 
> right now unless you're using a zipped egg with a C extension in it.  And 
> even then the runtime only gets imported if you actually try to import the 
> C extension.  So, it seems to me your approach creates more I/O overhead 
> for using installed packages.

If your application wants to support drop-in eggs for plugins,
I don't see the need to call some startup code in that application
as a problem.

If the application does not need drop-in eggs, the package
approach would be more effective.

> Finally, don't forget that eggs allow simultaneous installation of multiple 
> versions of a package.  So, you'd *still* have to have sys.path manipulation.

Nope - the .__path__ attribute of Python packages makes
this easy:

	http://www.python.org/doc/essays/packages.html

>>This would have the great advantage of allowing introspection
>>without too much fuzz and reduces the need to search paths,
>>directories and so-on which causes a lot of I/O overhead
>>and slows down startup times for applications needing
>>to check dependency requirements a lot.
> 
> 
> And the disadvantage of absolutely requiring install/uninstall steps, which 
> is anathema. 

Oops. I disagree on that one. Not only does install/uninstall
make system administration a whole lot easier, it also prevents
accidental misconfigurations, permission problems, problems
with finding the right paths and locations, etc. etc.

Also note that the pythoneggs package approach would still
allow you to use unmanaged eggs - albeit as fallback solution
or specifically for plugins.

>>>>Please make sure that your eggs catch all possible
>>>>Python binary build dimensions:
>>>>
>>>>* Python version
>>>>* Python Unicode variant (UCS2, UCS4)
>>>>* OS name
>>>>* OS version
>>>>* Platform architecture (e.g. 32-bit vs. 64-bit)
>
> Well, my presumption here is that we're going to get the scheme right for 
> Python at large, and make it standard.  Are you saying that some packages 
> should have their own scheme?  That's not really workable since in order to 
> import the package and use its scheme, we would have to first know that the 
> package was compatible!

We're talking about filenames here - they are intended
to be read and understood by humans, not machines (these
can use the PKG-INFO data inside the archives or from
PyPI).

That said, yes, the way platforms are setup, it does sometimes
make it necessary to add extra information to such a filename.

E.g. say you write a plugin for Zope that only works in Zope3
and not Zope2. Such a plugin would use the "zope3" distinguisher
in its archive name.

>>>If you have suggestions, please make them known, and let's get them into
>>>the distutils in general, not just our own offshoots thereof.
>>
>>This is what we use:
>>
>>def py_version(unicode_aware=1, include_patchlevel=0):
>>
>>[snip]
>>The result is a build system that can be used to build
>>all binaries for a single platform without getting
>>conflicts and binaries that include a proper platform
>>string, e.g.
>>
>>egenix-mxodbc-zopeda-1.0.9.darwin-8.2.0-Power_Macintosh-py2.3_ucs2.zip
>>egenix-mxodbc-zopeda-1.0.9.linux-i686-py2.3_ucs2.zip
>>egenix-mxodbc-zopeda-1.0.9.linux-i686-py2.3_ucs4.zip
> 
> 
> eggs put the Python version before the platform, because "pure" eggs that 
> don't contain any C code don't include the platform string. 

They should really use "noarch" like everybody else does :-)

> We also don't 
> have a UCS flag, but if we did it should be part of the platform string 
> rather than the Python version, since "pure" eggs don't care about the UCS 
> mode, and even if they did, that'd be a requirement of the package rather 
> than the egg itself being platform specific.

This is not correct: unichr(100000) won't work in UCS2
builds - it will in UCS4 builds, so even though the
.pyc files run on both builds unchanged, the application
may very well require the used Python version to be a
UCS4 build in order to be able to use UCS4 features.

>>>A single .pth file is certainly an option, and it's what easy_install
>>>itself uses.
>>
>>Fair enough.
>>
>>Could this be enforced and maybe also removed
>>completely by telling people to add the egg directory to
>>PYTHONPATH ?
> 
> If by "egg directory" you mean a single .egg directory (or zipfile) for a 
> particular package, then yes, for that particular package you could do 
> that.  If you mean, can you just put the directory *containing* eggs on 
> PYTHONPATH, then the answer is no, if you want the package to be on 
> sys.path without any special action taken (like calling 
> pkg_resources.require()).

Calling such an API is OK for applications supporting
eggs. I don't see that as a problem.

>>Note that the pythonegg package approach would pretty much
>>remove the need for these .pth files.
> 
> Only in the sense that it would require reinventing them in a different 
> form.  :)

Not really - but we seem to have different views on whether
installers are good thing or not, so there's little point in
argueing over this.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 11 2005)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::