[Distutils] python eggs and lxml

[Phillip J. Eby]

At 11:13 AM 9/24/2005 +0200, Martijn Faassen wrote:
>I guess for Windows, I'd have make distutils run the configure script, 
>then extract the dlls it produces and stuff them in the egg somehow. Any 
>direction you'd point me towards for this?

You'll need to have your setup script determine programmatically that it's 
running on Windows, and then add data files and 'eager_resources' to the 
project accordingly.  In other words, to include shared libraries that 
aren't distutils-built extensions, you have to treat them as data files, 
but you must also list them in the 'eager_resources' setup keyword, as 
described here:

    http://peak.telecommunity.com/DevCenter/setuptools#automatic-resource-extraction

so that the shared libraries will be automatically extracted to disk before 
they're linked to by any C extensions.

You can do this on Unix platforms, too, of course, but the filenames will 
naturally be different.


>In this case, on Linux, I'd want to run the configure script when the egg 
>is installed instead of when it's created, and stuff the .so files in the 
>same place the egg is being installed to.

Eggs are not source distributions - they're prebuilt binaries for a 
particular platform.  EasyInstall will find and use source distributions if 
there's no binary for the system, so from a user point of view it certainly 
happens when the egg is "installed", but that's only because the actual egg 
is being *built* locally.

So, in any situation where you're not using the platform libraries, you'll 
need to follow the same embedding steps as for Windows - i.e., build the 
libraries and include them as data files.

If you're linking with the platform libraries, you don't need to include 
them as data files.

It's best, however, if you not think of this as install vs. create.  The 
only installation steps eggs have are:

  * adding them to sys.path
  * creating local wrappers to run programs contained in them

Everything else occurs when the egg is built, it's just that on Unix-y 
platforms you're more likely to be building the egg locally from source, 
rather than downloading a pre-built egg, if it contains C extensions.  For 
Python-only projects, of course, eggs are cross-platform and ready-to-use 
binaries.


>>Not all libraries can be bundled by source, of course.  Sometimes you 
>>really need to use whatever the "system version" is, for one reason or 
>>another.  Database clients, for example, are something you really really 
>>want to use the local version for.
>
>Right, there are competing use cases here. What I'd like is an easy 
>install for lxml that just works for people, without them having to worry 
>about the right lxml2 versions being installed, etc. On Windows this means 
>binaries, and on Linux this likely means it'll just compile upon install.

Or more precisely, it means compiling on egg build, regardless of 
platform.  It's just that for some platforms, you'll build the egg and 
distribute it instead of the end user building their own.

Probably the simplest way to do this is to subclass setuptools 'build_ext' 
command and extend run() to 'configure' and 'make' the libraries before 
proceeding normally.


>Some classes of people, like distributors and some sysadmins, care about 
>using the platform version of libxml2, and I'd also want to create an egg 
>that allows you to install against the platform libraries. Would this be 
>possible to be the same egg or would a different egg be needed? If a 
>different egg, how does this work with the dependency system? I.e. these 
>two eggs would be alternatives of each other dependency-wise.

An egg is not a project, or vice versa.  What you mean is that you want 
your project to be able to be built against different targets.  The eggs 
are what gets built.  If you've only used EasyInstall as a user, not a 
developer, this line is more blurry because EasyInstall downloads projects 
and builds the eggs for you if the author did not provide any eggs usable 
on your platform.  But there is a distinction, and on the authoring side 
you should focus on the project and its ability to be built for different 
targets.  Providing actual eggs is a convenience for your users, and does 
not enter into your development process.  It's strictly a 
deployment/publishing step.

Anyway, since the audience that wants to do this are more advanced users, 
and probably building your project individually rather than as part of a 
multi-project EasyInstall, I think it's reasonable to just have some value 
in a configuration file or environment variable or the standard distutils 
setup.cfg that allows skipping the configure+make.  For that matter, just 
documenting how to patch the setup script to use the platform libraries 
might be sufficient for that audience in the case of this kind of embedding.