[Distutils] Optional C extensions in packages

[Phillip J. Eby]

At 05:32 PM 2/1/2007 -0500, Jim Fulton wrote:
>I'm still worried about the ambiguous case when there are both
>platform-dependent and platform-independent eggs installed.

How would this happen?

I think you're trying to solve a broader problem than the one I'm trying to 
solve, which is that I'd like to make it possible for people who don't have 
working compilers (i.e. mostly Windows, with some Mac users and some people 
in virtual hosting environments) to install packages that contain C extensions.

In that scenario, you're going to *always* want to use this option to 
suppress optional extensions, because there isn't any way for you to build 
them.  But, you would presumably still want to know about packages that 
*require* their extensions to be built.


>I think you were proposing an easy_install option.  This helps when
>someone installs a distribution directly, but doesn't help when a
>distribution is installed as a dependency.

This would be an option to suppress compiling *all* optional C extensions, 
period.


>   It also doesn't help with
>controlling selection of eggs after installation.  And I think it
>doesn't make it easy to change one's mind.  For example, one might
>install an egg with extensions and then install one without
>extensions to debug a problem using the Python debugger.  Would the
>option let them do that?

The idea was that it would be a build-time option.


>Is it possible to control this as part of the requirement specification?
>Perhaps this could be some kind of standard extra?
>
>I'd strongly prefer to be able to control this via the requirements
>mechanism.  I'd like to be able to say that I want or don't want
>extensions as part of a requirement string.

Yeah, I see the benefit of that, certainly.  The problem is that we're 
trying to solve different problems.  I just want to make it *possible* to 
suppress building extensions during easy_install.

I'll give some more thought to what you're asking for.  I have an inkling 
of an idea, but the problems have to do with things like having to actually 
check the egg's contents to see if it meets requirements, and there are 
problems regarding the need to clean up the build/ directory if you change 
what features you build something with.

You see, setuptools has an undocumented 'feature' mechanism (which is still 
used by some PEAK projects) to control the inclusion of various packages, 
extensions, etc.  The main reason this is undocumented is because it turns 
out that it's fragile to specify what features to use or not use on the 
command line alone, due to some distutils' commands just taking whatever's 
in the build/ directory as gospel.

Anyway, that feature mechanism could probably be tied in to the 
requirements system, as long as there was a way to wipe the build/ 
directory whenever the features changed between runs of setup.py, and there 
was a way to list the features in the .egg-info, and pkg_resources was 
changed to query a distribution's "features" info when validating a 
requirement that includes "extras".

I'm a little concerned that this will incur additional disk access under 
various circumstances, unless there is some way to statically distinguish 
between extras that denote "features" and ones that indicate additional 
requirements.  Of course, matching a requirement against a distribution 
when the requirement doesn't list any extras, will not incur overhead.

I guess we could do something like this for 0.7.  One thing that concerns 
me, however, is that it potentially *increases* the amount of conflicts and 
confusion possible regarding a single egg, unless there's a way to include 
the features in the filename.  You can't tell just by looking at it, if it 
meets your needs.

In contrast, the benefit of my current proposal is that it's intended 
strictly for those circumstances where the eggs are *supposed to be* 
interchangeable except for platform-specificity and performance, and you 
should be able to at least tell from the filename which kind you have.  In 
the case where we allow other choices of features, you would need some kind 
of tool to tell you what features the egg was built with.

Maybe another possibility is to have *subprojects* instead, where a 
subproject is something built using the same setup.py, but has a distinct 
project name, like "PyProtocols-CExtensions" or "Twisted-Foo".  By default, 
perhaps such a multi-project setup script would run each subproject with 
its own build directory, and dump multiple eggs or source distributions 
into the dist/ directory.  This might take some munging of EasyInstall to 
support picking up the distributions produced when running the bdist_egg, 
but it might be doable.

The principal downsides to this approach are the doubling up of eggs 
involved, and the need to keep a precise match of versions between the 
packages.  In particular, if someone installs a new version of a package 
without its C extensions, and the C extensions still exist for an older 
version, it will end up importing the wrong extensions -- and it will be 
hard to tell what happened and why.  The package will just seem broken.

Sigh.  I guess at this point I don't really see a way to do optional 
extensions that doesn't turn into a crazy madhouse of support later.  It 
seems to me that at least the problems with my approach would at most boil 
down to, "how come this thing is so slow"?  :)