[Distutils] special compiler options for only one file

[Erik Bray]

On Tue, Mar 19, 2013 at 7:54 PM, Andrew Dalke <dalke at dalkescientific.com> wrote:
> Hi all,
>
>   I have a Python extension which uses CPU-specific features,
> if available. This is done through a run-time check. If the
> hardware supports the POPCNT instruction then it selects one
> implementation of my inner loop, if SSSE3 is available then
> it selects another, otherwise it falls back to generic versions
> of my performance critical kernel. (Some 95%+ of the time is
> spent in this kernel.)
>
> Unfortunately, there's a failure mode I didn't expect. I
> use -mssse3 and -O3 to compile all of the C code, even though
> only one file needs that -mssse3 option.
>
> As a result, the other files are compiled with the expectation
> that SSSE3 will exist. This causes a segfault for the line
>
>      start_target_popcount = (int)(query_popcount * threshold);
>
> because the compiler used fisttpl, which is an SSSE-3 instruction.
> After all, I told it to assume that ssse3 exists.
>
> The Debian packager for my package recently ran into this problem,
> because the test machine has a gcc which understands -mssse3 but
> the machine itself has an older CPU without those instructions.
>
> I'm trying to come up with a solution that can be automated for
> the Debian distribution. I want a solution where the same binary
> can work on older machines and on newer ones
>
> Ideally I would like to say that only one file is compiled
> with the -mssse3 option. Since my selector code isn't part of this
> file, SSSE-3 code will never be executed unless the CPU supports is.
>
> However, I can't figure out any way to tell distutils that
> a set of compiler options are specific to a single file.
>
> Is that even possible?


One possible solution, albeit more complex code-wise than compiling
that single file as a static lib, would be to subclass the compiler
class you want to use and override its _compile() method, which is the
one responsible for compiling a single file.  You can then override
the arguments in there.  Look, for example, at
distutils.unixcompiler.UnixCCompiler._compile.

If you want to support different compiler implementations, you could
even detect at runtime which compiler was selected with the --compiler
option and subclass the appropriate implementation.  Also getting
distutils to accept a compiler subclass requires a little bit of
hackery, but it's not undoable.  If that optioni sounds viable to you
I can delve into more details about how to actually do it, as I've had
to do this sort of thing before myself.  So yes, it can be done.

Erik