[issue5404] Cross-compiling Python

[Joshua Kinard]

Joshua Kinard <kumba at gentoo.org> added the comment:

Roumen,

I took a look at 4010, and tried your patch (as well as attempted to
apply the latter patch, but they changes are too great).  Neither one
helped resolve my issue, but I should probably explain the angle I'm
attacking this from so you have an idea of what I'm doing.

Currently, in Gentoo, we have a package file (called an ebuild) for
2.5.4, and this is what our primary package manager, Portage, links up
against, so that's why I'm working on getting 2.5.4 to cross-compile
properly.  I have no idea when we will move towards using 2.6.0.  I'm
trying to help our embedded team get cross-compiling working in some
packages so we can cross-compile a "base system" from a fast host.  This
all started after I decided to resurrect an old Cobalt RaQ2 from the
grave.  That machine is extremely limited -- ~144MB of RAM and a 250MHz
RM5231 MIPS Processor with no L2 Cache and a small L1 cache.  Compiling
is very, very slow, so the ability to cross-compile about ~83 base
packages in about 2 hours versus an entire day is a very desirable
thing.  It has a CHOST tuple of mipsel-unknown-linux-gnu.

Right now, the three most difficult packages to cross-compile are Perl,
gettext, and Python, in that order.  Perl is probably beyond all hope. 
I don't even know if I'll tackle it.  Gettext is just a nightmare, even
for a GNU package.  We bypass this by disabling NLS support for the
cross-build.  Python, I feel, is doable.  It's got some autotools links,
so it's got the capability.  Python code is also easy to read for me
(unlike Perl, but I digress), so I can sort of follow the logic in some
spots in setup.py.

We have two packages that assist in cross-compiling.  The first is our
'crossdev' package.  It is akin to what you'll find in Debian or via Dan
Kegel's crosstools setup.  It takes care of installing cross-toolchains
of varying degrees (C-Only, Glibc/Uclibc-based, etc..) on a Gentoo
system.  Next, is 'crossdev-wrappers', which is a set of wrapper scripts
for the 'emerge' command, Gentoo's main package manager tool in Portage.
 This helps to pass a lot of the basic environment variables so that
common configure scripts can correctly identify the build system and
target/host system types, or allow simple, Makefile-only packages build
using cross-compile toolchains.



So what happens on a cross-compile of Python, is first, the _socket
module fails to cross-compile because for that very specific compiler
invocation, it passes -I/usr/include, and _socket references the
sys/param.h include.  Which it finds in my host system's /usr/include
(an amd64 machine currently, so x86_64-pc-linux-gnu is the CHOST tuple).
 This will pull in x86 assembler language, which is misunderstood by a
mipsel-based toolchain.  I dug around, and found some patches that
re-ordered how configure and setup.py do things, and after utilizing a
tool in our ebuild system to create config.sub/config.guess files, this
helps to get Python's core to compile for the most part (I also applied
a fix for the printf %zd issue).

However, the default compiling logic for python uses gcc to both compile
the *.o files AND link into the final *.so modules.  One of the patches
I dug up (and merged) converts this logic to explicitly calling $CC for
compiling the *.o files, and then $LD to link them into *.so modules. 
This is where part of my problem lies, because the patch doesn't address
the fact that $CFLAGS gets passed to the $CC call (which is fine) AND to
the $LD call (which is not fine, because 'ld' takes a different set of
arguments) -- via (I think) the $OPT variable.

The other problem was libffi, under the _ctypes/ folder.  The setup.py
script explicitly clears configure arguments out via 'config_args = []'
code, and then invokes libffi's configure.  Without a --host=${CHOST}
value passed to it, libffi's configure incorrectly assumes it's running
on an x86_64-pc-linux-gnu system, not in a cross-compile environment for
mipsel-unknown-linux-gnu.  Thus, the configure script will go on to
build for that system, BUT call the mipsel-unknown-linux-gnu-gcc
compiler to compile its internal test code.  This fails, and Python's
default behavior is to ignore build failures and continue on.

So, after addressing those two problems, I've been trying to figure out
how to pass $LD a minimal set of LDFLAGS -- mostly -L commands, although
I'd like to just send it nothing but the *.o files and see how it
handles those by itself.  The logic in the makefile is really quite
weird, though, and it's not helped by the patches I'm mixing in, some of
which are a mix of several patches from this bug tracker.

If you want to take a look at the two main patches, I've uploaded them
to an external host, because I didn't want to litter the bug tracker
with incomplete patches that don't fully address the core problem,
especially since they're for older versions.


This one addresses or hacks around:
- LONG_BIT detection (hack)
- Printf %zd (merged another patch)
- Cross-compiling _socket (merged another patch)

http://dev.gentoo.org/~kumba/tmp/python-2.5.4-cross-compile-alpha5.patch


This one is a patch I wrote which gets libffi to compile.  It also
includes a small snippet of code to dump a variable out to a text file
-- that's debug code.  Ignore that.
http://dev.gentoo.org/~kumba/tmp/python-2.5.4-cross-compile-libffi-configure.patch

If you have tips or anything on how to better attack this, I'm all ears!

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue5404>
_______________________________________