[Python-Dev] Cross compiling Python (for Android)

[Frank, Matthew I]

This email is about my experience getting CPython (3.4.1) to
cross-compile and run on x86 Android (4.4.2 with sdk 19 and ndk-r9).
I know that Android is not a supported architecture (and I won't
regale you with stories about the complete locale and mbstowcs support
I had to borrow from FreeBSD to get it working).  The purpose of this
email is that several things I found are arguably "bugs" in the Python
build system or code when it comes to cross-compiling that are exposed
by Android's poor Posix support.  I'd like some advice about what kind
of patch (if any) would be most suitable for fixing the problems on
the Python side.

Just to be complete:  I'm configuring with

     CPPFLAGS=-I../my-locale ../Python-3.4.1/configure --enable-shared
     --prefix=/path/to/install/dir --build=x86_64-linux-gnu
     --host=i686-linux-android --disable-ipv6 ac_cv_file__dev_ptmx=no
     ac_cv_file__dev_ptc=no ac_cv_little_endian_double=yes

(The CPPFLAGS addition is to get the headers for my locale fixes
instead of the default Android ones.  ac_cv_file__dev_ptmx=no and
ac_cv_file__dev_ptc=no are because I don't have /dev/whatever on my
build machine.  ac_cv_little_endian_double is because configure for
cross builds can't figure out the endianness of doubles on the host
(because it is running on the build machine not the host.)  (For ARM
it would be ac_cv_mixed_endian_double=yes.)

I've gotten to the point where `make; make install` succeeds up to the
point of building something that runs on my Android system (from the
command line) and `python -m test` runs 388 tests, with 321 ok, 24
test failures and 43 tests skipped (the skips mostly due, I think, to
me not yet having installed the right cross-building support for
things like bz2 and dbm.)

1. `make` succeeds but `make install` always fails at the end with
   something having to do with being unable to run "ensurepip"
   (presumably because ensurepip requires modules that only run on the
   host, not the build module.)  So it seems this should be wrapped in
   a test for cross compilation, but I haven't looked at exactly what
   yet.  The error is:

   /linux-python/bin/python3.4: Error while finding spec for
   'ensurepip.__main__' (<class 'ImportError'>:
   /build-directory/build/lib.linux-i686-3.4/time.cpython-34m.so:
   wrong ELF class: ELFCLASS32); 'ensurepip' is a package and cannot
   be directly executed make: *** [install] Error 1

2. setup.py is missing -lm flag for several modules.  On Linux this
   problem is hidden because libm is already loaded by the executable
   calling dlopen(), but Android's loader deals with unknown symbols
   differently (searches only the libs explicitly linked against the
   module being loaded.)  http://bugs.python.org/issue21668 reports
   the problem for selectmodule (can't find ceil()) and timemodule
   (fmod() and floor()).  But there are at least two more: audioop
   fails to load because it needs floor() and ctypes_test fails to
   load because it needs sqrt().  I'll happily update the patch in
   21668.

   Is there any fundamental objection to adding the -lm flag to the
   link step where it is necessary?

3. What is ossaudiodev?  It tries to include "sys/soundcard.h", which
   I don't have on my system.   (The rule in setup.py is
   wrapped in a test for host of Linux/FreeBSD/Darwin, but Android x86
   gets configured with --host=i686-linux-android so to turn it off
   requires an extra test for "and not cross_compiling".)

   Can I just turn off ossaudiodev for cross compiling or might
   someone want it in a different type of cross build?  (In which case
   I think I'll have to write some kind autoconf rule for it, which I
   don't quite know how to do yet.)

4. Module _decimal is failing to compile.  The problem is that it has
   a header called memory.h.  Android's libc has the problem that
   /usr/include/stdlib.h includes <memory.h>.  But the build system
   puts -I. on the include path before the system dirs (as it should)
   so when compiling _decimal, Modules/_decimal/libmpdec/memory.h gets
   found instead of /usr/include/memory.h.  Shiz has a patch here:
   https://github.com/rave-engine/python3-android/blob/master/mk/python/3.3.5/p\
ython-3.3.5-android-libmpdec.patch
   (which renames memory.h -> mpmemory.h) but I don't know

   a.  Is there a tracker for this yet?  and
   b.  Is Shiz's fix the desired one or should I be looking for
       another approach?  (Maybe modifying the -I flags for the build
       of just the build of _decimal or something?)

5. I'm not sure what test configure is actually doing for gethostby*()
   in a cross-compile environment.  In any case Android has a bug
   where gethostbyaddr_r() is declared in the headers, but not
   actually implemented in libc.  So I have to modify my pyconfig.h by
   hand to define HAVE_GETHOSTBYNAME and undef HAVE_GETHOSTBYNAME_R
   and HAVE_GETHOSTBYNAME_R_6_ARG.

   Is there a variable (like ac_cv_little_endian_double) that I can
   give to `configure` to make it set HAVE_GETHOSTBYNAME* the way I
   need?  If so I've been unable to figure it out.

6. Android's <pwd.h> header mysteriously leaves the pw_gecos field out
   of struct passwd.  Is a fix like defining a new variable
   HAVE_BROKEN_GECOS_FIELD the appropriate way to go with this?  (If
   this is an okay solution then the patch to Modules/pwdmodule.c is
   shown below, but I still have to figure out how to patch
   configure.ac to test for the condition and set the variable
   appropriately, so a pointer to a similar block of code in
   configure.ac would be appreciated.)

Sorry for the TL;DR.  I appreciate your having taken the time to read
this far.

Thanks,
-Matt

Proposed patch for pwdmodule.c:

--- a/Modules/pwdmodule.c 2014-05-19 00:19:39.000000000 -0500
+++ b/Modules/pwdmodule.c       2014-10-21 18:00:35.676331205 -0500
@@ -57,6 +57,10 @@
   }
}

+#if defined(HAVE_BROKEN_GECOS_FIELD)
+static char fakePwGecos[256] = "";
+#endif
+
static PyObject *
mkpwent(struct passwd *p)
{
@@ -72,7 +76,11 @@
     SETS(setIndex++, p->pw_passwd);
     PyStructSequence_SET_ITEM(v, setIndex++, _PyLong_FromUid(p->pw_uid));
     PyStructSequence_SET_ITEM(v, setIndex++, _PyLong_FromGid(p->pw_gid));
+#if !defined(HAVE_BROKEN_GECOS_FIELD)
     SETS(setIndex++, p->pw_gecos);
+#else
+    SETS(setIndex++, fakePwGecos);
+#endif
     SETS(setIndex++, p->pw_dir);
     SETS(setIndex++, p->pw_shell);

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20141023/314dd74b/attachment.html>