[Mobile-sig] python on android

Frank, Matthew I matthew.i.frank at intel.com
Thu Jan 29 18:53:54 CET 2015


I believe that I’ve seen this particular problem when building Python 3 for Android, but that it comes and goes.  The current open patch for Python 3.5 that you need to apply is: http://bugs.python.org/issue22625.

For Python 3 the problem is in Makefile.pre.in around line 750:

$(GRAMMAR_H): $(GRAMMAR_INPUT) $(PGEN)
Should be
$(GRAMMAR_H): $(GRAMMAR_INPUT) $(PGENSRCS)


For cross-building Python 3.4.2 from a Linux host I used the following configure command for the target build:

CPPFLAGS=-I/abs/path/to/source/Python-3.4.2/FIXLOCALE ../Python-3.4.2/configure --enable-shared --prefix=/absolute/path/to/android/install/of/python --build=x86_64-linux-gnu --host=i686-linux-android --disable-ipv6 ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no ac_cv_little_endian_double=yes --without-ensurepip

The –without-ensurepip, in particular was necessary because ensurepip needs to call the just-built Python interpreter (which won’t work when cross compiling).

There are some other things that need patching for Android (around the mbs to wcs functions and nl_langinfo() and such), but that’s the only problem with the Makefile that I’m aware of.

-Matt

From: Guido van Rossum [mailto:guido at python.org]
Sent: Thursday, January 29, 2015 9:54 AM
To: Russell Keith-Magee
Cc: mobile-sig
Subject: Re: [Mobile-sig] python on android

My only suggestion: try to understand *exactly* what the Makefile is doing at that point.
If it's really just invoking the wrong Python binary to do the byte-compilation, you could also skip that until you're further along. The byte-compilation is optional (it slows startup down a bit, but you currently don't even start up, so who cares. :-)

On Wed, Jan 28, 2015 at 10:58 PM, Russell Keith-Magee <russell at keith-magee.com<mailto:russell at keith-magee.com>> wrote:

Ok - so, yes, libPython is just copying and byte compiling. The catch is which version of Python and Python library you use to do the compiling.

To compile cross platform, you need to do 2 builds - a "host" build (x86 for most desktop purposes), and then a "target" build (the architecture of your phone).

(As an aside - in the case of iOS, you actually need to do 2 target builds - one for the simulator (which is x86, but with a different libc to OSX), and one for the device (which is an arm7/arm7s/arm64 triple target build) - but that's beside the point. The problems I'm having exist without this added complication).

So you do the host build, then you do the target build. The target build needs to call Python to invoke the compilation of modules etc; so you need to invoke the host python, but use the setup.py that was configured for the target Python.

This much I have working (at least, I think I do - I can't test yet because I don't have a working build, but all signs are positive).

But then, you get to libinstall, and you need to invoke Python to byte compile the pyc files. To do this, you need to invoke the host python, using the host Python's library, but over the *target* Python's sources. It's this last step that is tripping me up at the moment - I haven't worked out how to drive Autoconf to drive configure to pass in the set of arguments to invoke Python so that it will use the right binary and library with the right library tree. I keep end up running the iOS binary (which doesn't start), or the x86 binary with the iOS library tree (which is missing some parts that Python needs).

The problem manifests as an "ImportError: No module named _struct", because the compiled parts of the struct module aren't in the iOS tree (or, at least, aren't compiled for x86 in the iOS tree)

I appreciate that this isn't the easiest problem to debug over a mailing list. It's not even strictly "debugging" - it's a matter of working out what combination of arguments I need to pass in, and working backwards to the automake script. I've already made a bunch of changes to get this far, and I'm guessing any suggestions would have to be informed by what I've already done, which isn't a trivial thing to communicate without dumping a huge patch on your lap and saying "hey, fix this for me".

This is why, to date, I haven't sought out help - I've just been beavering through the problems one at a time :-) If you've got any suggestions on more productive ways to tackle this, I'm all ears. And, again, I'm happy to share what I've got to date if anyone is interested in helping out.

Yours,
Russ Magee %-)

On Thu, Jan 29, 2015 at 2:17 PM, Guido van Rossum <guido at python.org<mailto:guido at python.org>> wrote:
Well, libinstall is nearly 100 lines, not counting dependencies. OTOH it's just copying a lot of files and then byte-compiling them into .pyc files -- and .pyc files are portable. So perhaps you can go into a little more detail?

On Wed, Jan 28, 2015 at 9:10 PM, Russell Keith-Magee <russell at keith-magee.com<mailto:russell at keith-magee.com>> wrote:
Hi Guido,

I'll glady stop arguing about Kivy vs Toga. The only reason I brought it up at all is because I keep hearing arguments that seem to dispute that getting a libPython build working *is* the first step. You've now put that argument to bed, so I agree - lets move on.

For what it's worth, I've got a reasonable handle on how to compile libPython for mobile at this point - what I don't have is a good handle on is the intricacies of Python's build system, and in particular, how to drive Autoconf to support cross-platform builds.

I've almost worked out the patches to the Python 2.7.1 source tree to generate an iOS-compatible libPython. Once I've got that working, I'm planning to merge those changes up to the tip of 2.7 and 3.X, and submit those patches for inclusion in the source tree. However, at the moment, I'm hitting problems with cross-platform execution in the libinstall target; I'm happy to share what I have so far with anyone interested in collaborating.

Yours,
Russ Magee %-)

On Thu, Jan 29, 2015 at 12:31 PM, Guido van Rossum <guido at python.org<mailto:guido at python.org>> wrote:
Can we stop arguing about Kivy vs Toga and focus on the one thing that they have in common, the need for a working Python 3 port on Android and iOS (for a start)? This is apparently mostly a matter of solving a lot of small things with the build system, dependencies, improved config files, and getting stuff integrated upstream so it can be built out of the box, right? After that the layers 2-4 stuff can compete, but everybody wins when layer 1 is dealt with (even imperfectly). It's pretty sad that nobody apparently knows how to reproduce the build steps, and everybody just copies the one Python 2.7.{1,2} binary that someone built out of sheer willpower.

On Wed, Jan 28, 2015 at 8:04 PM, Russell Keith-Magee <russell at keith-magee.com<mailto:russell at keith-magee.com>> wrote:

Hi Bill,

On Thu, Jan 29, 2015 at 10:41 AM, Bill Janssen <janssen at parc.com<mailto:janssen at parc.com>> wrote:
Russell Keith-Magee <russell at keith-magee.com<mailto:russell at keith-magee.com>> wrote:

>  b) I don't like the Kivy build tools. They're a lot more complex than they
> need to be.

I didn't find it troublesome, but of course this wasn't my first rodeo.
I'd certainly agree it's not a push-button solution.  So, what would a
less complex system be like?

A less complex system is what Toga does.

 1. You use cookiecutter to generate a stub project. This gives you the full source tree for a project you can load into XCode (iOS), or build with ant (Android), including a "hello world" __main__.py

 2. You download the pre-compiled Python.framework for iOS, or libPython for Android, and copy it into a libs directory

 3. You start writing Python code, replacing the __main__.py with your own logic.

 4. You compile and deploy your project using XCode/ant.

Compare this with Kivy - My experience was spending a couple of days getting the Kivy build process to actually work - trying to find versions of the Android NDK that aren't being distributed any more (but are the only hard coded options in the build system), working out that the provided code doesn't work with the most recent versions of Cython, and sourcing libraries for all sorts of dependencies, so that I could compile SDL and a bunch of OpenGL stuff - none of which I needed. It took me a couple of days to get to "hello world" - and all because of something that could have been shipped as a pre-compiled binary.

> I'm going to guess the Kivy people are all Linux users, because
> they don't appear to have worked out that binary compatibility is a thing.

Sorry -- why is that a Linux thing?

If I want to distribute an app for OS/X or Windows, I give you an executable, and It Just Works (tm). Source *might* be provided as an option in the interest of being open source, but it's not how you distribute anything in practice. The "Linux way" for distribution is to distribute source, and tell you to compile it yourself. Distributing binaries is an afterthought, because ABI compatibility makes building and distributing binaries painful. Most projects don't have the infrastructure to distribute binaries for multiple platforms, so unless you can get the OS to provide a recent binary for you, you compile from source.

I see reflections of that bias here. Even though ABI compatibility exists for both iOS and Android as platforms, Kivy chooses to distribute as source.

> You don't need to have every user compile Python and the rest of the Kivy
> stack - you can just ship a binary library, and it will work on every phone
> with the same hardware (I know, because Toga does this. The Toga
> bootstrapping process is "clone this repo, and copy this file". You could
> reduce this to "clone this repo" if you were happy putting binary artefacts
> into version control.)

Sure.

>  c) Kivy's build tools are Python 2.7.1 only on iOS, and 2.7.2 on Android;

I believe the mobile platform packaging tools are still stuck at 2.7.
Kivy 1.8+ will run on Python 3 on desktop, though.

Yes - but in practice, the absence of Python 3 on Mobile means that Kivy on Mobile is Python 2 only, and an old version at that.

> and if you build them on OSX, when you're on the device, they report
> sys.platform as "darwin".

Seems like a bug; I imagine you're suggesting that the Kivy build
process should patch that file to return "android"?  Although I never
know what to look at to get that platform info correctly -- this is a
larger Python issue.

Yes, it's a bug (or at least a missing feature). The build system patches that Kivy use doesn't introduce anything for targeted builds (i.e., using an x86 platform to compile ARM64 binaries - which is what you're doing when you compile for mobile), and doesn't provide a platform definition for mobile.

And kivy.utils.platform seems to return the proper thing.

So... instead of they've introduced their own way to get access to information that Python already has a standard way of providing.

> Going back to my post

For those of you following along at home, here's Jeff's list (I had to
go and look it up).

Jeff?

>  1. A library build of Python
>  2. Templates to stub out a working Python project
>  3. Libraries to do bridge between native language environments and Python
>  (for me, that's Rubicon)
>  4. Libraries for utilising native system services (for me, that's Toga)

> - I agree with Kivy on layer 1, and I was able to use
> their build tools to bootstrap my own. However, I have very different
> opinions on layers 2-4.

Just to outline the Kivy approach to 2-4: Kivy doesn't really do 2 -- it
provides examples, and you're supposed to extrapolate from them.  I
guess that's a form of template.  For #3, there's Kivy's "pyjnius" (to
access Java via JNI) (https://github.com/kivy/pyobjus) and "pyobjus" (to
access Objective-C via runtime reflection)
(https://github.com/kivy/pyjnius).  For #4, as I said, there's "plyer"
(https://github.com/kivy/plyer).

I'd also include most of Kivy in #4 as well, because that's how they're tackling the widget issue.

We can have a separate discussion sometime about Django vs. Tornado :-).

So - a monkey knife-fight at dawn, then :-)

Yours,
Russ Magee %-)

_______________________________________________
Mobile-sig mailing list
Mobile-sig at python.org<mailto:Mobile-sig at python.org>
https://mail.python.org/mailman/listinfo/mobile-sig



--
--Guido van Rossum (python.org/~guido<http://python.org/~guido>)




--
--Guido van Rossum (python.org/~guido<http://python.org/~guido>)




--
--Guido van Rossum (python.org/~guido<http://python.org/~guido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/mobile-sig/attachments/20150129/7a4ebbd1/attachment-0001.html>


More information about the Mobile-sig mailing list