[Mobile-sig] python on android

[Russell Keith-Magee]

Hi Wes,

A buildbot is possible, but complicated.

An Android buildbot could be hosted on any "unix like" system - you can get
the Android development kits for Linux and OS/X. However, that wouldn't
allow you to run the test suite for the mobile platform - because the
binary you build would need to run on the mobile platform itself. Android
does provide an emulator that would allow "on server" testing, but I don't
know how well it can be driven by remote control. My experience using it as
a developer has been "don't", because it's so painfully slow to start and
control.

On iOS, there are other limitations. The host compiler *must* be iOS,
because XCode isn't available for any other platform (AFAIK). Assuming that
can be accommodated, iOS provides a *simulator*, not an emulator - and the
simulator uses x86, not ARM. So, we'd need to run the test suite on the x86
simulator *and* a native device to be truly sure it works.

So - I don't know how this would fit into the existing Python build farm
infrastructure, but we'd need to add a minimum of 1 device (an iOS device)
to the build farm - ideally, 2 (an Android and an iOS device) - and
possibly a couple more if we want to cover testing on different OS
versions.

This is, of course, is somewhat premature - we need to get the build
working first :-)

Russ %-)


On Fri, Jan 30, 2015 at 2:17 AM, Wes Turner <wes.turner at gmail.com> wrote:

> Could there be buildbots for (each version of) each mobile platform?
>
> https://www.python.org/dev/buildbot/
>
> https://wiki.python.org/moin/BuildBot
> On Jan 29, 2015 11:54 AM, "Frank, Matthew I" <matthew.i.frank at intel.com>
> wrote:
>
>>  I believe that I’ve seen this particular problem when building Python 3
>> for Android, but that it comes and goes.  The current open patch for Python
>> 3.5 that you need to apply is: http://bugs.python.org/issue22625.
>>
>>
>>
>> For Python 3 the problem is in Makefile.pre.in around line 750:
>>
>>
>>
>> $(GRAMMAR_H): $(GRAMMAR_INPUT) $(PGEN)
>>
>> Should be
>>
>> $(GRAMMAR_H): $(GRAMMAR_INPUT) $(PGENSRCS)
>>
>>
>>
>>
>>
>> For cross-building Python 3.4.2 from a Linux host I used the following
>> configure command for the target build:
>>
>>
>>
>> CPPFLAGS=-I/abs/path/to/source/Python-3.4.2/FIXLOCALE
>> ../Python-3.4.2/configure --enable-shared
>> --prefix=/absolute/path/to/android/install/of/python
>> --build=x86_64-linux-gnu --host=i686-linux-android --disable-ipv6
>> ac_cv_file__dev_ptmx=no ac_cv_file__dev_ptc=no
>> ac_cv_little_endian_double=yes --without-ensurepip
>>
>>
>>
>> The –without-ensurepip, in particular was necessary because ensurepip
>> needs to call the just-built Python interpreter (which won’t work when
>> cross compiling).
>>
>>
>>
>> There are some other things that need patching for Android (around the
>> mbs to wcs functions and nl_langinfo() and such), but that’s the only
>> problem with the Makefile that I’m aware of.
>>
>>
>>
>> -Matt
>>
>>
>>
>> *From:* Guido van Rossum [mailto:guido at python.org]
>> *Sent:* Thursday, January 29, 2015 9:54 AM
>> *To:* Russell Keith-Magee
>> *Cc:* mobile-sig
>> *Subject:* Re: [Mobile-sig] python on android
>>
>>
>>
>> My only suggestion: try to understand *exactly* what the Makefile is
>> doing at that point.
>>
>> If it's really just invoking the wrong Python binary to do the
>> byte-compilation, you could also skip that until you're further along. The
>> byte-compilation is optional (it slows startup down a bit, but you
>> currently don't even start up, so who cares. :-)
>>
>>
>>
>> On Wed, Jan 28, 2015 at 10:58 PM, Russell Keith-Magee <
>> russell at keith-magee.com> wrote:
>>
>>
>>
>> Ok - so, yes, libPython is just copying and byte compiling. The catch is
>> which version of Python and Python library you use to do the compiling.
>>
>>
>>
>> To compile cross platform, you need to do 2 builds - a "host" build (x86
>> for most desktop purposes), and then a "target" build (the architecture of
>> your phone).
>>
>>
>>
>> (As an aside - in the case of iOS, you actually need to do 2 target
>> builds - one for the simulator (which is x86, but with a different libc to
>> OSX), and one for the device (which is an arm7/arm7s/arm64 triple target
>> build) - but that's beside the point. The problems I'm having exist without
>> this added complication).
>>
>>
>>
>> So you do the host build, then you do the target build. The target build
>> needs to call Python to invoke the compilation of modules etc; so you need
>> to invoke the host python, but use the setup.py that was configured for the
>> target Python.
>>
>>
>>
>> This much I have working (at least, I think I do - I can't test yet
>> because I don't have a working build, but all signs are positive).
>>
>>
>>
>> But then, you get to libinstall, and you need to invoke Python to byte
>> compile the pyc files. To do this, you need to invoke the host python,
>> using the host Python's library, but over the *target* Python's sources.
>> It's this last step that is tripping me up at the moment - I haven't worked
>> out how to drive Autoconf to drive configure to pass in the set of
>> arguments to invoke Python so that it will use the right binary and library
>> with the right library tree. I keep end up running the iOS binary (which
>> doesn't start), or the x86 binary with the iOS library tree (which is
>> missing some parts that Python needs).
>>
>>
>>
>> The problem manifests as an "ImportError: No module named _struct",
>> because the compiled parts of the struct module aren't in the iOS tree (or,
>> at least, aren't compiled for x86 in the iOS tree)
>>
>>
>>
>> I appreciate that this isn't the easiest problem to debug over a mailing
>> list. It's not even strictly "debugging" - it's a matter of working out
>> what combination of arguments I need to pass in, and working backwards to
>> the automake script. I've already made a bunch of changes to get this far,
>> and I'm guessing any suggestions would have to be informed by what I've
>> already done, which isn't a trivial thing to communicate without dumping a
>> huge patch on your lap and saying "hey, fix this for me".
>>
>>
>>
>> This is why, to date, I haven't sought out help - I've just been
>> beavering through the problems one at a time :-) If you've got any
>> suggestions on more productive ways to tackle this, I'm all ears. And,
>> again, I'm happy to share what I've got to date if anyone is interested in
>> helping out.
>>
>>
>>
>> Yours,
>>
>> Russ Magee %-)
>>
>>
>>
>> On Thu, Jan 29, 2015 at 2:17 PM, Guido van Rossum <guido at python.org>
>> wrote:
>>
>>  Well, libinstall is nearly 100 lines, not counting dependencies. OTOH
>> it's just copying a lot of files and then byte-compiling them into .pyc
>> files -- and .pyc files are portable. So perhaps you can go into a little
>> more detail?
>>
>>
>>
>> On Wed, Jan 28, 2015 at 9:10 PM, Russell Keith-Magee <
>> russell at keith-magee.com> wrote:
>>
>>  Hi Guido,
>>
>>
>>
>> I'll glady stop arguing about Kivy vs Toga. The only reason I brought it
>> up at all is because I keep hearing arguments that seem to dispute that
>> getting a libPython build working *is* the first step. You've now put that
>> argument to bed, so I agree - lets move on.
>>
>>
>>
>> For what it's worth, I've got a reasonable handle on how to compile
>> libPython for mobile at this point - what I don't have is a good handle on
>> is the intricacies of Python's build system, and in particular, how to
>> drive Autoconf to support cross-platform builds.
>>
>>
>>
>> I've almost worked out the patches to the Python 2.7.1 source tree to
>> generate an iOS-compatible libPython. Once I've got that working, I'm
>> planning to merge those changes up to the tip of 2.7 and 3.X, and submit
>> those patches for inclusion in the source tree. However, at the moment, I'm
>> hitting problems with cross-platform execution in the libinstall target;
>> I'm happy to share what I have so far with anyone interested in
>> collaborating.
>>
>>
>>
>> Yours,
>>
>> Russ Magee %-)
>>
>>
>>
>> On Thu, Jan 29, 2015 at 12:31 PM, Guido van Rossum <guido at python.org>
>> wrote:
>>
>>  Can we stop arguing about Kivy vs Toga and focus on the one thing that
>> they have in common, the need for a working Python 3 port on Android and
>> iOS (for a start)? This is apparently mostly a matter of solving a lot of
>> small things with the build system, dependencies, improved config files,
>> and getting stuff integrated upstream so it can be built out of the box,
>> right? After that the layers 2-4 stuff can compete, but everybody wins when
>> layer 1 is dealt with (even imperfectly). It's pretty sad that nobody
>> apparently knows how to reproduce the build steps, and everybody just
>> copies the one Python 2.7.{1,2} binary that someone built out of sheer
>> willpower.
>>
>>
>>
>> On Wed, Jan 28, 2015 at 8:04 PM, Russell Keith-Magee <
>> russell at keith-magee.com> wrote:
>>
>>
>>
>> Hi Bill,
>>
>>
>>
>> On Thu, Jan 29, 2015 at 10:41 AM, Bill Janssen <janssen at parc.com> wrote:
>>
>> Russell Keith-Magee <russell at keith-magee.com> wrote:
>>
>> >  b) I don't like the Kivy build tools. They're a lot more complex than
>> they
>> > need to be.
>>
>> I didn't find it troublesome, but of course this wasn't my first rodeo.
>> I'd certainly agree it's not a push-button solution.  So, what would a
>> less complex system be like?
>>
>>
>>
>> A less complex system is what Toga does.
>>
>>
>>
>>  1. You use cookiecutter to generate a stub project. This gives you the
>> full source tree for a project you can load into XCode (iOS), or build with
>> ant (Android), including a "hello world" __main__.py
>>
>>
>>
>>  2. You download the pre-compiled Python.framework for iOS, or libPython
>> for Android, and copy it into a libs directory
>>
>>
>>
>>  3. You start writing Python code, replacing the __main__.py with your
>> own logic.
>>
>>
>>
>>  4. You compile and deploy your project using XCode/ant.
>>
>>
>>
>> Compare this with Kivy - My experience was spending a couple of days
>> getting the Kivy build process to actually work - trying to find versions
>> of the Android NDK that aren't being distributed any more (but are the only
>> hard coded options in the build system), working out that the provided code
>> doesn't work with the most recent versions of Cython, and sourcing
>> libraries for all sorts of dependencies, so that I could compile SDL and a
>> bunch of OpenGL stuff - none of which I needed. It took me a couple of days
>> to get to "hello world" - and all because of something that could have been
>> shipped as a pre-compiled binary.
>>
>>
>>
>> > I'm going to guess the Kivy people are all Linux users, because
>> > they don't appear to have worked out that binary compatibility is a
>> thing.
>>
>> Sorry -- why is that a Linux thing?
>>
>>
>>
>> If I want to distribute an app for OS/X or Windows, I give you an
>> executable, and It Just Works (tm). Source *might* be provided as an option
>> in the interest of being open source, but it's not how you distribute
>> anything in practice. The "Linux way" for distribution is to distribute
>> source, and tell you to compile it yourself. Distributing binaries is an
>> afterthought, because ABI compatibility makes building and distributing
>> binaries painful. Most projects don't have the infrastructure to distribute
>> binaries for multiple platforms, so unless you can get the OS to provide a
>> recent binary for you, you compile from source.
>>
>>
>>
>> I see reflections of that bias here. Even though ABI compatibility exists
>> for both iOS and Android as platforms, Kivy chooses to distribute as source.
>>
>>
>>
>> > You don't need to have every user compile Python and the rest of the
>> Kivy
>> > stack - you can just ship a binary library, and it will work on every
>> phone
>> > with the same hardware (I know, because Toga does this. The Toga
>> > bootstrapping process is "clone this repo, and copy this file". You
>> could
>> > reduce this to "clone this repo" if you were happy putting binary
>> artefacts
>> > into version control.)
>>
>> Sure.
>>
>> >  c) Kivy's build tools are Python 2.7.1 only on iOS, and 2.7.2 on
>> Android;
>>
>> I believe the mobile platform packaging tools are still stuck at 2.7.
>> Kivy 1.8+ will run on Python 3 on desktop, though.
>>
>>
>>
>> Yes - but in practice, the absence of Python 3 on Mobile means that Kivy
>> on Mobile is Python 2 only, and an old version at that.
>>
>>
>>
>> > and if you build them on OSX, when you're on the device, they report
>> > sys.platform as "darwin".
>>
>> Seems like a bug; I imagine you're suggesting that the Kivy build
>> process should patch that file to return "android"?  Although I never
>> know what to look at to get that platform info correctly -- this is a
>> larger Python issue.
>>
>>
>>
>> Yes, it's a bug (or at least a missing feature). The build system patches
>> that Kivy use doesn't introduce anything for targeted builds (i.e., using
>> an x86 platform to compile ARM64 binaries - which is what you're doing when
>> you compile for mobile), and doesn't provide a platform definition for
>> mobile.
>>
>>
>>
>> And kivy.utils.platform seems to return the proper thing.
>>
>>
>>
>> So... instead of they've introduced their own way to get access to
>> information that Python already has a standard way of providing.
>>
>>
>>
>> > Going back to my post
>>
>> For those of you following along at home, here's Jeff's list (I had to
>> go and look it up).
>>
>>
>>
>> Jeff?
>>
>>
>>
>> >  1. A library build of Python
>> >  2. Templates to stub out a working Python project
>> >  3. Libraries to do bridge between native language environments and
>> Python
>> >  (for me, that's Rubicon)
>> >  4. Libraries for utilising native system services (for me, that's Toga)
>>
>> > - I agree with Kivy on layer 1, and I was able to use
>> > their build tools to bootstrap my own. However, I have very different
>> > opinions on layers 2-4.
>>
>> Just to outline the Kivy approach to 2-4: Kivy doesn't really do 2 -- it
>> provides examples, and you're supposed to extrapolate from them.  I
>> guess that's a form of template.  For #3, there's Kivy's "pyjnius" (to
>> access Java via JNI) (https://github.com/kivy/pyobjus) and "pyobjus" (to
>> access Objective-C via runtime reflection)
>> (https://github.com/kivy/pyjnius).  For #4, as I said, there's "plyer"
>> (https://github.com/kivy/plyer).
>>
>>
>>
>> I'd also include most of Kivy in #4 as well, because that's how they're
>> tackling the widget issue.
>>
>>
>>
>> We can have a separate discussion sometime about Django vs. Tornado :-).
>>
>>
>>
>> So - a monkey knife-fight at dawn, then :-)
>>
>>
>>
>> Yours,
>>
>> Russ Magee %-)
>>
>>
>>
>> _______________________________________________
>> Mobile-sig mailing list
>> Mobile-sig at python.org
>> https://mail.python.org/mailman/listinfo/mobile-sig
>>
>>
>>
>>
>> --
>>
>> --Guido van Rossum (python.org/~guido)
>>
>>
>>
>>
>>
>>
>> --
>>
>> --Guido van Rossum (python.org/~guido)
>>
>>
>>
>>
>>
>>
>> --
>>
>> --Guido van Rossum (python.org/~guido)
>>
>> _______________________________________________
>> Mobile-sig mailing list
>> Mobile-sig at python.org
>> https://mail.python.org/mailman/listinfo/mobile-sig
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/mobile-sig/attachments/20150130/202aeb02/attachment-0001.html>