Adding iOS/Android support to Python
Hi all, tl;dr - I'd like to add iOS and Android to the list of platforms that Python supports out-of-the-box, and I'm looking for guidance on the way forward (and assistance, if anyone is keen). Longer version: For the last 9 months or so, I've been working on tools to enable developers to write native mobile apps using Python. My motivation is to get Toga (http://pybee.org/toga) to the point where it's a viable way to write mobile apps. However, I'm not alone in the "Python on mobile" goal - Kivy has been around for a few years, with similar goals (although they've got some different ideas and priorities). To date, I've been using the groundwork established by Kivy, which takes the Python 2.7.1 (for iOS) and Python 2.7.2 (for Android) source code, applies a bunch of patches, and wraps those in a set of build scripts to yield Python libraries that can be used on iOS and Android. I'd like to start supporting more recent versions of Python - most importantly, Python 3. While I could certainly continue maintaining patches externally (and I imagine I'll have to if I want to maintain Python 2.7 support), I'd like to see that effort integrated into Python's repository. In particular, there are four areas where I can see changes required: 1) At least 2 new sys.platform definitions - "ios" and "android" (a third "iossimulator" platform might also be needed - the iOS simulator has enough differences that it can be helpful to be able to identify the platform) 2) Modifications to the build system to support cross-platform builds for mobile - AFAICT, cross platform builds currently only work on Linux and require readelf; iOS libraries can only be built on Mac, and building Android libraries don't require readelf (AFAICT). 3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms. 4) Support for building fat libraries for iOS. Xcode uses a tool called lipo to mash multiple platform libraries into a single library file. I don't imagine 1 & 2 are especially controversial. However, I would anticipate that 3 might raise some concerns, as the general Python "batteries included" philosophy would be modified to "unless you're on one of these platforms". Any guidance on whether this approach would be palatable? Is there any precedent for "module not supported on platform X" that I'm not aware of? Any platforms where Python is *only* available as a library? I have no idea how to tackle 4. To create a complete iOS build, you have to do at least 4 complete Python builds - a native system build (so you can run Python scripts that are part of the build), an i386 build (for the simulator), an armv7 build, and an arm64 build - and then the build products of the last three need to be combined into a single framework directory. Given three independent platform-specific build directories, it's relatively easy to construct a "fat" framework, but it isn't clear to me how that would fit into Python's build system. I can see that the build system has support for "universal" builds, which looks like an artefact of the OS X PPC->i386 conversion, but the build approach appears to be different to that required for iOS. Is there any documentation of the build process itself? Or is there anyone who can point me in the right direction? I'm also not clear how to tackle the testing and CI problems. Since you can't run iOS code on a non-iOS machine, it isn't entirely clear to me what an acceptance test would look like for a mobile build. So - am I completely wasting my time? Are patches for mobile platforms likely to be accepted into Python's repository? Or should I stick with the "external patches" approach? Any pointers, hints, guidance or assistance would be gratefully accepted. Yours, Russ Magee %-)
This idea seems to be in the air. Just yesterday someone posted a set of patches and suggestions to python-dev about making cross-compilation for Android easier. I expect most people in the core Python team might not be eager to jump on this bandwagon because they have no immediate need, but these mobile platforms are only getting faster more popular, and I think it would be a good idea to be prepared for when suddenly everybody wants to develop mobile apps in Python. On Fri, Oct 24, 2014 at 7:20 PM, Russell Keith-Magee < russell@keith-magee.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On 10/24/2014 11:06 PM, Guido van Rossum wrote:
Matt Frank, "Cross Compiling Python for Android", or close to that. He experimented with 3.4.1.
You (freakboy3742) and / or Matt Frank (WanderingLogic) should read the developer guide (https://docs.python.org/devguide), submit patches to the tracker for 3.5*, put each other as nosy (using the tracker ids above), review each other's patches, respond to other issue participants, especially core developers, and modify patches as needed to get them committed. Hopefully, one of you would have the goal of joining the developer team with commit privileges at least for android or ios development. * The first beta for 3.5 is scheduled for late May, 2015. At least a working prototype of any new feature for 3.5 should be in that release.
To add to what other said, many tests in the test suite are limited to certain platforms already -- skip if windows, skip if not windows, etc. Part of your patching would consist of adding skips for tests. We would also need to expand the possible Availability notes in the doc, adding something like 'Availability - desktop systems' and 'Availablity - mobile (Android/IOS)' if mobile-only features were needed. -- Terry Jan Reedy
Hi Terry, On Sun, Oct 26, 2014 at 1:14 AM, Terry Reedy <tjreedy@udel.edu> wrote:
I've seen that thread - thanks.
Good to know the timeline. If I get to the point where it looks like I have a viable patch, I'll open a ticket.
Ok - thanks for the pointers. Yours, Russ Magee %-)
3) Disabling certain modules on mobile platforms. Supporting modules
On Oct 25, 2014 4:22 AM, "Russell Keith-Magee" <russell@keith-magee.com> wrote: like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms. I would definitely be extremely interested in a python shell in android. One thing I feel are lacking on android are good advanced mathematical tools and and python shell with appropriate libraries could make a very powerful open-source tool for that. There have been some attempts at that already. I would also differentiate android and iOs more. Android seems to be betting on multi-core performance while iOs seems to be betting on single-chore performance. So while multiprocessing may not make much sense on iOs, I think it may be more sense on Android, especially if they move from 4 to 8 cores. Similarly, ultimately android is Linux-based, so things like readline should work, and it seems others have been able to get it to work, and bz2 seems to work fine on third-party android file managers.
On 2014-10-25, at 04:20 , Russell Keith-Magee <russell@keith-magee.com> wrote:
3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms.
However, I would anticipate that 3 might raise some concerns, as the general Python "batteries included" philosophy would be modified to "unless you're on one of these platforms". Any guidance on whether this approach would be palatable? Is there any precedent for "module not supported on platform X" that I'm not aware of?
* IIRC, Python will not setup e.g. bz2 if there's no libbz2 on the system, likewise ssl and libssl, so the underlying library is not available on a platform it wouldn't be entirely surprising for it to be missing in the corresponding Python * there are a number of modules which are only available on a restricted number of platforms, though for the most part they're either windows-specific[0] or unix-specific[1]. There are also limitations within modules e.g. os.path.relpath is documented as only available on Unix and Windows, samepath and sameopenfile used to be Unix-only and are now Unix+Windows * some modules also have very different contents or behaviors based on the platform e.g. ssl, signal or mmap So there seems to be a precedent for stdlib modules entirely missing on some platforms, or having restricted or platform-specific content. [0] https://docs.python.org/3/library/windows.html [1] https://docs.python.org/3/library/unix.html
On 25 October 2014 12:20, Russell Keith-Magee <russell@keith-magee.com> wrote:
Maintaining support for new versions of existing platforms is already within scope for Python 2.7, so if you get this working for Python 3, it *might* be possible to make the case for 2.x. However, it would only be feasible to make that determination after we had a better idea as to the magnitude of the changes involved, and the risks introduced for other platforms.
Yes, platform specific features are actually quite common (search the docs for "Availability:"). It's just the fact that folks writing cross-platform code tend to avoid those modules/features, so they're largely only known to folks writing lower level (or otherwise platform specific) code.
Any platforms where Python is *only* available as a library?
The CPython main.c is a fairly thin shim around Py_Main anyway, so I don't see it as a big deal whether the standard CLI is included as part of the mobile builds.
So - am I completely wasting my time? Are patches for mobile platforms likely to be accepted into Python's repository?
I think both iOS and Android are well and truly established enough now as platforms to be worth targeting directly. For the CI problem, it may be worth approaching Xamarin, as setting up the infrastructure for doing that ourselves looks like a formidable engineering challenge we don't currently have the relevant skillsets to provide. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi Nick, On Sat, Oct 25, 2014 at 11:25 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Good to know - I'll keep this in mind.
Good point - I hadn't thought about the OS-specific bits of the existing std lib.
Ok - I'll keep that in mind as I work on my patches.
Is this said in the context of "we (Python) have an existing relationship with Xamarin, and we should lean on those contacts", or "we should really develop a relationship with Xamarin"? Yours, Russ Magee %-)
On 26 October 2014 09:36, Russell Keith-Magee <russell@keith-magee.com> wrote:
The latter - although as Antoine noted, we *do* have some nominally supported platforms that don't have buildbots. That configuration just means that *other* core devs aren't likely to be too worried about breaking the platform - more of the burden will fall back on the folks actually keeping that platform running. However, it did occur to me that running in the iOS and Android simulators should be feasible on a normal x86_64 system, assuming you can figure out a way to run the regression test suite without a normal console. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Oct 26, 2014 at 7:07 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Understood. That's essentially the status quo anyway, just with the code outside the Python repository. However, it did occur to me that running in the iOS and Android
Both iOS and Android have a console that could be used to display test output - it's just not visible on the phone itself. The "Hello world" you get at the end of the template project I linked to earlier produces a couple of lines on the debug console. In theory, I guess it should be possible to build a full iOS/Android "App" whose only purpose is to run the Python test suite. I haven't tried this; I'll put it on my todo list. Regarding the test process: It's certainly possible to run the iOS simulator and Android emulator on x86_64 machines -- but the iOS simulator is a completely different CPU architecture to that used on a device. It might be useful as a sanity check, but you'd still want to run the actual iOS build on an actual iOS device to confirm it was fully working as expected. Yours, Russ Magee %-)
Nick Coghlan <ncoghlan@gmail.com> wrote:
I think both iOS and Android are well and truly established enough now as platforms to be worth targeting directly.
Does Apple allow Python apps in AppStore now? I thought they had a ban on anything not written in Objective-C or Swift? Sturla
On Tue, Oct 28, 2014 at 8:52 AM, Sturla Molden <sturla.molden@gmail.com> wrote:
Hi Sturla, That restriction was lifted a while back. Apple now allows apps that embed other languages, with some restrictions about the source of the scripts being run - for example, your app must contain all the scripts it's going to use, you can't download new scripts from the web. Evidence of this is Kivy - there are a number of apps written using Kivy on the App Store: https://github.com/kivy/kivy/wiki/List-of-Kivy-Projects Of course, there's no guarantee this won't change back in the future. Working at the whims and fancies of the AppStore T&C's will always be a risk. :-) Yours, Russ Magee %-)
On Sat, 25 Oct 2014 10:20:05 +0800 Russell Keith-Magee <russell@keith-magee.com> wrote:
"ios" and "android" sound ok to me. A specific platform for the simulator sounds wrong, though. If it simulates iOS, it should certainly simulate its platform identification.
I don't know anything about cross-compiling. The best is that you try to whip up a patch and submit it on the tracker. Also, I suggest you tackle one system at a time. It will be easier to review and accept patches if they target only Android or only iOS, as opposed to both at the same time.
Special support for this isn't necessary. If setup.py doesn't find the necessary external dependencies, it will skip building those modules. If OTOH those modules are built, you can remove them when creating your installer (I imagine you won't use "setup.py install" on those platforms).
No idea. Perhaps start by tackling the first 3 points. When you arrive at 4, open an issue and our resident Mac experts can help you :-)
We don't have acceptance tests for all platforms. You'll have to check that Python still builds on a regular basis. Regards Antoine.
On Sat, Oct 25, 2014 at 11:42 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
The iOS simulator is an odd beast. Yes, it's *mostly* like the actual device, but there are a couple of places where you need to be aware that you're on a simulator, and deal with it accordingly. For one thing, it's an i386 "device", rather than ARM; there are also some subtle differences in the way it deals with device "hardware", like the camera, or interacts with built-in apps (like the Photo library). However, for the sake of simplicity, I'll start by assuming that there's just one "iOS" platform.
Understood.
Good point - I hadn't thought of that. Yours, Russ Magee %-)
On Oct 24, 2014, at 19:20, Russell Keith-Magee <russell@keith-magee.com> wrote:
I'd like to start supporting more recent versions of Python - most importantly, Python 3. While I could certainly continue maintaining patches externally (and I imagine I'll have to if I want to maintain Python 2.7 support), I'd like to see that effort integrated into Python's repository.
People have already been making Python 3 builds for iOS, wrapping them in mini-IDEs, and selling them on the App Store for quite some time. Have you talked to Xiaowen Huang or any of the others? As far as I know, most of them build Python as a static lib, and link in the C extensions from the stdlib, to get around the issues with shared-lib frameworks before iOS 8. And I'm not sure how they deal with third-party modules (at least one of them includes numpy, for example). You also may want to talk to whoever maintains the ports for jailbroken iOS, because that's just a standard-as-possible build, with the command line interpreter and pip and everything.
In particular, there are four areas where I can see changes required:
1) At least 2 new sys.platform definitions - "ios" and "android" (a third "iossimulator" platform might also be needed - the iOS simulator has enough differences that it can be helpful to be able to identify the platform)
There are other fields in platform and sys that can be used to distinguish iOS from the simulator. And, since you need to check iOS simulator or real iOS vs. everything else at least as often as you need to distinguish them from each other, I'm not sure it's important whether it's treated as a separate platform or a different version of the same platform--as long as there's some documented and guaranteed way to distinguish them.
3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms.
How is multiprocessing difficult to provide on mobile? iOS has had fork, posix_spawn, semaphores, etc. since the first release with an SDK, although you'd get rejected from the App Store for using them in the early days. Of course Apple would prefer people to use XPC via grand central, but that doesn't mean you need to make it impossible to use fork instead; at most, you just need to warn people to read the App Store guidelines. In the opposite direction, it might be worth looking at the "extra batteries" in Apple's OS X builds. At least PyObjC seems like something you'd want to get working on iOS.
I have no idea how to tackle 4. To create a complete iOS build, you have to do at least 4 complete Python builds - a native system build (so you can run Python scripts that are part of the build), an i386 build (for the simulator), an armv7 build, and an arm64 build - and then the build products of the last three need to be combined into a single framework directory.
Given three independent platform-specific build directories, it's relatively easy to construct a "fat" framework, but it isn't clear to me how that would fit into Python's build system. I can see that the build system has support for "universal" builds, which looks like an artefact of the OS X PPC->i386 conversion, but the build approach appears to be different to that required for iOS.
It's still perfectly possible to build Python universal on OS X. In fact, both the python.org builds and Apple's pre-installed builds are fat i386/x86_64. iOS does put some limitations on things, especially when building shared libs, but if you need to fall back to building each target separately and lipo'ing them together manually, I'm pretty sure there are a few third-party libs that already use that workaround. Didn't PyObjC used to build that way?
In article <AF04D1E2-058D-4322-91C7-E544E837A1D7@yahoo.com>, Andrew Barnert <abarnert@yahoo.com.dmarc.invalid> wrote:
I have no personal experience with iOS builds but I *think* the complexity of multi-arch builds is handled by the Apple-supplied compiler tools in the same way for iOS as is done for OS X builds. I assume that, like for OS X builds, the compiler driver handles the multiple builds and lipos automatically under the covers based on the requested archs, the SDK, and platform tools in use. As Andrew notes, we have long fully supported SDK builds and all of the universal archs for OS X (again using Apple's compiler chain distributed via Xcode and/or the Command Line Tools). I wouldn't expect that to be a big problem to adapt for iOS. (Cross building on platforms other than OS X is another matter.) The main issue, as always, is getting buy-in to "standardize" support for these platforms and all of the issues of long-term maintenance. Applying a set of changes is just the first step. We need to have subject experts for code and documentation. We'd need to have buildbots to give some assurance that we aren't breaking things. We need to have people willing to test pre-releases. All these things are doable and probably desirable. But they don't happen automatically. The changes that have already been added to the Python build system to support various kinds of cross-builds are today largely undocumented, unsupported by python-dev, and untested by our buildbots. We need to solve these problems for them as well before we end up with more bitrot. -- Ned Deily, nad@acm.org
On Sun, Oct 26, 2014 at 7:52 AM, Ned Deily <nad@acm.org> wrote:
That would be a nice if it were true, but unfortunately, it isn't AFAICT. Under OS/X, to compile a single .c file, you make a single call to clang; if you reference an external library, you only need to point at a single library file. The system libraries provided by Apple are "fat", and contain multiple architectures. If there's an analogous set of options for iOS, I haven't been able to find them. In particular, there's a very distinct split between the x86 simulator libraries, and the ARM device libraries. In order to compile a fat iOS library, you have to do *at least* 2 compiles - one for i386, and one for ARM - because there are two completely separate - and incompatible - system libraries. On top of that, the OS/X build is self hosting - once you've compiled python.exe, you can use that executable to compile the python sources and to run the setup.py script that compiles module extensions. Netiher of the iOS builds are self hosting - you have to have a working native python compiler before you . It looks like it *might* be possible to compile the multiple *device* architectures (armv7, armv7s, arm64) on a single compile pass, but I haven't quite worked out the incantation yet. In the meantime, I'm having to do 5 complete builds (native, i386 simulator, armv7, armv7s, and arm64) to get a single library that can be used in a standalone project. I
... and that's what I've been trying to gauge with my post. I have a personal need for "Python on Mobile" support, and I don't think I'm alone in this. I can maintain a set of my own build tools and patch sets that provide Python support to my own satisfaction, but I think there's value in having that capability in Python's trunk. However, I've got enough experience maintaining a large Open Source Python project to know that the story doesn't stop with submitting a patch that scratches my own itch - there's the long term maintenance to consider. And in this case, there's all the additional complications of getting testing, CI etc working on a mobile platform. So - at the core of all of this is the big question: Should I bother? If I was to do the work, is the core team interested? And if so, what constitutes "the work" from Python's perspective? Is it just adding platform libraries and some patches to the build system to allow cross-platform builds for iOS and Android? Or do I need to clean up and document the entire cross-platform build system? And a quick meta-question - Is there *any* documentation of the current build system? I've had a poke around, and I can't find anything that seems on topic. The "Setup and Usage' guide (or maybe the Windows FAW) is the closest I can find. Yours, Russ Magee %-)
In article <CAJxq84_zofWVVGiin_JMO+aKuD9i2=6rZ9_R8WzOFnE4tZJ5zA@mail.gmail.com>, Russell Keith-Magee <russell@keith-magee.com> wrote:
This is all really off-topic but, after a little bit of experimentation, I think it is the case that you can compile and link multiple iOS archs (armv7, armv7s, arm64) in a single cc/clang call just as you can for OS X archs; you just need to specify the proper sdk via xcrun (or the other supported methods) and the proper -arch values to clang. One way: xcrun --sdk iphoneos8.1 cc -arch arm64 -arch armv7 -arch armv7s -c test.c But I think you are correct that it is not possible to do iOS builds and iOS simulators builds in one step as they are definitely different platforms from the tool chain perspective (just os OS X is a different platforms); I'm not sure how important that is. $ xcodebuild -showsdks OS X SDKs: OS X 10.9 -sdk macosx10.9 OS X 10.10 -sdk macosx10.10 iOS SDKs: iOS 8.1 -sdk iphoneos8.1 iOS Simulator SDKs: Simulator - iOS 8.1 -sdk iphonesimulator8.1
As I mentioned, there already is some, mostly undocumented support for cross-building with separation of build host tools from target host tools. Presumably that could be extended as needed.
That's a really good question. I'm not sure how you get an answer. Perhaps one approach is through producing a PEP and getting at least one core developer to champion it. Guido's positive response is important. There would need to be some sort to consensus over on python-dev as well, I think.
The Python Developer's Guide is probably the best place to start. But you'll find very little about cross-building since, AFAIK, there are few core developers who are familiar with the topic. Matthias Klose added the most recent cross-build changes. https://docs.python.org/devguide/ -- Ned Deily, nad@acm.org
On Sun, Oct 26, 2014 at 4:58 AM, Andrew Barnert <abarnert@yahoo.com> wrote:
To clarify - this isn't a theoretical thing - I've got a Python 2.7 static library running on non-jailbroken iOS and Android devices, and I've published the tools (and builds) to let others do the same: https://github.com/pybee/Python-iOS-support https://github.com/pybee/Python-Android-support And I've published cookiecutter templates for iOS and Android projects if you want to write your apps in Python: https://github.com/pybee/Python-iOS-template https://github.com/pybee/Python-Android-template The support libraries are mostly patches and build system tweaks, and at present, they're all for stale versions of Python 2.7. I'm on Python-ideas because I want to move as much as possible back upstream into Python's sources.
That's a good point; I'll investigate what can be done using other fields in platform and sys.
I looked at PyObjC; I was unsatisfied with the performance, and it has a native compiled component. I ended up writing my own ObjC bindings, based off an initial collection of code from Pyglet: https://github.com/pybee/rubicon-objc This uses ctypes, so there's no compiled component.
Yes, but AFAICT, this is done by leveraging the ability of clang to compile a binary that supports multiple architectures at once.. The iOS toolchain doesn't do that (AFAICT) - you need to build separately and lipo the products together. OSX only needs a single compiler pass; iOS needs multiple passes. iOS does put some limitations on things, especially when building shared
I'll take a closer look at PyObjC's build process and see what I can work out. Yours, Russ Magee %-)
On 25 Oct 2014, at 22:58, Andrew Barnert <abarnert@yahoo.com.dmarc.invalid> wrote:
On Oct 24, 2014, at 19:20, Russell Keith-Magee <russell@keith-magee.com> wrote:
Nope. PyObjC directly builds fat binaries without explicitly using lipo. That’s for PyObjC on OSX, I have no idea if there still is a port for PyObjC to iOS and how that’s build. Ronald
Hi Stefan, On Sun, Oct 26, 2014 at 4:18 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
We've had some initial introductory conversations on Twitter, but we're not working closely on anything at the moment. There's certainly potential for collaboration, though. Yours, Russ Magee %-)
On 27 October 2014 11:53, Russell Keith-Magee <russell@keith-magee.com> wrote:
Some other folks potentially worth contacting: * Jonathan Hosmer (creator of http://pythonforios.com/) * Ole Zorn (creator of http://omz-software.com/pythonista/) * Karl Traunmüller (creator of http://computableapp.com/) If working on better iOS and Android upstream happens to help bring those apps to Android as well, that would be a rather nice outcome :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
This idea seems to be in the air. Just yesterday someone posted a set of patches and suggestions to python-dev about making cross-compilation for Android easier. I expect most people in the core Python team might not be eager to jump on this bandwagon because they have no immediate need, but these mobile platforms are only getting faster more popular, and I think it would be a good idea to be prepared for when suddenly everybody wants to develop mobile apps in Python. On Fri, Oct 24, 2014 at 7:20 PM, Russell Keith-Magee < russell@keith-magee.com> wrote:
-- --Guido van Rossum (python.org/~guido)
On 10/24/2014 11:06 PM, Guido van Rossum wrote:
Matt Frank, "Cross Compiling Python for Android", or close to that. He experimented with 3.4.1.
You (freakboy3742) and / or Matt Frank (WanderingLogic) should read the developer guide (https://docs.python.org/devguide), submit patches to the tracker for 3.5*, put each other as nosy (using the tracker ids above), review each other's patches, respond to other issue participants, especially core developers, and modify patches as needed to get them committed. Hopefully, one of you would have the goal of joining the developer team with commit privileges at least for android or ios development. * The first beta for 3.5 is scheduled for late May, 2015. At least a working prototype of any new feature for 3.5 should be in that release.
To add to what other said, many tests in the test suite are limited to certain platforms already -- skip if windows, skip if not windows, etc. Part of your patching would consist of adding skips for tests. We would also need to expand the possible Availability notes in the doc, adding something like 'Availability - desktop systems' and 'Availablity - mobile (Android/IOS)' if mobile-only features were needed. -- Terry Jan Reedy
Hi Terry, On Sun, Oct 26, 2014 at 1:14 AM, Terry Reedy <tjreedy@udel.edu> wrote:
I've seen that thread - thanks.
Good to know the timeline. If I get to the point where it looks like I have a viable patch, I'll open a ticket.
Ok - thanks for the pointers. Yours, Russ Magee %-)
3) Disabling certain modules on mobile platforms. Supporting modules
On Oct 25, 2014 4:22 AM, "Russell Keith-Magee" <russell@keith-magee.com> wrote: like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms. I would definitely be extremely interested in a python shell in android. One thing I feel are lacking on android are good advanced mathematical tools and and python shell with appropriate libraries could make a very powerful open-source tool for that. There have been some attempts at that already. I would also differentiate android and iOs more. Android seems to be betting on multi-core performance while iOs seems to be betting on single-chore performance. So while multiprocessing may not make much sense on iOs, I think it may be more sense on Android, especially if they move from 4 to 8 cores. Similarly, ultimately android is Linux-based, so things like readline should work, and it seems others have been able to get it to work, and bz2 seems to work fine on third-party android file managers.
On 2014-10-25, at 04:20 , Russell Keith-Magee <russell@keith-magee.com> wrote:
3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms.
However, I would anticipate that 3 might raise some concerns, as the general Python "batteries included" philosophy would be modified to "unless you're on one of these platforms". Any guidance on whether this approach would be palatable? Is there any precedent for "module not supported on platform X" that I'm not aware of?
* IIRC, Python will not setup e.g. bz2 if there's no libbz2 on the system, likewise ssl and libssl, so the underlying library is not available on a platform it wouldn't be entirely surprising for it to be missing in the corresponding Python * there are a number of modules which are only available on a restricted number of platforms, though for the most part they're either windows-specific[0] or unix-specific[1]. There are also limitations within modules e.g. os.path.relpath is documented as only available on Unix and Windows, samepath and sameopenfile used to be Unix-only and are now Unix+Windows * some modules also have very different contents or behaviors based on the platform e.g. ssl, signal or mmap So there seems to be a precedent for stdlib modules entirely missing on some platforms, or having restricted or platform-specific content. [0] https://docs.python.org/3/library/windows.html [1] https://docs.python.org/3/library/unix.html
On 25 October 2014 12:20, Russell Keith-Magee <russell@keith-magee.com> wrote:
Maintaining support for new versions of existing platforms is already within scope for Python 2.7, so if you get this working for Python 3, it *might* be possible to make the case for 2.x. However, it would only be feasible to make that determination after we had a better idea as to the magnitude of the changes involved, and the risks introduced for other platforms.
Yes, platform specific features are actually quite common (search the docs for "Availability:"). It's just the fact that folks writing cross-platform code tend to avoid those modules/features, so they're largely only known to folks writing lower level (or otherwise platform specific) code.
Any platforms where Python is *only* available as a library?
The CPython main.c is a fairly thin shim around Py_Main anyway, so I don't see it as a big deal whether the standard CLI is included as part of the mobile builds.
So - am I completely wasting my time? Are patches for mobile platforms likely to be accepted into Python's repository?
I think both iOS and Android are well and truly established enough now as platforms to be worth targeting directly. For the CI problem, it may be worth approaching Xamarin, as setting up the infrastructure for doing that ourselves looks like a formidable engineering challenge we don't currently have the relevant skillsets to provide. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi Nick, On Sat, Oct 25, 2014 at 11:25 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Good to know - I'll keep this in mind.
Good point - I hadn't thought about the OS-specific bits of the existing std lib.
Ok - I'll keep that in mind as I work on my patches.
Is this said in the context of "we (Python) have an existing relationship with Xamarin, and we should lean on those contacts", or "we should really develop a relationship with Xamarin"? Yours, Russ Magee %-)
On 26 October 2014 09:36, Russell Keith-Magee <russell@keith-magee.com> wrote:
The latter - although as Antoine noted, we *do* have some nominally supported platforms that don't have buildbots. That configuration just means that *other* core devs aren't likely to be too worried about breaking the platform - more of the burden will fall back on the folks actually keeping that platform running. However, it did occur to me that running in the iOS and Android simulators should be feasible on a normal x86_64 system, assuming you can figure out a way to run the regression test suite without a normal console. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Sun, Oct 26, 2014 at 7:07 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
Understood. That's essentially the status quo anyway, just with the code outside the Python repository. However, it did occur to me that running in the iOS and Android
Both iOS and Android have a console that could be used to display test output - it's just not visible on the phone itself. The "Hello world" you get at the end of the template project I linked to earlier produces a couple of lines on the debug console. In theory, I guess it should be possible to build a full iOS/Android "App" whose only purpose is to run the Python test suite. I haven't tried this; I'll put it on my todo list. Regarding the test process: It's certainly possible to run the iOS simulator and Android emulator on x86_64 machines -- but the iOS simulator is a completely different CPU architecture to that used on a device. It might be useful as a sanity check, but you'd still want to run the actual iOS build on an actual iOS device to confirm it was fully working as expected. Yours, Russ Magee %-)
Nick Coghlan <ncoghlan@gmail.com> wrote:
I think both iOS and Android are well and truly established enough now as platforms to be worth targeting directly.
Does Apple allow Python apps in AppStore now? I thought they had a ban on anything not written in Objective-C or Swift? Sturla
On Tue, Oct 28, 2014 at 8:52 AM, Sturla Molden <sturla.molden@gmail.com> wrote:
Hi Sturla, That restriction was lifted a while back. Apple now allows apps that embed other languages, with some restrictions about the source of the scripts being run - for example, your app must contain all the scripts it's going to use, you can't download new scripts from the web. Evidence of this is Kivy - there are a number of apps written using Kivy on the App Store: https://github.com/kivy/kivy/wiki/List-of-Kivy-Projects Of course, there's no guarantee this won't change back in the future. Working at the whims and fancies of the AppStore T&C's will always be a risk. :-) Yours, Russ Magee %-)
On Sat, 25 Oct 2014 10:20:05 +0800 Russell Keith-Magee <russell@keith-magee.com> wrote:
"ios" and "android" sound ok to me. A specific platform for the simulator sounds wrong, though. If it simulates iOS, it should certainly simulate its platform identification.
I don't know anything about cross-compiling. The best is that you try to whip up a patch and submit it on the tracker. Also, I suggest you tackle one system at a time. It will be easier to review and accept patches if they target only Android or only iOS, as opposed to both at the same time.
Special support for this isn't necessary. If setup.py doesn't find the necessary external dependencies, it will skip building those modules. If OTOH those modules are built, you can remove them when creating your installer (I imagine you won't use "setup.py install" on those platforms).
No idea. Perhaps start by tackling the first 3 points. When you arrive at 4, open an issue and our resident Mac experts can help you :-)
We don't have acceptance tests for all platforms. You'll have to check that Python still builds on a regular basis. Regards Antoine.
On Sat, Oct 25, 2014 at 11:42 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
The iOS simulator is an odd beast. Yes, it's *mostly* like the actual device, but there are a couple of places where you need to be aware that you're on a simulator, and deal with it accordingly. For one thing, it's an i386 "device", rather than ARM; there are also some subtle differences in the way it deals with device "hardware", like the camera, or interacts with built-in apps (like the Photo library). However, for the sake of simplicity, I'll start by assuming that there's just one "iOS" platform.
Understood.
Good point - I hadn't thought of that. Yours, Russ Magee %-)
On Oct 24, 2014, at 19:20, Russell Keith-Magee <russell@keith-magee.com> wrote:
I'd like to start supporting more recent versions of Python - most importantly, Python 3. While I could certainly continue maintaining patches externally (and I imagine I'll have to if I want to maintain Python 2.7 support), I'd like to see that effort integrated into Python's repository.
People have already been making Python 3 builds for iOS, wrapping them in mini-IDEs, and selling them on the App Store for quite some time. Have you talked to Xiaowen Huang or any of the others? As far as I know, most of them build Python as a static lib, and link in the C extensions from the stdlib, to get around the issues with shared-lib frameworks before iOS 8. And I'm not sure how they deal with third-party modules (at least one of them includes numpy, for example). You also may want to talk to whoever maintains the ports for jailbroken iOS, because that's just a standard-as-possible build, with the command line interpreter and pip and everything.
In particular, there are four areas where I can see changes required:
1) At least 2 new sys.platform definitions - "ios" and "android" (a third "iossimulator" platform might also be needed - the iOS simulator has enough differences that it can be helpful to be able to identify the platform)
There are other fields in platform and sys that can be used to distinguish iOS from the simulator. And, since you need to check iOS simulator or real iOS vs. everything else at least as often as you need to distinguish them from each other, I'm not sure it's important whether it's treated as a separate platform or a different version of the same platform--as long as there's some documented and guaranteed way to distinguish them.
3) Disabling certain modules on mobile platforms. Supporting modules like linuxaudiodev, ossaudiodev, readline, curses, idle and tkinter on mobile platforms doesn't make much sense; modules likes bsddb and bz2 are difficult to support due to library dependencies; and the need for modules like multiprocessing is arguable (and difficult to support on mobile). Even providing a Python executable/shell is arguable for these platforms.
How is multiprocessing difficult to provide on mobile? iOS has had fork, posix_spawn, semaphores, etc. since the first release with an SDK, although you'd get rejected from the App Store for using them in the early days. Of course Apple would prefer people to use XPC via grand central, but that doesn't mean you need to make it impossible to use fork instead; at most, you just need to warn people to read the App Store guidelines. In the opposite direction, it might be worth looking at the "extra batteries" in Apple's OS X builds. At least PyObjC seems like something you'd want to get working on iOS.
I have no idea how to tackle 4. To create a complete iOS build, you have to do at least 4 complete Python builds - a native system build (so you can run Python scripts that are part of the build), an i386 build (for the simulator), an armv7 build, and an arm64 build - and then the build products of the last three need to be combined into a single framework directory.
Given three independent platform-specific build directories, it's relatively easy to construct a "fat" framework, but it isn't clear to me how that would fit into Python's build system. I can see that the build system has support for "universal" builds, which looks like an artefact of the OS X PPC->i386 conversion, but the build approach appears to be different to that required for iOS.
It's still perfectly possible to build Python universal on OS X. In fact, both the python.org builds and Apple's pre-installed builds are fat i386/x86_64. iOS does put some limitations on things, especially when building shared libs, but if you need to fall back to building each target separately and lipo'ing them together manually, I'm pretty sure there are a few third-party libs that already use that workaround. Didn't PyObjC used to build that way?
In article <AF04D1E2-058D-4322-91C7-E544E837A1D7@yahoo.com>, Andrew Barnert <abarnert@yahoo.com.dmarc.invalid> wrote:
I have no personal experience with iOS builds but I *think* the complexity of multi-arch builds is handled by the Apple-supplied compiler tools in the same way for iOS as is done for OS X builds. I assume that, like for OS X builds, the compiler driver handles the multiple builds and lipos automatically under the covers based on the requested archs, the SDK, and platform tools in use. As Andrew notes, we have long fully supported SDK builds and all of the universal archs for OS X (again using Apple's compiler chain distributed via Xcode and/or the Command Line Tools). I wouldn't expect that to be a big problem to adapt for iOS. (Cross building on platforms other than OS X is another matter.) The main issue, as always, is getting buy-in to "standardize" support for these platforms and all of the issues of long-term maintenance. Applying a set of changes is just the first step. We need to have subject experts for code and documentation. We'd need to have buildbots to give some assurance that we aren't breaking things. We need to have people willing to test pre-releases. All these things are doable and probably desirable. But they don't happen automatically. The changes that have already been added to the Python build system to support various kinds of cross-builds are today largely undocumented, unsupported by python-dev, and untested by our buildbots. We need to solve these problems for them as well before we end up with more bitrot. -- Ned Deily, nad@acm.org
On Sun, Oct 26, 2014 at 7:52 AM, Ned Deily <nad@acm.org> wrote:
That would be a nice if it were true, but unfortunately, it isn't AFAICT. Under OS/X, to compile a single .c file, you make a single call to clang; if you reference an external library, you only need to point at a single library file. The system libraries provided by Apple are "fat", and contain multiple architectures. If there's an analogous set of options for iOS, I haven't been able to find them. In particular, there's a very distinct split between the x86 simulator libraries, and the ARM device libraries. In order to compile a fat iOS library, you have to do *at least* 2 compiles - one for i386, and one for ARM - because there are two completely separate - and incompatible - system libraries. On top of that, the OS/X build is self hosting - once you've compiled python.exe, you can use that executable to compile the python sources and to run the setup.py script that compiles module extensions. Netiher of the iOS builds are self hosting - you have to have a working native python compiler before you . It looks like it *might* be possible to compile the multiple *device* architectures (armv7, armv7s, arm64) on a single compile pass, but I haven't quite worked out the incantation yet. In the meantime, I'm having to do 5 complete builds (native, i386 simulator, armv7, armv7s, and arm64) to get a single library that can be used in a standalone project. I
... and that's what I've been trying to gauge with my post. I have a personal need for "Python on Mobile" support, and I don't think I'm alone in this. I can maintain a set of my own build tools and patch sets that provide Python support to my own satisfaction, but I think there's value in having that capability in Python's trunk. However, I've got enough experience maintaining a large Open Source Python project to know that the story doesn't stop with submitting a patch that scratches my own itch - there's the long term maintenance to consider. And in this case, there's all the additional complications of getting testing, CI etc working on a mobile platform. So - at the core of all of this is the big question: Should I bother? If I was to do the work, is the core team interested? And if so, what constitutes "the work" from Python's perspective? Is it just adding platform libraries and some patches to the build system to allow cross-platform builds for iOS and Android? Or do I need to clean up and document the entire cross-platform build system? And a quick meta-question - Is there *any* documentation of the current build system? I've had a poke around, and I can't find anything that seems on topic. The "Setup and Usage' guide (or maybe the Windows FAW) is the closest I can find. Yours, Russ Magee %-)
In article <CAJxq84_zofWVVGiin_JMO+aKuD9i2=6rZ9_R8WzOFnE4tZJ5zA@mail.gmail.com>, Russell Keith-Magee <russell@keith-magee.com> wrote:
This is all really off-topic but, after a little bit of experimentation, I think it is the case that you can compile and link multiple iOS archs (armv7, armv7s, arm64) in a single cc/clang call just as you can for OS X archs; you just need to specify the proper sdk via xcrun (or the other supported methods) and the proper -arch values to clang. One way: xcrun --sdk iphoneos8.1 cc -arch arm64 -arch armv7 -arch armv7s -c test.c But I think you are correct that it is not possible to do iOS builds and iOS simulators builds in one step as they are definitely different platforms from the tool chain perspective (just os OS X is a different platforms); I'm not sure how important that is. $ xcodebuild -showsdks OS X SDKs: OS X 10.9 -sdk macosx10.9 OS X 10.10 -sdk macosx10.10 iOS SDKs: iOS 8.1 -sdk iphoneos8.1 iOS Simulator SDKs: Simulator - iOS 8.1 -sdk iphonesimulator8.1
As I mentioned, there already is some, mostly undocumented support for cross-building with separation of build host tools from target host tools. Presumably that could be extended as needed.
That's a really good question. I'm not sure how you get an answer. Perhaps one approach is through producing a PEP and getting at least one core developer to champion it. Guido's positive response is important. There would need to be some sort to consensus over on python-dev as well, I think.
The Python Developer's Guide is probably the best place to start. But you'll find very little about cross-building since, AFAIK, there are few core developers who are familiar with the topic. Matthias Klose added the most recent cross-build changes. https://docs.python.org/devguide/ -- Ned Deily, nad@acm.org
On Sun, Oct 26, 2014 at 4:58 AM, Andrew Barnert <abarnert@yahoo.com> wrote:
To clarify - this isn't a theoretical thing - I've got a Python 2.7 static library running on non-jailbroken iOS and Android devices, and I've published the tools (and builds) to let others do the same: https://github.com/pybee/Python-iOS-support https://github.com/pybee/Python-Android-support And I've published cookiecutter templates for iOS and Android projects if you want to write your apps in Python: https://github.com/pybee/Python-iOS-template https://github.com/pybee/Python-Android-template The support libraries are mostly patches and build system tweaks, and at present, they're all for stale versions of Python 2.7. I'm on Python-ideas because I want to move as much as possible back upstream into Python's sources.
That's a good point; I'll investigate what can be done using other fields in platform and sys.
I looked at PyObjC; I was unsatisfied with the performance, and it has a native compiled component. I ended up writing my own ObjC bindings, based off an initial collection of code from Pyglet: https://github.com/pybee/rubicon-objc This uses ctypes, so there's no compiled component.
Yes, but AFAICT, this is done by leveraging the ability of clang to compile a binary that supports multiple architectures at once.. The iOS toolchain doesn't do that (AFAICT) - you need to build separately and lipo the products together. OSX only needs a single compiler pass; iOS needs multiple passes. iOS does put some limitations on things, especially when building shared
I'll take a closer look at PyObjC's build process and see what I can work out. Yours, Russ Magee %-)
On 25 Oct 2014, at 22:58, Andrew Barnert <abarnert@yahoo.com.dmarc.invalid> wrote:
On Oct 24, 2014, at 19:20, Russell Keith-Magee <russell@keith-magee.com> wrote:
Nope. PyObjC directly builds fat binaries without explicitly using lipo. That’s for PyObjC on OSX, I have no idea if there still is a port for PyObjC to iOS and how that’s build. Ronald
Hi Stefan, On Sun, Oct 26, 2014 at 4:18 PM, Stefan Behnel <stefan_ml@behnel.de> wrote:
We've had some initial introductory conversations on Twitter, but we're not working closely on anything at the moment. There's certainly potential for collaboration, though. Yours, Russ Magee %-)
On 27 October 2014 11:53, Russell Keith-Magee <russell@keith-magee.com> wrote:
Some other folks potentially worth contacting: * Jonathan Hosmer (creator of http://pythonforios.com/) * Ole Zorn (creator of http://omz-software.com/pythonista/) * Karl Traunmüller (creator of http://computableapp.com/) If working on better iOS and Android upstream happens to help bring those apps to Android as well, that would be a rather nice outcome :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (12)
-
Andrew Barnert -
Antoine Pitrou -
Guido van Rossum -
Masklinn -
Ned Deily -
Nick Coghlan -
Ronald Oussoren -
Russell Keith-Magee -
Stefan Behnel -
Sturla Molden -
Terry Reedy -
Todd