Support of the Android platform
The following note is a proposal to add the support of the Android platform. The note is easier to read with clickable links at https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst Motivations =========== * Android is ubiquitous. * This would be the first platform supported by Python that is cross-compiled, thanks to many contributors. * Although the Android operating system is linux, it is different from most linux platforms, for example it does not use GNU libc and runs SELinux in enforcing mode. Therefore supporting this platform would make Python more robust and also would allow testing it on arm 64-bit processors. * Python running on Android is also a handheld calculator, a successor of the slide rule and the `HP 41`_. Current status ============== * The Python test suite succeeds when run on Android emulators using buildbot strenuous settings with the following architectures on API 24: x86, x86_64, armv7 and arm64. * The `Android build system`_ is described in another section. * The `buildmaster-config PR 26`_ proposes to update ``master.cfg`` to enable buildbots to run a given Android API and architecture on the emulators. * The Android emulator is actually ``qemu``, so the test suites for x86 and x86_64 last about the same time as the test suite run natively when the processor of the build system is of the x86 family. The test suites for the arm architectures last much longer: about 8 hours for arm64 and 10 hours for armv7 on a four years old laptop. * The changes that have been made to achieve this status are listed in `bpo-26865`_, the Android meta-issue. * Given the cpu resources required to run the test suite on the arm emulators, it may be difficult to find a contributed buildbot worker. So it remains to find the hardware to run these buildbots. Proposal ======== Support the Android platform on API 24 [1]_ for the x86_64, armv7 and arm64 architectures built with NDK 14b. *API 24* * API 21 is the first version to provide usable support for wide characters and where SELinux is run in enforcing mode. * API 22 introduces an annoying bug on the linker that prints something like this when python is started:: ``WARNING: linker: libpython3.6m.so.1.0: unused DT entry: type 0x6ffffffe arg 0x14554``. The `termux`_ Android terminal emulator describes this problem at the end of its `termux-packages`_ gitlab page and has implemented a ``termux-elf-cleaner`` tool to strip the useless entries from the ELF header of executables. * API 24 is the first version where the `adb`_ shell is run on the emulator as a ``shell`` user instead of the ``root`` user previously, and the first version that supports arm64. *x86_64* It seems that no handheld device exists using that architecture. It is supported because the x86_64 Android emulator runs fast and therefore is a good candidate as a buildbot worker. *NDK 14b* This release of the NDK is the first one to use `Unified headers`_ fixing numerous problems that had been fixed by updating the Python configure script until now (those changes have been reverted by now). Android idiosyncrasies ====================== * The default shell is ``/system/bin/sh``. * The file system layout is not a traditional unix layout, there is no ``/tmp`` for example. Most directories have user restricted access, ``/sdcard`` is mounted as ``noexec`` for example. * The (java) applications are allocated a unix user id and a subdirectory on ``/data/data``. * SELinux is run in enforcing mode. * Shared memory and semaphores are not supported. * The default encoding is UTF-8. Android build system ==================== The Android build system is implemented at `bpo-30386`_ with `PR 1629`_ and is documented by its `README`_. It provides the following features: * To build a distribution for a device or an emulator with a given API level and a given architecture. * To start the emulator and + install the distribution + start a remote interactive shell + or run remotely a python command + or run remotely the buildbottest * Run gdb on the python process that is running on the emulator with python pretty-printing. The build system adds the ``Android/`` directory and the ``configure-android`` script to the root of the Python source directory on the master branch without modifying any other file. The build system can be installed, upgraded (i.e. the SDK and NDK) and run remotely, through ssh for example. The following external libraries, when they are configured in the build system, are downloaded from the internet and cross-compiled (only once, on the first run of the build system) before the cross-compilation of the extension modules: * ``ncurses`` * ``readline`` * ``sqlite`` * ``libffi`` * ``openssl``, the cross-compilation of openssl fails on x86_64 and arm64 and this step is skipped on those architectures. The following extension modules are disabled by adding them to the ``*disabled*`` section of ``Modules/Setup``: * ``_uuid``, Android has no uuid/uuid.h header. * ``grp`` some grp.h functions are not declared. * ``_crypt``, Android does not have crypt.h. * ``_ctypes`` on x86_64 where all long double tests fail (`bpo-32202`_) and on arm64 (see `bpo-32203`_). .. [1] On Wikipedia `Android version history`_ lists the correspondence between API level, commercial name and version for each release. It also provides information on the global Android version distribution, see the two charts on top. .. _`README`: https://github.com/xdegaye/cpython/blob/bpo-30386/Android/README.rst .. _`bpo-26865`: https://bugs.python.org/issue26865 .. _`bpo-30386`: https://bugs.python.org/issue30386 .. _`bpo-32202`: https://bugs.python.org/issue32202 .. _`bpo-32203`: https://bugs.python.org/issue32203 .. _`PR 1629`: https://github.com/python/cpython/pull/1629 .. _`buildmaster-config PR 26`: https://github.com/python/buildmaster-config/pull/26 .. _`Android version history`: https://en.wikipedia.org/wiki/Android_version_history .. _`termux`: https://termux.com/ .. _`termux-packages`: https://gitlab.com/jbwhips883/termux-packages .. _`adb`: https://developer.android.com/studio/command-line/adb.html .. _`Unified headers`: https://android.googlesource.com/platform/ndk.git/+/ndk-r14-release/docs/Uni... .. _`HP 41`: https://en.wikipedia.org/wiki/HP-41C .. vim:filetype=rst:tw=78:ts=8:sts=2:sw=2:et:
While the note from a technical standpoint is interest, Xavier, I don't quite see what needs to be done to support Android at this point. Are you simply asking we add Android API 24 as an official platform? Or permission to add your note to the Misc/ directory? Basically what are you wanting to see happen? :) On Sun, 10 Dec 2017 at 06:19 Xavier de Gaye <xdegaye@gmail.com> wrote:
The following note is a proposal to add the support of the Android platform.
The note is easier to read with clickable links at https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst
Motivations ===========
* Android is ubiquitous. * This would be the first platform supported by Python that is cross-compiled, thanks to many contributors. * Although the Android operating system is linux, it is different from most linux platforms, for example it does not use GNU libc and runs SELinux in enforcing mode. Therefore supporting this platform would make Python more robust and also would allow testing it on arm 64-bit processors. * Python running on Android is also a handheld calculator, a successor of the slide rule and the `HP 41`_.
Current status ==============
* The Python test suite succeeds when run on Android emulators using buildbot strenuous settings with the following architectures on API 24: x86, x86_64, armv7 and arm64. * The `Android build system`_ is described in another section. * The `buildmaster-config PR 26`_ proposes to update ``master.cfg`` to enable buildbots to run a given Android API and architecture on the emulators. * The Android emulator is actually ``qemu``, so the test suites for x86 and x86_64 last about the same time as the test suite run natively when the processor of the build system is of the x86 family. The test suites for the arm architectures last much longer: about 8 hours for arm64 and 10 hours for armv7 on a four years old laptop. * The changes that have been made to achieve this status are listed in `bpo-26865`_, the Android meta-issue. * Given the cpu resources required to run the test suite on the arm emulators, it may be difficult to find a contributed buildbot worker. So it remains to find the hardware to run these buildbots.
Proposal ========
Support the Android platform on API 24 [1]_ for the x86_64, armv7 and arm64 architectures built with NDK 14b.
*API 24* * API 21 is the first version to provide usable support for wide characters and where SELinux is run in enforcing mode.
* API 22 introduces an annoying bug on the linker that prints something like this when python is started::
``WARNING: linker: libpython3.6m.so.1.0: unused DT entry: type 0x6ffffffe arg 0x14554``.
The `termux`_ Android terminal emulator describes this problem at the end of its `termux-packages`_ gitlab page and has implemented a ``termux-elf-cleaner`` tool to strip the useless entries from the ELF header of executables.
* API 24 is the first version where the `adb`_ shell is run on the emulator as a ``shell`` user instead of the ``root`` user previously, and the first version that supports arm64.
*x86_64* It seems that no handheld device exists using that architecture. It is supported because the x86_64 Android emulator runs fast and therefore is a good candidate as a buildbot worker.
*NDK 14b* This release of the NDK is the first one to use `Unified headers`_ fixing numerous problems that had been fixed by updating the Python configure script until now (those changes have been reverted by now).
Android idiosyncrasies ======================
* The default shell is ``/system/bin/sh``. * The file system layout is not a traditional unix layout, there is no ``/tmp`` for example. Most directories have user restricted access, ``/sdcard`` is mounted as ``noexec`` for example. * The (java) applications are allocated a unix user id and a subdirectory on ``/data/data``. * SELinux is run in enforcing mode. * Shared memory and semaphores are not supported. * The default encoding is UTF-8.
Android build system ====================
The Android build system is implemented at `bpo-30386`_ with `PR 1629`_ and is documented by its `README`_. It provides the following features:
* To build a distribution for a device or an emulator with a given API level and a given architecture. * To start the emulator and + install the distribution + start a remote interactive shell + or run remotely a python command + or run remotely the buildbottest * Run gdb on the python process that is running on the emulator with python pretty-printing.
The build system adds the ``Android/`` directory and the ``configure-android`` script to the root of the Python source directory on the master branch without modifying any other file. The build system can be installed, upgraded (i.e. the SDK and NDK) and run remotely, through ssh for example.
The following external libraries, when they are configured in the build system, are downloaded from the internet and cross-compiled (only once, on the first run of the build system) before the cross-compilation of the extension modules:
* ``ncurses`` * ``readline`` * ``sqlite`` * ``libffi`` * ``openssl``, the cross-compilation of openssl fails on x86_64 and arm64 and this step is skipped on those architectures.
The following extension modules are disabled by adding them to the ``*disabled*`` section of ``Modules/Setup``:
* ``_uuid``, Android has no uuid/uuid.h header. * ``grp`` some grp.h functions are not declared. * ``_crypt``, Android does not have crypt.h. * ``_ctypes`` on x86_64 where all long double tests fail (`bpo-32202`_) and on arm64 (see `bpo-32203`_).
.. [1] On Wikipedia `Android version history`_ lists the correspondence between API level, commercial name and version for each release. It also provides information on the global Android version distribution, see the two charts on top.
.. _`README`: https://github.com/xdegaye/cpython/blob/bpo-30386/Android/README.rst .. _`bpo-26865`: https://bugs.python.org/issue26865 .. _`bpo-30386`: https://bugs.python.org/issue30386 .. _`bpo-32202`: https://bugs.python.org/issue32202 .. _`bpo-32203`: https://bugs.python.org/issue32203 .. _`PR 1629`: https://github.com/python/cpython/pull/1629 .. _`buildmaster-config PR 26`: https://github.com/python/buildmaster-config/pull/26 .. _`Android version history`: https://en.wikipedia.org/wiki/Android_version_history .. _`termux`: https://termux.com/ .. _`termux-packages`: https://gitlab.com/jbwhips883/termux-packages .. _`adb`: https://developer.android.com/studio/command-line/adb.html .. _`Unified headers`: https://android.googlesource.com/platform/ndk.git/+/ndk-r14-release/docs/Uni... .. _`HP 41`: https://en.wikipedia.org/wiki/HP-41C .. vim:filetype=rst:tw=78:ts=8:sts=2:sw=2:et: _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
Maybe it should be a PEP? On Dec 10, 2017 12:29, "Brett Cannon" <brett@python.org> wrote:
While the note from a technical standpoint is interest, Xavier, I don't quite see what needs to be done to support Android at this point. Are you simply asking we add Android API 24 as an official platform? Or permission to add your note to the Misc/ directory? Basically what are you wanting to see happen? :)
On Sun, 10 Dec 2017 at 06:19 Xavier de Gaye <xdegaye@gmail.com> wrote:
The following note is a proposal to add the support of the Android platform.
The note is easier to read with clickable links at https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst
Motivations ===========
* Android is ubiquitous. * This would be the first platform supported by Python that is cross-compiled, thanks to many contributors. * Although the Android operating system is linux, it is different from most linux platforms, for example it does not use GNU libc and runs SELinux in enforcing mode. Therefore supporting this platform would make Python more robust and also would allow testing it on arm 64-bit processors. * Python running on Android is also a handheld calculator, a successor of the slide rule and the `HP 41`_.
Current status ==============
* The Python test suite succeeds when run on Android emulators using buildbot strenuous settings with the following architectures on API 24: x86, x86_64, armv7 and arm64. * The `Android build system`_ is described in another section. * The `buildmaster-config PR 26`_ proposes to update ``master.cfg`` to enable buildbots to run a given Android API and architecture on the emulators. * The Android emulator is actually ``qemu``, so the test suites for x86 and x86_64 last about the same time as the test suite run natively when the processor of the build system is of the x86 family. The test suites for the arm architectures last much longer: about 8 hours for arm64 and 10 hours for armv7 on a four years old laptop. * The changes that have been made to achieve this status are listed in `bpo-26865`_, the Android meta-issue. * Given the cpu resources required to run the test suite on the arm emulators, it may be difficult to find a contributed buildbot worker. So it remains to find the hardware to run these buildbots.
Proposal ========
Support the Android platform on API 24 [1]_ for the x86_64, armv7 and arm64 architectures built with NDK 14b.
*API 24* * API 21 is the first version to provide usable support for wide characters and where SELinux is run in enforcing mode.
* API 22 introduces an annoying bug on the linker that prints something like this when python is started::
``WARNING: linker: libpython3.6m.so.1.0: unused DT entry: type 0x6ffffffe arg 0x14554``.
The `termux`_ Android terminal emulator describes this problem at the end of its `termux-packages`_ gitlab page and has implemented a ``termux-elf-cleaner`` tool to strip the useless entries from the ELF header of executables.
* API 24 is the first version where the `adb`_ shell is run on the emulator as a ``shell`` user instead of the ``root`` user previously, and the first version that supports arm64.
*x86_64* It seems that no handheld device exists using that architecture. It is supported because the x86_64 Android emulator runs fast and therefore is a good candidate as a buildbot worker.
*NDK 14b* This release of the NDK is the first one to use `Unified headers`_ fixing numerous problems that had been fixed by updating the Python configure script until now (those changes have been reverted by now).
Android idiosyncrasies ======================
* The default shell is ``/system/bin/sh``. * The file system layout is not a traditional unix layout, there is no ``/tmp`` for example. Most directories have user restricted access, ``/sdcard`` is mounted as ``noexec`` for example. * The (java) applications are allocated a unix user id and a subdirectory on ``/data/data``. * SELinux is run in enforcing mode. * Shared memory and semaphores are not supported. * The default encoding is UTF-8.
Android build system ====================
The Android build system is implemented at `bpo-30386`_ with `PR 1629`_ and is documented by its `README`_. It provides the following features:
* To build a distribution for a device or an emulator with a given API level and a given architecture. * To start the emulator and + install the distribution + start a remote interactive shell + or run remotely a python command + or run remotely the buildbottest * Run gdb on the python process that is running on the emulator with python pretty-printing.
The build system adds the ``Android/`` directory and the ``configure-android`` script to the root of the Python source directory on the master branch without modifying any other file. The build system can be installed, upgraded (i.e. the SDK and NDK) and run remotely, through ssh for example.
The following external libraries, when they are configured in the build system, are downloaded from the internet and cross-compiled (only once, on the first run of the build system) before the cross-compilation of the extension modules:
* ``ncurses`` * ``readline`` * ``sqlite`` * ``libffi`` * ``openssl``, the cross-compilation of openssl fails on x86_64 and arm64 and this step is skipped on those architectures.
The following extension modules are disabled by adding them to the ``*disabled*`` section of ``Modules/Setup``:
* ``_uuid``, Android has no uuid/uuid.h header. * ``grp`` some grp.h functions are not declared. * ``_crypt``, Android does not have crypt.h. * ``_ctypes`` on x86_64 where all long double tests fail (`bpo-32202`_) and on arm64 (see `bpo-32203`_).
.. [1] On Wikipedia `Android version history`_ lists the correspondence between API level, commercial name and version for each release. It also provides information on the global Android version distribution, see the two charts on top.
.. _`README`: https://github.com/xdegaye/cpython/blob/bpo-30386/ Android/README.rst .. _`bpo-26865`: https://bugs.python.org/issue26865 .. _`bpo-30386`: https://bugs.python.org/issue30386 .. _`bpo-32202`: https://bugs.python.org/issue32202 .. _`bpo-32203`: https://bugs.python.org/issue32203 .. _`PR 1629`: https://github.com/python/cpython/pull/1629 .. _`buildmaster-config PR 26`: https://github.com/python/ buildmaster-config/pull/26 .. _`Android version history`: https://en.wikipedia.org/wiki/ Android_version_history .. _`termux`: https://termux.com/ .. _`termux-packages`: https://gitlab.com/jbwhips883/termux-packages .. _`adb`: https://developer.android.com/studio/command-line/adb.html .. _`Unified headers`: https://android.googlesource. com/platform/ndk.git/+/ndk-r14-release/docs/UnifiedHeaders.md .. _`HP 41`: https://en.wikipedia.org/wiki/HP-41C .. vim:filetype=rst:tw=78:ts=8:sts=2:sw=2:et: _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ brett%40python.org
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ guido%40python.org
On Dec 10, 2017, at 16:26, Guido van Rossum <gvanrossum@gmail.com> wrote:
On Dec 10, 2017 12:29, "Brett Cannon" <brett@python.org> wrote:
On Sun, 10 Dec 2017 at 06:19 Xavier de Gaye <xdegaye@gmail.com> wrote:
The following note is a proposal to add the support of the Android platform. [...] While the note from a technical standpoint is interest, Xavier, I don't quite see what needs to be done to support Android at this point. Are you simply asking we add Android API 24 as an official platform? Or permission to add your note to the Misc/ directory? Basically what are you wanting to see happen? :) Maybe it should be a PEP?
Yes, I agree there needs to be a PEP for this. I have conflicting thoughts about formalizing Android support. On the one hand, it would be nice to have. But on the other, it does add a large non-zero burden to all core developers and to the release teams, to the minimum extent of trying to make sure that all ongoing changes don't break platform support. At a minimum a PEP needs to address the minimum platform support requirement outlined in PEP 11 (https://www.python.org/dev/peps/pep-0011/#supporting-platforms). As long as Xavier is willing to keep supporting the platform, the first requirement, having a core developer, should be met. But for a platform that, understandably, has as many special requirements as Android does, the second requirement, having a stable buildbot, seems to me to be an absolute necessity, and the PEP needs to address exactly what sort of buildbot requirements make sense here: emulators, SDKs, etc. Otherwise, we run the risk of ending up with an ongoing maintenance headache and unhappy users, as has been the case in the past with support for other platforms. -- Ned Deily nad@python.org -- []
I think someone may have to mentor Xavier on how to get this accepted. The note already looks a bit like a PEP, but I suspect that Xavier is not sufficiently familiar with our process to realize the difference. On Sun, Dec 10, 2017 at 2:07 PM, Ned Deily <nad@python.org> wrote:
On Dec 10, 2017 12:29, "Brett Cannon" <brett@python.org> wrote:
On Sun, 10 Dec 2017 at 06:19 Xavier de Gaye <xdegaye@gmail.com> wrote:
The following note is a proposal to add the support of the Android
On Dec 10, 2017, at 16:26, Guido van Rossum <gvanrossum@gmail.com> wrote: platform.
[...] While the note from a technical standpoint is interest, Xavier, I don't quite see what needs to be done to support Android at this point. Are you simply asking we add Android API 24 as an official platform? Or permission to add your note to the Misc/ directory? Basically what are you wanting to see happen? :) Maybe it should be a PEP?
Yes, I agree there needs to be a PEP for this. I have conflicting thoughts about formalizing Android support. On the one hand, it would be nice to have. But on the other, it does add a large non-zero burden to all core developers and to the release teams, to the minimum extent of trying to make sure that all ongoing changes don't break platform support. At a minimum a PEP needs to address the minimum platform support requirement outlined in PEP 11 (https://www.python.org/dev/peps/pep-0011/#supporting- platforms). As long as Xavier is willing to keep supporting the platform, the first requirement, having a core developer, should be met. But for a platform that, understandably, has as many special requirements as Android does, the second requirement, having a stable buildbot, seems to me to be an absolute necessity, and the PEP needs to address exactly what sort of buildbot requirements make sense here: emulators, SDKs, etc. Otherwise, we run the risk of ending up with an ongoing maintenance headache and unhappy users, as has been the case in the past with support for other platforms.
-- Ned Deily nad@python.org -- []
-- --Guido van Rossum (python.org/~guido)
Le 10 déc. 2017 23:10, "Ned Deily" <nad@python.org> a écrit :
But on the other, it does add a large non-zero burden to all core developers (...)
As long as Xavier is willing to keep supporting the platform, the first requirement, having a core developer, should be met. But for a platform
I reviewed some of the Android' pull requests. Most changes are small and self contained. that, understandably, has as many special requirements as Android does, the second requirement, having a stable buildbot, seems to me to be an absolute necessity, and the PEP needs to address exactly what sort of buildbot requirements make sense here: emulators, SDKs, etc. Xavier is a core dev and wants to add a buildbot to finish to support of Android. Victor
On 12/10/2017 11:07 PM, Ned Deily wrote:
On the one hand, it would be nice to have. But on the other, it does add a large non-zero burden to all core developers and to the release teams, to the minimum extent of trying to make sure that all ongoing changes don't break platform support.
Yes this platform has few quirks especially when SELinux is involved and some tests are harder to write because of that :-( Xavier
Xavier is working on fixing random issues specific to Android since 2 years. He is almost done, the last step is just to add a build infra to get a buildbot. https://github.com/python/cpython/pull/1629 https://bugs.python.org/issue30386 It's a set of scripts to cross compile Python from Linux to Android.
From the ones who missed it, Xavier is a core dev and will maintain this stuff ;-)
Since these changes add a new directory without touching the rest of the code, I don't see a reason to not add it. Victor Le 10 déc. 2017 21:29, "Brett Cannon" <brett@python.org> a écrit :
While the note from a technical standpoint is interest, Xavier, I don't quite see what needs to be done to support Android at this point. Are you simply asking we add Android API 24 as an official platform? Or permission to add your note to the Misc/ directory? Basically what are you wanting to see happen? :)
On Sun, 10 Dec 2017 at 06:19 Xavier de Gaye <xdegaye@gmail.com> wrote:
The following note is a proposal to add the support of the Android platform.
The note is easier to read with clickable links at https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst
Motivations ===========
* Android is ubiquitous. * This would be the first platform supported by Python that is cross-compiled, thanks to many contributors. * Although the Android operating system is linux, it is different from most linux platforms, for example it does not use GNU libc and runs SELinux in enforcing mode. Therefore supporting this platform would make Python more robust and also would allow testing it on arm 64-bit processors. * Python running on Android is also a handheld calculator, a successor of the slide rule and the `HP 41`_.
Current status ==============
* The Python test suite succeeds when run on Android emulators using buildbot strenuous settings with the following architectures on API 24: x86, x86_64, armv7 and arm64. * The `Android build system`_ is described in another section. * The `buildmaster-config PR 26`_ proposes to update ``master.cfg`` to enable buildbots to run a given Android API and architecture on the emulators. * The Android emulator is actually ``qemu``, so the test suites for x86 and x86_64 last about the same time as the test suite run natively when the processor of the build system is of the x86 family. The test suites for the arm architectures last much longer: about 8 hours for arm64 and 10 hours for armv7 on a four years old laptop. * The changes that have been made to achieve this status are listed in `bpo-26865`_, the Android meta-issue. * Given the cpu resources required to run the test suite on the arm emulators, it may be difficult to find a contributed buildbot worker. So it remains to find the hardware to run these buildbots.
Proposal ========
Support the Android platform on API 24 [1]_ for the x86_64, armv7 and arm64 architectures built with NDK 14b.
*API 24* * API 21 is the first version to provide usable support for wide characters and where SELinux is run in enforcing mode.
* API 22 introduces an annoying bug on the linker that prints something like this when python is started::
``WARNING: linker: libpython3.6m.so.1.0: unused DT entry: type 0x6ffffffe arg 0x14554``.
The `termux`_ Android terminal emulator describes this problem at the end of its `termux-packages`_ gitlab page and has implemented a ``termux-elf-cleaner`` tool to strip the useless entries from the ELF header of executables.
* API 24 is the first version where the `adb`_ shell is run on the emulator as a ``shell`` user instead of the ``root`` user previously, and the first version that supports arm64.
*x86_64* It seems that no handheld device exists using that architecture. It is supported because the x86_64 Android emulator runs fast and therefore is a good candidate as a buildbot worker.
*NDK 14b* This release of the NDK is the first one to use `Unified headers`_ fixing numerous problems that had been fixed by updating the Python configure script until now (those changes have been reverted by now).
Android idiosyncrasies ======================
* The default shell is ``/system/bin/sh``. * The file system layout is not a traditional unix layout, there is no ``/tmp`` for example. Most directories have user restricted access, ``/sdcard`` is mounted as ``noexec`` for example. * The (java) applications are allocated a unix user id and a subdirectory on ``/data/data``. * SELinux is run in enforcing mode. * Shared memory and semaphores are not supported. * The default encoding is UTF-8.
Android build system ====================
The Android build system is implemented at `bpo-30386`_ with `PR 1629`_ and is documented by its `README`_. It provides the following features:
* To build a distribution for a device or an emulator with a given API level and a given architecture. * To start the emulator and + install the distribution + start a remote interactive shell + or run remotely a python command + or run remotely the buildbottest * Run gdb on the python process that is running on the emulator with python pretty-printing.
The build system adds the ``Android/`` directory and the ``configure-android`` script to the root of the Python source directory on the master branch without modifying any other file. The build system can be installed, upgraded (i.e. the SDK and NDK) and run remotely, through ssh for example.
The following external libraries, when they are configured in the build system, are downloaded from the internet and cross-compiled (only once, on the first run of the build system) before the cross-compilation of the extension modules:
* ``ncurses`` * ``readline`` * ``sqlite`` * ``libffi`` * ``openssl``, the cross-compilation of openssl fails on x86_64 and arm64 and this step is skipped on those architectures.
The following extension modules are disabled by adding them to the ``*disabled*`` section of ``Modules/Setup``:
* ``_uuid``, Android has no uuid/uuid.h header. * ``grp`` some grp.h functions are not declared. * ``_crypt``, Android does not have crypt.h. * ``_ctypes`` on x86_64 where all long double tests fail (`bpo-32202`_) and on arm64 (see `bpo-32203`_).
.. [1] On Wikipedia `Android version history`_ lists the correspondence between API level, commercial name and version for each release. It also provides information on the global Android version distribution, see the two charts on top.
.. _`README`: https://github.com/xdegaye/cpython/blob/bpo-30386/ Android/README.rst .. _`bpo-26865`: https://bugs.python.org/issue26865 .. _`bpo-30386`: https://bugs.python.org/issue30386 .. _`bpo-32202`: https://bugs.python.org/issue32202 .. _`bpo-32203`: https://bugs.python.org/issue32203 .. _`PR 1629`: https://github.com/python/cpython/pull/1629 .. _`buildmaster-config PR 26`: https://github.com/python/ buildmaster-config/pull/26 .. _`Android version history`: https://en.wikipedia.org/wiki/ Android_version_history .. _`termux`: https://termux.com/ .. _`termux-packages`: https://gitlab.com/jbwhips883/termux-packages .. _`adb`: https://developer.android.com/studio/command-line/adb.html .. _`Unified headers`: https://android.googlesource. com/platform/ndk.git/+/ndk-r14-release/docs/UnifiedHeaders.md .. _`HP 41`: https://en.wikipedia.org/wiki/HP-41C .. vim:filetype=rst:tw=78:ts=8:sts=2:sw=2:et: _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ brett%40python.org
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ victor.stinner%40gmail.com
On 12/10/2017 09:27 PM, Brett Cannon wrote:
While the note from a technical standpoint is interest, Xavier, I don't quite see what needs to be done to support Android at this point. Are you simply asking we add Android API 24 as an official platform? Or permission to add your note to the Misc/ directory? Basically what are you wanting to see happen? :)
This is mainly a proposal written in the form of a PEP and open for discussion, not a PEP because it is not required by PEP 11 but it may become one. I guess the discussion may be about the choice of the API level, or whether the buildbot should run on emulators or on devices (not currently supported by the build system) or on any other subject. What is left to be done to support Android at this point is to merge the PR that implements the build system, make the change in the buildbot buildmaster-config repo and test it on a worker and then find the hardware to run all the workers for these architectures and set it up. Xavier
I'm not familiar with software development on/for Android, but wouldn't official support also involve suitable package creation or does that just fall out for free from the build-for-emulator PR? Skip
On 12/10/2017 10:27 PM, Skip Montanaro wrote:
I'm not familiar with software development on/for Android, but wouldn't official support also involve suitable package creation or does that just fall out for free from the build-for-emulator PR?
Skip
The build-for-emulator PR allows building for a device but the packaging of an Android application into an installable APK (a zip file actually) is the responsability of the java application developer. Xavier
2017-12-10 15:19 GMT+01:00 Xavier de Gaye <xdegaye@gmail.com>:
Motivations ===========
* Android is ubiquitous. * This would be the first platform supported by Python that is cross-compiled, thanks to many contributors. * Although the Android operating system is linux, it is different from most linux platforms, for example it does not use GNU libc and runs SELinux in enforcing mode. Therefore supporting this platform would make Python more robust and also would allow testing it on arm 64-bit processors. * Python running on Android is also a handheld calculator, a successor of the slide rule and the `HP 41`_.
I still don't understand what is "Android". What is the license of Android?
* The Python test suite succeeds when run on Android emulators using buildbot
Great achievement! Congrats! I know that it has been a long travel to reach this point! (Fix each invidivual test failure, fix many tiny things.)
* Given the cpu resources required to run the test suite on the arm emulators, it may be difficult to find a contributed buildbot worker. So it remains to find the hardware to run these buildbots.
Do you have the hardware to host such worker? Or are you looking for a host somewhere? Which kind of hardware are you looking for? CPU, memory, network bandwidth, etc.
*API 24* * API 21 is the first version to provide usable support for wide characters and where SELinux is run in enforcing mode.
Some people are looking for API 19 support. Would it be doable, or would it require too many changes? I know that people are running heavily patched Python 2.7 and 3.5 on Android with API 19. I'm not asking for a "full support" for API 19, but more if it would be possible to get a "best effort" level of support, like accept patches if someone writes them.
The following extension modules are disabled by adding them to the ``*disabled*`` section of ``Modules/Setup``:
* ``_uuid``, Android has no uuid/uuid.h header. * ``grp`` some grp.h functions are not declared. * ``_crypt``, Android does not have crypt.h. * ``_ctypes`` on x86_64 where all long double tests fail (`bpo-32202`_) and on arm64 (see `bpo-32203`_).
That's a very short list, it's ok. It's not the most popular modules of the stdlib :-) ctypes would be nice to have, but it can be done later. Victor
On 12/11/2017 12:56 PM, Victor Stinner wrote:
2017-12-10 15:19 GMT+01:00 Xavier de Gaye <xdegaye@gmail.com>:
* Given the cpu resources required to run the test suite on the arm emulators, it may be difficult to find a contributed buildbot worker. So it remains to find the hardware to run these buildbots.
Do you have the hardware to host such worker? Or are you looking for a host somewhere? Which kind of hardware are you looking for? CPU, memory, network bandwidth, etc.
I cannot host the buildbots or any buildbot for that matter. I can maintain them. The host running the buildbots must be able to run 6 (i.e. 3 x (version 3.x + maintenance version)) emulators simultaneously, so with an eight core cpu, that will be 6 cores running at 100%. The armv7 and arm64 buildbot may be set to run only daily but the tests last a long time on these architectures anyway.
*API 24* * API 21 is the first version to provide usable support for wide characters and where SELinux is run in enforcing mode.
Some people are looking for API 19 support. Would it be doable, or would it require too many changes? I know that people are running heavily patched Python 2.7 and 3.5 on Android with API 19.
I'm not asking for a "full support" for API 19, but more if it would be possible to get a "best effort" level of support, like accept patches if someone writes them.
Not sure that python can be built on API 19. What I remember about API 19 at the time I started this project, is that wide characters support is not usable. If you look at the Android version history [1] on Wikipedia referred to in my initial post, the Kit Kat (API 19) share is 16 % now and will probably be 8 % next year. Another point to consider is that working on a change specific to Android is tedious: the test case must be ok on the build platform and on the emulator. The emulator must be started and an installation made from scratch, and after few file modifications on the emulator there is no 'git status' command to tell exactly what change you are running and you must re-install from scratch. Is there a way to browse these patches to get a better idea of the changes involved ? Xavier [1] https://en.wikipedia.org/wiki/Android_version_history
2017-12-11 14:58 GMT+01:00 Xavier de Gaye <xdegaye@gmail.com>:
The host running the buildbots must be able to run 6 (i.e. 3 x (version 3.x + maintenance version)) emulators simultaneously, so with an eight core cpu, that will be 6 cores running at 100%. The armv7 and arm64 buildbot may be set to run only daily but the tests last a long time on these architectures anyway.
What do you mean by "maintenance version"? Do you want to add Android support to Python 2.7 and 3.6 as well? I would prefer to only support Android since the master branch. Would it be possible to only run an emulator to run a test, and then stop it? So we could test many combo in the same host? Victor
I'm asking for precise hardware specifications since Red Hat may be able to provide one through the https://osci.io/ program. Victor 2017-12-11 15:40 GMT+01:00 Victor Stinner <victor.stinner@gmail.com>:
2017-12-11 14:58 GMT+01:00 Xavier de Gaye <xdegaye@gmail.com>:
The host running the buildbots must be able to run 6 (i.e. 3 x (version 3.x + maintenance version)) emulators simultaneously, so with an eight core cpu, that will be 6 cores running at 100%. The armv7 and arm64 buildbot may be set to run only daily but the tests last a long time on these architectures anyway.
What do you mean by "maintenance version"? Do you want to add Android support to Python 2.7 and 3.6 as well? I would prefer to only support Android since the master branch.
Would it be possible to only run an emulator to run a test, and then stop it? So we could test many combo in the same host?
Victor
On 12/11/2017 04:14 PM, Victor Stinner wrote:
I'm asking for precise hardware specifications since Red Hat may be able to provide one through the https://osci.io/ program.
Is it acceptable to run the arm buildbots only on a weekly basis ? For all the architectures (x86_64, armv7 and arm64), the tests are run with the same Python code, built with the same tools and running on the same operating system that runs on emulators using the same configuration. If we do not take into account the ctypes issue, there is only one issue (issue 26939 [1]) among all the Android issues that is specific to arm, and it is not even really an arm issue but an issue related to the slowness of the emulator that has been fixed by increasing in a test the switch interval with sys.setswitchinterval(). Xavier [1] https://bugs.python.org/issue26939
On Tue, 12 Dec 2017 10:50:53 +0100 Xavier de Gaye <xdegaye@gmail.com> wrote:
On 12/11/2017 04:14 PM, Victor Stinner wrote:
I'm asking for precise hardware specifications since Red Hat may be able to provide one through the https://osci.io/ program.
Is it acceptable to run the arm buildbots only on a weekly basis ?
It sounds reasonable to me, as long as someone is monitoring their results (that would probably be you). Regards Antoine.
2017-12-12 15:02 GMT+01:00 Antoine Pitrou <solipsis@pitrou.net>:
It sounds reasonable to me, as long as someone is monitoring their results (that would probably be you).
There is now the buildbot-status where failures are reported. Trust me, I like making a lot of noise when something is broken :-) FYI I'm trying https://github.com/python/cpython/pull/1629 : I succeeded to "cross-compile" a binary on Linux for Android. I'm now downloading an ISO to boot an x86-64 Android VM to test this binary ;-) Victor
On 12/11/2017 03:40 PM, Victor Stinner wrote:
2017-12-11 14:58 GMT+01:00 Xavier de Gaye <xdegaye@gmail.com>:
The host running the buildbots must be able to run 6 (i.e. 3 x (version 3.x + maintenance version)) emulators simultaneously, so with an eight core cpu, that will be 6 cores running at 100%. The armv7 and arm64 buildbot may be set to run only daily but the tests last a long time on these architectures anyway.
What do you mean by "maintenance version"? Do you want to add Android support to Python 2.7 and 3.6 as well? I would prefer to only support Android since the master branch.
Yes, today only the master branch, and when 3.8 is released then both 3.7 and 3.8 would be supported. That is what I (implicitly) meant.
Would it be possible to only run an emulator to run a test, and then stop it? So we could test many combo in the same host?
I may not understand your question. An emulator runs with an AVD image (Android Virtual Device) that is specific to the API and architecture (armv7, x86_64, ...), and that contains the data (file systems) and the configuration. The AVD image is created once and for all for each of the combinations (API, architecture). You start the emulator with the AVD that matches the type of the built python (API, architecture) and install that build. Multiple emulators can run simultaneously except that the build system imposes the restriction that no two identical (API, architecture) emulators can run concurrently except if they are not used for the same Python version (emulators are identified by their port numbers by the Android tools and the build system allocates those ports statically). Xavier
Hi Xavier, I looked at your scripts to build Android but I failed to use them. Anyway, I'm not sure why these scripts have to be part of the CPython git repository. Technically, is there a reason to put it aside the source code and Unix build scripts (configure/Makefile/setup)? Your https://github.com/python/cpython/pull/1629 only adds new files without touching existing files. I suggest to create new Git project. It may be in the python organization, or you may start with your GitHub account. Cross-compilation is hard, and I'm not sure that it's possible to build a single recipe for all Android API versions, all configuration, any set of libraries, etc. For Android, it seems like each developer might want a subtle different configuration which might not be easy to support. Having a separated Git project would allow people to contribute more easily, experiment their fork, etc. What do you think? I'm only talking about the proposed Android/ directory and https://github.com/python/cpython/pull/1629. Everything else is fine :-) Victor 2017-12-10 15:19 GMT+01:00 Xavier de Gaye <xdegaye@gmail.com>:
The following note is a proposal to add the support of the Android platform.
The note is easier to read with clickable links at https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst
Motivations ===========
* Android is ubiquitous. * This would be the first platform supported by Python that is cross-compiled, thanks to many contributors. * Although the Android operating system is linux, it is different from most linux platforms, for example it does not use GNU libc and runs SELinux in enforcing mode. Therefore supporting this platform would make Python more robust and also would allow testing it on arm 64-bit processors. * Python running on Android is also a handheld calculator, a successor of the slide rule and the `HP 41`_.
Current status ==============
* The Python test suite succeeds when run on Android emulators using buildbot strenuous settings with the following architectures on API 24: x86, x86_64, armv7 and arm64. * The `Android build system`_ is described in another section. * The `buildmaster-config PR 26`_ proposes to update ``master.cfg`` to enable buildbots to run a given Android API and architecture on the emulators. * The Android emulator is actually ``qemu``, so the test suites for x86 and x86_64 last about the same time as the test suite run natively when the processor of the build system is of the x86 family. The test suites for the arm architectures last much longer: about 8 hours for arm64 and 10 hours for armv7 on a four years old laptop. * The changes that have been made to achieve this status are listed in `bpo-26865`_, the Android meta-issue. * Given the cpu resources required to run the test suite on the arm emulators, it may be difficult to find a contributed buildbot worker. So it remains to find the hardware to run these buildbots.
Proposal ========
Support the Android platform on API 24 [1]_ for the x86_64, armv7 and arm64 architectures built with NDK 14b.
*API 24* * API 21 is the first version to provide usable support for wide characters and where SELinux is run in enforcing mode.
* API 22 introduces an annoying bug on the linker that prints something like this when python is started::
``WARNING: linker: libpython3.6m.so.1.0: unused DT entry: type 0x6ffffffe arg 0x14554``.
The `termux`_ Android terminal emulator describes this problem at the end of its `termux-packages`_ gitlab page and has implemented a ``termux-elf-cleaner`` tool to strip the useless entries from the ELF header of executables.
* API 24 is the first version where the `adb`_ shell is run on the emulator as a ``shell`` user instead of the ``root`` user previously, and the first version that supports arm64.
*x86_64* It seems that no handheld device exists using that architecture. It is supported because the x86_64 Android emulator runs fast and therefore is a good candidate as a buildbot worker.
*NDK 14b* This release of the NDK is the first one to use `Unified headers`_ fixing numerous problems that had been fixed by updating the Python configure script until now (those changes have been reverted by now).
Android idiosyncrasies ======================
* The default shell is ``/system/bin/sh``. * The file system layout is not a traditional unix layout, there is no ``/tmp`` for example. Most directories have user restricted access, ``/sdcard`` is mounted as ``noexec`` for example. * The (java) applications are allocated a unix user id and a subdirectory on ``/data/data``. * SELinux is run in enforcing mode. * Shared memory and semaphores are not supported. * The default encoding is UTF-8.
Android build system ====================
The Android build system is implemented at `bpo-30386`_ with `PR 1629`_ and is documented by its `README`_. It provides the following features:
* To build a distribution for a device or an emulator with a given API level and a given architecture. * To start the emulator and + install the distribution + start a remote interactive shell + or run remotely a python command + or run remotely the buildbottest * Run gdb on the python process that is running on the emulator with python pretty-printing.
The build system adds the ``Android/`` directory and the ``configure-android`` script to the root of the Python source directory on the master branch without modifying any other file. The build system can be installed, upgraded (i.e. the SDK and NDK) and run remotely, through ssh for example.
The following external libraries, when they are configured in the build system, are downloaded from the internet and cross-compiled (only once, on the first run of the build system) before the cross-compilation of the extension modules:
* ``ncurses`` * ``readline`` * ``sqlite`` * ``libffi`` * ``openssl``, the cross-compilation of openssl fails on x86_64 and arm64 and this step is skipped on those architectures.
The following extension modules are disabled by adding them to the ``*disabled*`` section of ``Modules/Setup``:
* ``_uuid``, Android has no uuid/uuid.h header. * ``grp`` some grp.h functions are not declared. * ``_crypt``, Android does not have crypt.h. * ``_ctypes`` on x86_64 where all long double tests fail (`bpo-32202`_) and on arm64 (see `bpo-32203`_).
.. [1] On Wikipedia `Android version history`_ lists the correspondence between API level, commercial name and version for each release. It also provides information on the global Android version distribution, see the two charts on top.
.. _`README`: https://github.com/xdegaye/cpython/blob/bpo-30386/Android/README.rst .. _`bpo-26865`: https://bugs.python.org/issue26865 .. _`bpo-30386`: https://bugs.python.org/issue30386 .. _`bpo-32202`: https://bugs.python.org/issue32202 .. _`bpo-32203`: https://bugs.python.org/issue32203 .. _`PR 1629`: https://github.com/python/cpython/pull/1629 .. _`buildmaster-config PR 26`: https://github.com/python/buildmaster-config/pull/26 .. _`Android version history`: https://en.wikipedia.org/wiki/Android_version_history .. _`termux`: https://termux.com/ .. _`termux-packages`: https://gitlab.com/jbwhips883/termux-packages .. _`adb`: https://developer.android.com/studio/command-line/adb.html .. _`Unified headers`: https://android.googlesource.com/platform/ndk.git/+/ndk-r14-release/docs/Uni... .. _`HP 41`: https://en.wikipedia.org/wiki/HP-41C .. vim:filetype=rst:tw=78:ts=8:sts=2:sw=2:et: _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.co...
On 12/13/2017 10:56 PM, Victor Stinner wrote:
I looked at your scripts to build Android but I failed to use them.
You failed because you did not read the README and tried to improvise. I will change the documentation and the build process to make it simpler for those that do not have the time to RTFM :-) I have documented the correct steps in the PR especially for you (they are just a summary of what is in the README) after you reported that failure. So you can follow these steps any time now and use correctly the build system and report the results here. This would be only fair, I think. BTW You keep saying they are scripts when the build is actually driven by Makefiles (with the '.mk' extension).
Anyway, I'm not sure why these scripts have to be part of the CPython git repository.
Technically, is there a reason to put it aside the source code and Unix build scripts (configure/Makefile/setup)? Your https://github.com/python/cpython/pull/1629 only adds new files without touching existing files.
I suggest to create new Git project. It may be in the python organization, or you may start with your GitHub account.
The 'Mac' build system has its own subdirectory in the source tree and it makes sense as it is the reference build system for this platform. I do not see why this should be different for Android.
Cross-compilation is hard, and I'm not sure that it's possible to build a single recipe for all Android API versions, all configuration, any set of libraries, etc. For Android, it seems like each developer might want a subtle different configuration which might not be easy to support.
You are mistaken, this proposal does not suggest that we are going to support "all Android API versions, all configuration, any set of libraries, etc.", quite the opposite actually. The proposal is an Android build system for a specific API, a set of architectures using the NDK r14 toolchain and a set of optional external libraries. The build system enforces the use of NDK r14 for example (as you have painfuly experienced). Python is tested on emulators so that there is no interference with vendor specific additions found on the Android devices or with installed PlayStore applications.
Having a separated Git project would allow people to contribute more easily, experiment their fork, etc.
What do you think?
Certainly not. We, core-devs, are very happy that no one is experimenting with our build system, it is complex enough as it is. The same goes for this Android build system. Your suggestion seems to be driven by the failure you have experienced with this new build system and the fact that a user is also reporting a failure. The origin of this other failure is unclear because I cannot reproduce it even though all the components used for the build are well defined and identical for everyone: the NDK includes the clang compiler, the libraries and the headers, the external libraries are downloaded by the build system, all the users use identical tools (same versions) and the same source code and the only difference may be with some utility tools such as sed, awk, etc... This is a bad start for this proposal and it would have been fair to inform me that you were working in irc collaboration with this other user in testing the build system. On the other hand, these problems may have some positive consequences since it allows us to be aware of the fact that the bpo audience may change if we support Android and that this may be a problem. Android attracts all kind of developers that do not have the average expertise of unix developers and more importantly that do not have the same motivations and the same etiquette. I am now concerned by the fact that the quality of the bug reports on bpo may dramatically decrease if we adopt this proposal. Xavier
2017-12-14 10:26 GMT+01:00 Xavier de Gaye <xdegaye@gmail.com>:
The 'Mac' build system has its own subdirectory in the source tree and it makes sense as it is the reference build system for this platform. I do not see why this should be different for Android.
Hum, Mac/ is mostly the recipe to build the installer and a .dmg image, no? macOS uses the same configure/Makefile than Linux, no? Building Python on macOS is quite simple, there is nothing special about macOS. And macOS is used on a *very* limited set of hardware, only sold by a single vendor, Apple. Android is quite the opposite. The propose Android directory is complex recipe for cross-compiling for a wide range of hardware, and likely different usages of Python. Some people may want a REPL, some people may only be interested by a GUI (like Kivy or Panda3D?), some people probably want to do something else. All Python modules work on macOS, since macOS is just one flavor of POSIX. While Android is compatible with POSIX, from the bug reports that I saw, many modules don't work and will not work on Android. So supporting Android is much more complex than supporting macOS.
You are mistaken, this proposal does not suggest that we are going to support "all Android API versions, all configuration, any set of libraries, etc.", quite the opposite actually.
Since we are talking about the future, I would like to remain open to support wider Android API versions and any configuration. As I wrote, I don't require to support all configurations and all API versions, but just provide best effort support, and only fully support one specific API version and one specific config.
The build system enforces the use of NDK r14 for example (as you have painfuly experienced).
It seems like Android is evolving quickly, I would say quicker than Python releases. I'm asking if it's a good idea to put a recipe aside the Python source code for one specific Android API version? Would it still make sense to build for NDK v14 in 2 or 5 years?
This is a bad start for this proposal and it would have been fair to inform me that you were working in irc collaboration with this other user in testing the build system.
Right, I'm working with Paul Peny (pmpp on IRC). He is helping me to test your PR. Paul understand Android much better than me. For me, it's still a huge black box. Basically, I don't understand what I'm doing :-) Victor
On 12/14/2017 02:59 PM, Victor Stinner wrote:
It seems like Android is evolving quickly, would say quicker than Python releases. I'm asking if it's a good idea to put a recipe aside the Python source code for one specific Android API version? Would it still make sense to build for NDK v14 in 2 or 5 years?
NDK 14 has been released in march 2017 and the latest release is NDK 16. There are sometimes major changes between releases and I think it is critical to ensure that the builds all use the same NDK release for that reason. Supporting another NDK release is just a substitution in one of the files of the build system and I am sure that in 2 or 5 years there would have been a core developer smart enough to make that substitution (this could even have been me, I will only be 71 years old in 5 years :-)). Anyway if this is a problem, this should have been discussed in a review of the PR. There are concerns, including a concern raised by me, about supporting Android with that build system or to supporting Android at all. It has been interesting and gratifying to work on this build system and to get the Python test suite running on Android without failures. Given these concerns and the lack of interest in the support of Android it is time for me to switch to something else, maybe improve the bdb module, why not ? Xavier
On 12/15/2017 8:29 AM, Xavier de Gaye wrote:
On 12/14/2017 02:59 PM, Victor Stinner wrote:
It seems like Android is evolving quickly, would say quicker than Python releases. I'm asking if it's a good idea to put a recipe aside the Python source code for one specific Android API version? Would it still make sense to build for NDK v14 in 2 or 5 years?
NDK 14 has been released in march 2017 and the latest release is NDK 16. There are sometimes major changes between releases and I think it is critical to ensure that the builds all use the same NDK release for that reason. Supporting another NDK release is just a substitution in one of the files of the build system and I am sure that in 2 or 5 years there would have been a core developer smart enough to make that substitution (this could even have been me, I will only be 71 years old in 5 years :-)). Anyway if this is a problem, this should have been discussed in a review of the PR.
There are concerns, including a concern raised by me, about supporting Android with that build system or to supporting Android at all. It has been interesting and gratifying to work on this build system and to get the Python test suite running on Android without failures. Given these concerns and the lack of interest in the support of Android it is time for me to switch to something else, maybe improve the bdb module, why not ?
Xavier
I, for one, would love to see Android become a supported platform. My understanding is that APIs are generally backward compatible, so that programs created with one API continue to work on future APIs... there may be new features they don't use and maybe can't support, because they don't know about newer APIs, but that is true of Windows and Linux and Mac also. It doesn't seem like it would be necessary to "support" or "release" an official Python for every new Android version, but it would be nice to have ongoing support for a recent version each time Python is released, if Xavier or someone can do it. It is certainly a mainstream platform for development these days. I see a variety of Python apps on Android, but I have no idea how well they work, or what their source is, or if they are supported, or what features of Python or its standard library might not be available, etc. I've been too busy on other projects to take time to investigate them. Glenn
I've used Kivy with buildozer on Android and it generally works well, with a few issues. Currently it uses the Crystax NDK for Python 3 support. Does anyone know how this development will affect it? -Mike On 2017-12-15 10:06, Glenn Linderman wrote:
I see a variety of Python apps on Android, but I have no idea how well they work, or what their source is, or if they are supported, or what features of Python or its standard library might not be available, etc. I've been too busy on other projects to take time to investigate them.
Hi, I'm still talking with Paul Peny (pmpp on IRC) who is trying to build the master branch of Python on Android, using cross-compilation or directly on an Android device. I started to took notes since Android is a complex platforms and it's not easy for me to remember everything. http://vstinner.readthedocs.io/python_android.html Paul would like to support Android 4.4 Kitkat (API 19) just because it's possible to find cheap devices running Android, but usually only with old Android versions. Technically, it doesn't see difficult to support API 19+ (instead of 21+), a few tiny patches are needed. But I don't want to have a "full support" of API 19+, only basic support like "make sure that the compilation doesn't fail", not "all tests must pass". It seems like sys.platform == 'android' would be more appropriate since Android is not Linux: different libc, different filesystems, etc. While Xavier promotes cross-compilation, Paul would like to build Python directly on Android to get pip and wheels. Honestly, I have no strong opinion, since I don't know well Android. I'm trying to help everybody working on the Android support. IMHO it's fine to support multiple ways to build Python for Android. It's not like there is very obvious option which has no drawback... Cross compilation is complex, getting a C compiler on Android also seems to be complex. From my point of view, compared to a common Fedora Linux, Android doesn't seem easy to use to develop on CPython... Victor 2017-12-10 15:19 GMT+01:00 Xavier de Gaye <xdegaye@gmail.com>:
The following note is a proposal to add the support of the Android platform.
The note is easier to read with clickable links at https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst
Motivations ===========
* Android is ubiquitous. * This would be the first platform supported by Python that is cross-compiled, thanks to many contributors. * Although the Android operating system is linux, it is different from most linux platforms, for example it does not use GNU libc and runs SELinux in enforcing mode. Therefore supporting this platform would make Python more robust and also would allow testing it on arm 64-bit processors. * Python running on Android is also a handheld calculator, a successor of the slide rule and the `HP 41`_.
Current status ==============
* The Python test suite succeeds when run on Android emulators using buildbot strenuous settings with the following architectures on API 24: x86, x86_64, armv7 and arm64. * The `Android build system`_ is described in another section. * The `buildmaster-config PR 26`_ proposes to update ``master.cfg`` to enable buildbots to run a given Android API and architecture on the emulators. * The Android emulator is actually ``qemu``, so the test suites for x86 and x86_64 last about the same time as the test suite run natively when the processor of the build system is of the x86 family. The test suites for the arm architectures last much longer: about 8 hours for arm64 and 10 hours for armv7 on a four years old laptop. * The changes that have been made to achieve this status are listed in `bpo-26865`_, the Android meta-issue. * Given the cpu resources required to run the test suite on the arm emulators, it may be difficult to find a contributed buildbot worker. So it remains to find the hardware to run these buildbots.
Proposal ========
Support the Android platform on API 24 [1]_ for the x86_64, armv7 and arm64 architectures built with NDK 14b.
*API 24* * API 21 is the first version to provide usable support for wide characters and where SELinux is run in enforcing mode.
* API 22 introduces an annoying bug on the linker that prints something like this when python is started::
``WARNING: linker: libpython3.6m.so.1.0: unused DT entry: type 0x6ffffffe arg 0x14554``.
The `termux`_ Android terminal emulator describes this problem at the end of its `termux-packages`_ gitlab page and has implemented a ``termux-elf-cleaner`` tool to strip the useless entries from the ELF header of executables.
* API 24 is the first version where the `adb`_ shell is run on the emulator as a ``shell`` user instead of the ``root`` user previously, and the first version that supports arm64.
*x86_64* It seems that no handheld device exists using that architecture. It is supported because the x86_64 Android emulator runs fast and therefore is a good candidate as a buildbot worker.
*NDK 14b* This release of the NDK is the first one to use `Unified headers`_ fixing numerous problems that had been fixed by updating the Python configure script until now (those changes have been reverted by now).
Android idiosyncrasies ======================
* The default shell is ``/system/bin/sh``. * The file system layout is not a traditional unix layout, there is no ``/tmp`` for example. Most directories have user restricted access, ``/sdcard`` is mounted as ``noexec`` for example. * The (java) applications are allocated a unix user id and a subdirectory on ``/data/data``. * SELinux is run in enforcing mode. * Shared memory and semaphores are not supported. * The default encoding is UTF-8.
Android build system ====================
The Android build system is implemented at `bpo-30386`_ with `PR 1629`_ and is documented by its `README`_. It provides the following features:
* To build a distribution for a device or an emulator with a given API level and a given architecture. * To start the emulator and + install the distribution + start a remote interactive shell + or run remotely a python command + or run remotely the buildbottest * Run gdb on the python process that is running on the emulator with python pretty-printing.
The build system adds the ``Android/`` directory and the ``configure-android`` script to the root of the Python source directory on the master branch without modifying any other file. The build system can be installed, upgraded (i.e. the SDK and NDK) and run remotely, through ssh for example.
The following external libraries, when they are configured in the build system, are downloaded from the internet and cross-compiled (only once, on the first run of the build system) before the cross-compilation of the extension modules:
* ``ncurses`` * ``readline`` * ``sqlite`` * ``libffi`` * ``openssl``, the cross-compilation of openssl fails on x86_64 and arm64 and this step is skipped on those architectures.
The following extension modules are disabled by adding them to the ``*disabled*`` section of ``Modules/Setup``:
* ``_uuid``, Android has no uuid/uuid.h header. * ``grp`` some grp.h functions are not declared. * ``_crypt``, Android does not have crypt.h. * ``_ctypes`` on x86_64 where all long double tests fail (`bpo-32202`_) and on arm64 (see `bpo-32203`_).
.. [1] On Wikipedia `Android version history`_ lists the correspondence between API level, commercial name and version for each release. It also provides information on the global Android version distribution, see the two charts on top.
.. _`README`: https://github.com/xdegaye/cpython/blob/bpo-30386/Android/README.rst .. _`bpo-26865`: https://bugs.python.org/issue26865 .. _`bpo-30386`: https://bugs.python.org/issue30386 .. _`bpo-32202`: https://bugs.python.org/issue32202 .. _`bpo-32203`: https://bugs.python.org/issue32203 .. _`PR 1629`: https://github.com/python/cpython/pull/1629 .. _`buildmaster-config PR 26`: https://github.com/python/buildmaster-config/pull/26 .. _`Android version history`: https://en.wikipedia.org/wiki/Android_version_history .. _`termux`: https://termux.com/ .. _`termux-packages`: https://gitlab.com/jbwhips883/termux-packages .. _`adb`: https://developer.android.com/studio/command-line/adb.html .. _`Unified headers`: https://android.googlesource.com/platform/ndk.git/+/ndk-r14-release/docs/Uni... .. _`HP 41`: https://en.wikipedia.org/wiki/HP-41C .. vim:filetype=rst:tw=78:ts=8:sts=2:sw=2:et: _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.co...
On Mon, 22 Jan 2018 at 09:29 Victor Stinner <victor.stinner@gmail.com> wrote:
Hi,
I'm still talking with Paul Peny (pmpp on IRC) who is trying to build the master branch of Python on Android, using cross-compilation or directly on an Android device. I started to took notes since Android is a complex platforms and it's not easy for me to remember everything.
http://vstinner.readthedocs.io/python_android.html
Paul would like to support Android 4.4 Kitkat (API 19) just because it's possible to find cheap devices running Android, but usually only with old Android versions. Technically, it doesn't see difficult to support API 19+ (instead of 21+), a few tiny patches are needed. But I don't want to have a "full support" of API 19+, only basic support like "make sure that the compilation doesn't fail", not "all tests must pass".
It seems like sys.platform == 'android' would be more appropriate since Android is not Linux: different libc, different filesystems, etc.
I've had a similar thought myself. -Brett
While Xavier promotes cross-compilation, Paul would like to build Python directly on Android to get pip and wheels.
Honestly, I have no strong opinion, since I don't know well Android. I'm trying to help everybody working on the Android support. IMHO it's fine to support multiple ways to build Python for Android. It's not like there is very obvious option which has no drawback... Cross compilation is complex, getting a C compiler on Android also seems to be complex. From my point of view, compared to a common Fedora Linux, Android doesn't seem easy to use to develop on CPython...
Victor
The following note is a proposal to add the support of the Android
2017-12-10 15:19 GMT+01:00 Xavier de Gaye <xdegaye@gmail.com>: platform.
The note is easier to read with clickable links at https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst
Motivations ===========
* Android is ubiquitous. * This would be the first platform supported by Python that is cross-compiled, thanks to many contributors. * Although the Android operating system is linux, it is different from
linux platforms, for example it does not use GNU libc and runs SELinux in enforcing mode. Therefore supporting this platform would make Python more robust and also would allow testing it on arm 64-bit processors. * Python running on Android is also a handheld calculator, a successor of the slide rule and the `HP 41`_.
Current status ==============
* The Python test suite succeeds when run on Android emulators using buildbot strenuous settings with the following architectures on API 24: x86, x86_64, armv7 and arm64. * The `Android build system`_ is described in another section. * The `buildmaster-config PR 26`_ proposes to update ``master.cfg`` to enable buildbots to run a given Android API and architecture on the emulators. * The Android emulator is actually ``qemu``, so the test suites for x86 and x86_64 last about the same time as the test suite run natively when the processor of the build system is of the x86 family. The test suites for the arm architectures last much longer: about 8 hours for arm64 and 10 hours for armv7 on a four years old laptop. * The changes that have been made to achieve this status are listed in `bpo-26865`_, the Android meta-issue. * Given the cpu resources required to run the test suite on the arm emulators, it may be difficult to find a contributed buildbot worker. So it remains to find the hardware to run these buildbots.
Proposal ========
Support the Android platform on API 24 [1]_ for the x86_64, armv7 and arm64 architectures built with NDK 14b.
*API 24* * API 21 is the first version to provide usable support for wide characters and where SELinux is run in enforcing mode.
* API 22 introduces an annoying bug on the linker that prints something like this when python is started::
``WARNING: linker: libpython3.6m.so.1.0: unused DT entry: type 0x6ffffffe arg 0x14554``.
The `termux`_ Android terminal emulator describes this problem at the end of its `termux-packages`_ gitlab page and has implemented a ``termux-elf-cleaner`` tool to strip the useless entries from the ELF header of executables.
* API 24 is the first version where the `adb`_ shell is run on the emulator as a ``shell`` user instead of the ``root`` user previously, and the first version that supports arm64.
*x86_64* It seems that no handheld device exists using that architecture. It is supported because the x86_64 Android emulator runs fast and therefore is a good candidate as a buildbot worker.
*NDK 14b* This release of the NDK is the first one to use `Unified headers`_ fixing numerous problems that had been fixed by updating the Python configure script until now (those changes have been reverted by now).
Android idiosyncrasies ======================
* The default shell is ``/system/bin/sh``. * The file system layout is not a traditional unix layout, there is no ``/tmp`` for example. Most directories have user restricted access, ``/sdcard`` is mounted as ``noexec`` for example. * The (java) applications are allocated a unix user id and a subdirectory on ``/data/data``. * SELinux is run in enforcing mode. * Shared memory and semaphores are not supported. * The default encoding is UTF-8.
Android build system ====================
The Android build system is implemented at `bpo-30386`_ with `PR 1629`_ and is documented by its `README`_. It provides the following features:
* To build a distribution for a device or an emulator with a given API level and a given architecture. * To start the emulator and + install the distribution + start a remote interactive shell + or run remotely a python command + or run remotely the buildbottest * Run gdb on the python process that is running on the emulator with
pretty-printing.
The build system adds the ``Android/`` directory and the ``configure-android`` script to the root of the Python source directory on the master branch without modifying any other file. The build system can be installed, upgraded (i.e. the SDK and NDK) and run remotely, through ssh for example.
The following external libraries, when they are configured in the build system, are downloaded from the internet and cross-compiled (only once, on the first run of the build system) before the cross-compilation of the extension modules:
* ``ncurses`` * ``readline`` * ``sqlite`` * ``libffi`` * ``openssl``, the cross-compilation of openssl fails on x86_64 and arm64 and this step is skipped on those architectures.
The following extension modules are disabled by adding them to the ``*disabled*`` section of ``Modules/Setup``:
* ``_uuid``, Android has no uuid/uuid.h header. * ``grp`` some grp.h functions are not declared. * ``_crypt``, Android does not have crypt.h. * ``_ctypes`` on x86_64 where all long double tests fail (`bpo-32202`_) and on arm64 (see `bpo-32203`_).
.. [1] On Wikipedia `Android version history`_ lists the correspondence between API level, commercial name and version for each release. It also
most python provides
information on the global Android version distribution, see the two charts on top.
.. _`README`: https://github.com/xdegaye/cpython/blob/bpo-30386/Android/README.rst .. _`bpo-26865`: https://bugs.python.org/issue26865 .. _`bpo-30386`: https://bugs.python.org/issue30386 .. _`bpo-32202`: https://bugs.python.org/issue32202 .. _`bpo-32203`: https://bugs.python.org/issue32203 .. _`PR 1629`: https://github.com/python/cpython/pull/1629 .. _`buildmaster-config PR 26`: https://github.com/python/buildmaster-config/pull/26 .. _`Android version history`: https://en.wikipedia.org/wiki/Android_version_history .. _`termux`: https://termux.com/ .. _`termux-packages`: https://gitlab.com/jbwhips883/termux-packages .. _`adb`: https://developer.android.com/studio/command-line/adb.html .. _`Unified headers`:
https://android.googlesource.com/platform/ndk.git/+/ndk-r14-release/docs/Uni...
.. _`HP 41`: https://en.wikipedia.org/wiki/HP-41C .. vim:filetype=rst:tw=78:ts=8:sts=2:sw=2:et: _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.co... _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
Ok, I created https://bugs.python.org/issue32637 "Android: set sys.platform and os.name to android" Victor 2018-01-23 18:01 GMT+01:00 Brett Cannon <brett@python.org>:
On Mon, 22 Jan 2018 at 09:29 Victor Stinner <victor.stinner@gmail.com> wrote:
Hi,
I'm still talking with Paul Peny (pmpp on IRC) who is trying to build the master branch of Python on Android, using cross-compilation or directly on an Android device. I started to took notes since Android is a complex platforms and it's not easy for me to remember everything.
http://vstinner.readthedocs.io/python_android.html
Paul would like to support Android 4.4 Kitkat (API 19) just because it's possible to find cheap devices running Android, but usually only with old Android versions. Technically, it doesn't see difficult to support API 19+ (instead of 21+), a few tiny patches are needed. But I don't want to have a "full support" of API 19+, only basic support like "make sure that the compilation doesn't fail", not "all tests must pass".
It seems like sys.platform == 'android' would be more appropriate since Android is not Linux: different libc, different filesystems, etc.
I've had a similar thought myself.
-Brett
While Xavier promotes cross-compilation, Paul would like to build Python directly on Android to get pip and wheels.
Honestly, I have no strong opinion, since I don't know well Android. I'm trying to help everybody working on the Android support. IMHO it's fine to support multiple ways to build Python for Android. It's not like there is very obvious option which has no drawback... Cross compilation is complex, getting a C compiler on Android also seems to be complex. From my point of view, compared to a common Fedora Linux, Android doesn't seem easy to use to develop on CPython...
Victor
2017-12-10 15:19 GMT+01:00 Xavier de Gaye <xdegaye@gmail.com>:
The following note is a proposal to add the support of the Android platform.
The note is easier to read with clickable links at https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst
Motivations ===========
* Android is ubiquitous. * This would be the first platform supported by Python that is cross-compiled, thanks to many contributors. * Although the Android operating system is linux, it is different from most linux platforms, for example it does not use GNU libc and runs SELinux in enforcing mode. Therefore supporting this platform would make Python more robust and also would allow testing it on arm 64-bit processors. * Python running on Android is also a handheld calculator, a successor of the slide rule and the `HP 41`_.
Current status ==============
* The Python test suite succeeds when run on Android emulators using buildbot strenuous settings with the following architectures on API 24: x86, x86_64, armv7 and arm64. * The `Android build system`_ is described in another section. * The `buildmaster-config PR 26`_ proposes to update ``master.cfg`` to enable buildbots to run a given Android API and architecture on the emulators. * The Android emulator is actually ``qemu``, so the test suites for x86 and x86_64 last about the same time as the test suite run natively when the processor of the build system is of the x86 family. The test suites for the arm architectures last much longer: about 8 hours for arm64 and 10 hours for armv7 on a four years old laptop. * The changes that have been made to achieve this status are listed in `bpo-26865`_, the Android meta-issue. * Given the cpu resources required to run the test suite on the arm emulators, it may be difficult to find a contributed buildbot worker. So it remains to find the hardware to run these buildbots.
Proposal ========
Support the Android platform on API 24 [1]_ for the x86_64, armv7 and arm64 architectures built with NDK 14b.
*API 24* * API 21 is the first version to provide usable support for wide characters and where SELinux is run in enforcing mode.
* API 22 introduces an annoying bug on the linker that prints something like this when python is started::
``WARNING: linker: libpython3.6m.so.1.0: unused DT entry: type 0x6ffffffe arg 0x14554``.
The `termux`_ Android terminal emulator describes this problem at the end of its `termux-packages`_ gitlab page and has implemented a ``termux-elf-cleaner`` tool to strip the useless entries from the ELF header of executables.
* API 24 is the first version where the `adb`_ shell is run on the emulator as a ``shell`` user instead of the ``root`` user previously, and the first version that supports arm64.
*x86_64* It seems that no handheld device exists using that architecture. It is supported because the x86_64 Android emulator runs fast and therefore is a good candidate as a buildbot worker.
*NDK 14b* This release of the NDK is the first one to use `Unified headers`_ fixing numerous problems that had been fixed by updating the Python configure script until now (those changes have been reverted by now).
Android idiosyncrasies ======================
* The default shell is ``/system/bin/sh``. * The file system layout is not a traditional unix layout, there is no ``/tmp`` for example. Most directories have user restricted access, ``/sdcard`` is mounted as ``noexec`` for example. * The (java) applications are allocated a unix user id and a subdirectory on ``/data/data``. * SELinux is run in enforcing mode. * Shared memory and semaphores are not supported. * The default encoding is UTF-8.
Android build system ====================
The Android build system is implemented at `bpo-30386`_ with `PR 1629`_ and is documented by its `README`_. It provides the following features:
* To build a distribution for a device or an emulator with a given API level and a given architecture. * To start the emulator and + install the distribution + start a remote interactive shell + or run remotely a python command + or run remotely the buildbottest * Run gdb on the python process that is running on the emulator with python pretty-printing.
The build system adds the ``Android/`` directory and the ``configure-android`` script to the root of the Python source directory on the master branch without modifying any other file. The build system can be installed, upgraded (i.e. the SDK and NDK) and run remotely, through ssh for example.
The following external libraries, when they are configured in the build system, are downloaded from the internet and cross-compiled (only once, on the first run of the build system) before the cross-compilation of the extension modules:
* ``ncurses`` * ``readline`` * ``sqlite`` * ``libffi`` * ``openssl``, the cross-compilation of openssl fails on x86_64 and arm64 and this step is skipped on those architectures.
The following extension modules are disabled by adding them to the ``*disabled*`` section of ``Modules/Setup``:
* ``_uuid``, Android has no uuid/uuid.h header. * ``grp`` some grp.h functions are not declared. * ``_crypt``, Android does not have crypt.h. * ``_ctypes`` on x86_64 where all long double tests fail (`bpo-32202`_) and on arm64 (see `bpo-32203`_).
.. [1] On Wikipedia `Android version history`_ lists the correspondence between API level, commercial name and version for each release. It also provides information on the global Android version distribution, see the two charts on top.
.. _`README`: https://github.com/xdegaye/cpython/blob/bpo-30386/Android/README.rst .. _`bpo-26865`: https://bugs.python.org/issue26865 .. _`bpo-30386`: https://bugs.python.org/issue30386 .. _`bpo-32202`: https://bugs.python.org/issue32202 .. _`bpo-32203`: https://bugs.python.org/issue32203 .. _`PR 1629`: https://github.com/python/cpython/pull/1629 .. _`buildmaster-config PR 26`: https://github.com/python/buildmaster-config/pull/26 .. _`Android version history`: https://en.wikipedia.org/wiki/Android_version_history .. _`termux`: https://termux.com/ .. _`termux-packages`: https://gitlab.com/jbwhips883/termux-packages .. _`adb`: https://developer.android.com/studio/command-line/adb.html .. _`Unified headers`:
https://android.googlesource.com/platform/ndk.git/+/ndk-r14-release/docs/Uni... .. _`HP 41`: https://en.wikipedia.org/wiki/HP-41C .. vim:filetype=rst:tw=78:ts=8:sts=2:sw=2:et: _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.co...
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
For this to move forward more rapidly it would really help if there were a utility VM appliance available with a ready-installed support an SDK. Or at least that would lower impedance to joining the development effort. Does any such beast by chance exist? S Steve Holden On Tue, Jan 23, 2018 at 5:17 PM, Victor Stinner <victor.stinner@gmail.com> wrote:
Ok, I created https://bugs.python.org/issue32637 "Android: set sys.platform and os.name to android"
Victor
2018-01-23 18:01 GMT+01:00 Brett Cannon <brett@python.org>:
On Mon, 22 Jan 2018 at 09:29 Victor Stinner <victor.stinner@gmail.com> wrote:
Hi,
I'm still talking with Paul Peny (pmpp on IRC) who is trying to build the master branch of Python on Android, using cross-compilation or directly on an Android device. I started to took notes since Android is a complex platforms and it's not easy for me to remember everything.
http://vstinner.readthedocs.io/python_android.html
Paul would like to support Android 4.4 Kitkat (API 19) just because it's possible to find cheap devices running Android, but usually only with old Android versions. Technically, it doesn't see difficult to support API 19+ (instead of 21+), a few tiny patches are needed. But I don't want to have a "full support" of API 19+, only basic support like "make sure that the compilation doesn't fail", not "all tests must pass".
It seems like sys.platform == 'android' would be more appropriate since Android is not Linux: different libc, different filesystems, etc.
I've had a similar thought myself.
-Brett
While Xavier promotes cross-compilation, Paul would like to build Python directly on Android to get pip and wheels.
Honestly, I have no strong opinion, since I don't know well Android. I'm trying to help everybody working on the Android support. IMHO it's fine to support multiple ways to build Python for Android. It's not like there is very obvious option which has no drawback... Cross compilation is complex, getting a C compiler on Android also seems to be complex. From my point of view, compared to a common Fedora Linux, Android doesn't seem easy to use to develop on CPython...
Victor
2017-12-10 15:19 GMT+01:00 Xavier de Gaye <xdegaye@gmail.com>:
The following note is a proposal to add the support of the Android platform.
The note is easier to read with clickable links at https://github.com/xdegaye/cagibi/blob/master/doc/android_support.rst
Motivations ===========
* Android is ubiquitous. * This would be the first platform supported by Python that is cross-compiled, thanks to many contributors. * Although the Android operating system is linux, it is different from most linux platforms, for example it does not use GNU libc and runs
in enforcing mode. Therefore supporting this platform would make Python more robust and also would allow testing it on arm 64-bit processors. * Python running on Android is also a handheld calculator, a successor of the slide rule and the `HP 41`_.
Current status ==============
* The Python test suite succeeds when run on Android emulators using buildbot strenuous settings with the following architectures on API 24: x86, x86_64, armv7 and arm64. * The `Android build system`_ is described in another section. * The `buildmaster-config PR 26`_ proposes to update ``master.cfg`` to enable buildbots to run a given Android API and architecture on the emulators. * The Android emulator is actually ``qemu``, so the test suites for x86 and x86_64 last about the same time as the test suite run natively when the processor of the build system is of the x86 family. The test suites for the arm architectures last much longer: about 8 hours for arm64 and 10 hours for armv7 on a four years old laptop. * The changes that have been made to achieve this status are listed in `bpo-26865`_, the Android meta-issue. * Given the cpu resources required to run the test suite on the arm emulators, it may be difficult to find a contributed buildbot worker. So it remains to find the hardware to run these buildbots.
Proposal ========
Support the Android platform on API 24 [1]_ for the x86_64, armv7 and arm64 architectures built with NDK 14b.
*API 24* * API 21 is the first version to provide usable support for wide characters and where SELinux is run in enforcing mode.
* API 22 introduces an annoying bug on the linker that prints something like this when python is started::
``WARNING: linker: libpython3.6m.so.1.0: unused DT entry: type 0x6ffffffe arg 0x14554``.
The `termux`_ Android terminal emulator describes this problem at the end of its `termux-packages`_ gitlab page and has implemented a ``termux-elf-cleaner`` tool to strip the useless entries from the ELF header of executables.
* API 24 is the first version where the `adb`_ shell is run on the emulator as a ``shell`` user instead of the ``root`` user previously, and
first version that supports arm64.
*x86_64* It seems that no handheld device exists using that architecture. It is supported because the x86_64 Android emulator runs fast and
SELinux the therefore
is a good candidate as a buildbot worker.
*NDK 14b* This release of the NDK is the first one to use `Unified headers`_ fixing numerous problems that had been fixed by updating the Python configure script until now (those changes have been reverted by now).
Android idiosyncrasies ======================
* The default shell is ``/system/bin/sh``. * The file system layout is not a traditional unix layout, there is no ``/tmp`` for example. Most directories have user restricted access, ``/sdcard`` is mounted as ``noexec`` for example. * The (java) applications are allocated a unix user id and a subdirectory on ``/data/data``. * SELinux is run in enforcing mode. * Shared memory and semaphores are not supported. * The default encoding is UTF-8.
Android build system ====================
The Android build system is implemented at `bpo-30386`_ with `PR 1629`_ and is documented by its `README`_. It provides the following features:
* To build a distribution for a device or an emulator with a given API level and a given architecture. * To start the emulator and + install the distribution + start a remote interactive shell + or run remotely a python command + or run remotely the buildbottest * Run gdb on the python process that is running on the emulator with python pretty-printing.
The build system adds the ``Android/`` directory and the ``configure-android`` script to the root of the Python source directory on the master branch without modifying any other file. The build system can be installed, upgraded (i.e. the SDK and NDK) and run remotely, through ssh for example.
The following external libraries, when they are configured in the build system, are downloaded from the internet and cross-compiled (only once, on the first run of the build system) before the cross-compilation of the extension modules:
* ``ncurses`` * ``readline`` * ``sqlite`` * ``libffi`` * ``openssl``, the cross-compilation of openssl fails on x86_64 and arm64 and this step is skipped on those architectures.
The following extension modules are disabled by adding them to the ``*disabled*`` section of ``Modules/Setup``:
* ``_uuid``, Android has no uuid/uuid.h header. * ``grp`` some grp.h functions are not declared. * ``_crypt``, Android does not have crypt.h. * ``_ctypes`` on x86_64 where all long double tests fail (`bpo-32202`_) and on arm64 (see `bpo-32203`_).
.. [1] On Wikipedia `Android version history`_ lists the correspondence between API level, commercial name and version for each release. It also provides information on the global Android version distribution, see the two charts on top.
.. _`README`: https://github.com/xdegaye/cpython/blob/bpo-30386/Android/README.rst .. _`bpo-26865`: https://bugs.python.org/issue26865 .. _`bpo-30386`: https://bugs.python.org/issue30386 .. _`bpo-32202`: https://bugs.python.org/issue32202 .. _`bpo-32203`: https://bugs.python.org/issue32203 .. _`PR 1629`: https://github.com/python/cpython/pull/1629 .. _`buildmaster-config PR 26`: https://github.com/python/buildmaster-config/pull/26 .. _`Android version history`: https://en.wikipedia.org/wiki/Android_version_history .. _`termux`: https://termux.com/ .. _`termux-packages`: https://gitlab.com/jbwhips883/termux-packages .. _`adb`: https://developer.android.com/studio/command-line/adb.html .. _`Unified headers`:
https://android.googlesource.com/platform/ndk.git/+/ndk- r14-release/docs/UnifiedHeaders.md .. _`HP 41`: https://en.wikipedia.org/wiki/HP-41C .. vim:filetype=rst:tw=78:ts=8:sts=2:sw=2:et: _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ victor.stinner%40gmail.com
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ steve%40holdenweb.com
participants (11)
-
Antoine Pitrou -
Brett Cannon -
Glenn Linderman -
Guido van Rossum -
Guido van Rossum -
Mike Miller -
Ned Deily -
Skip Montanaro -
Steve Holden -
Victor Stinner -
Xavier de Gaye