Hi Python Developers, I'm Paul Monson, I've spent about 20 years working with embedded software. Since 2010 I've worked for Microsoft as a developer. Our team is working with CPython on Azure IoT Edge devices that run on x64-based devices. We would like to extend that support to Windows running on ARM 32-bit devices and have a working proof-of-concept. Our team is prepared to provide support for CPython for Windows on ARM32 for 10 years, and to provide build bots for ARM32. I like to propose that the initial sequence of PRs could be: - Update to OpenSSL 1.1.1 (without anything ARM specific) - ready to go - Migrate to libffi directly (finish https://github.com/python/cpython/pull/3806<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpython%2Fcpython%2Fpull%2F3806&data=02%7C01%7CPaul.Monson%40microsoft.com%7C0e38d9367d2d4240725d08d680dedde6%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636838092030320471&sdata=7Mgsyy9SifmIRroagp1NZ0cx77vhRC%2BzxpUoHSQblB4%3D&reserved=0>) - Build file updates for OpenSSL ARM and check into cpython-bin-deps - Build file updates for CPython ARM - ctypes updates for ARM - Test module skips for ARM - Library updates and related test fixes for ARM Updating OpenSSL and libffi are independent of ARM support but need to be done as prerequisites. OpenSSL 1.1.0 doesn't have support for ARM32 on Windows but OpenSSL 1.1.1 does. I have OpenSSL 1.1.1a ready to check in to master with all tests passing on x86 and x64 on Windows. Since work has already been done on this for other platforms only very small changes were needed for Windows. I have also integrated and tested the current libffi on Windows x64. Some additonal porting of x86 assembler to MSVC tools will need to be done. I have a working port of ARM32 assembler for MSVC but it may need to be brought up to date and cleaned up. The last four all need to go in together, but can be reviewed separately. We are not planning to support Tk/Tcl on ARM32 because Windows IoT Core, Windows containers don't support GDI, which is a depenency of Tk/Tcl. Since Window IoT Core and Windows container don't support the .msi or .exe installers found on python.org my team at Microsoft will build the CPython for Windows ARM32 from the official repo and distribute it. Thanks in advance, Paul
Just confirming for the list that I'm aware of this and supportive, but am not the dedicated support for this effort. I also haven't reviewed the changes yet, but provided nobody is strongly opposed to taking on a supported platform (without additional releases on python.org), I expect I'll do a big part of the reviewing then. Cheers, Steve On 05Feb.2019 1709, Paul Monson via Python-Dev wrote:
Hi Python Developers,
I'm Paul Monson, I've spent about 20 years working with embedded software. Since 2010 I've worked for Microsoft as a developer.
Our team is working with CPython on Azure IoT Edge devices that run on x64-based devices.
We would like to extend that support to Windows running on ARM 32-bit devices and have a working proof-of-concept. Our team is prepared to provide support for CPython for Windows on ARM32 for 10 years, and to provide build bots for ARM32.
I like to propose that the initial sequence of PRs could be: - Update to OpenSSL 1.1.1 (without anything ARM specific) - ready to go - Migrate to libffi directly (finish https://github.com/python/cpython/pull/3806) - Build file updates for OpenSSL ARM and check into cpython-bin-deps - Build file updates for CPython ARM - ctypes updates for ARM - Test module skips for ARM - Library updates and related test fixes for ARM
Updating OpenSSL and libffi are independent of ARM support but need to be done as prerequisites. OpenSSL 1.1.0 doesn't have support for ARM32 on Windows but OpenSSL 1.1.1 does.
I have OpenSSL 1.1.1a ready to check in to master with all tests passing on x86 and x64 on Windows. Since work has already been done on this for other platforms only very small changes were needed for Windows.
I have also integrated and tested the current libffi on Windows x64. Some additonal porting of x86 assembler to MSVC tools will need to be done. I have a working port of ARM32 assembler for MSVC but it may need to be brought up to date and cleaned up.
The last four all need to go in together, but can be reviewed separately.
We are not planning to support Tk/Tcl on ARM32 because Windows IoT Core, Windows containers don't support GDI, which is a depenency of Tk/Tcl.
Since Window IoT Core and Windows container don't support the .msi or .exe installers found on python.org my team at Microsoft will build the CPython for Windows ARM32 from the official repo and distribute it.
Thanks in advance,
Paul
On Tue, Feb 5, 2019 at 7:37 PM Steve Dower <steve.dower@python.org> wrote:
I also haven't reviewed the changes yet, but provided nobody is strongly opposed to taking on a supported platform (without additional releases on python.org), I expect I'll do a big part of the reviewing then.
I'm all for the first two changes (especially the second), and if 10 years of pledged corporate support for a new platform is the price we have to pay for them, I'm ok with that :). I expect I'll be automatically added to any issues/PRs that come of this, but I'll keep an eye out for them anyway and give reviews as I'm able. I'll also help get the build bots set up when we're ready for them. -- Zach
On 2/5/2019 10:10 PM, Zachary Ware wrote:
I'm all for the first two changes (especially the second), and if 10 years of pledged corporate support for a new platform is the price we have to pay for them, I'm ok with that :).
I would expect that the main question should be the density of WinArm32-specific ifdefs in the main code and extensions other than ctypes. -- Terry Jan Reedy
On 06Feb2019 0054, Terry Reedy wrote:
On 2/5/2019 10:10 PM, Zachary Ware wrote:
I'm all for the first two changes (especially the second), and if 10 years of pledged corporate support for a new platform is the price we have to pay for them, I'm ok with that :).
I would expect that the main question should be the density of WinArm32-specific ifdefs in the main code and extensions other than ctypes.
Agreed. I've asked Paul to post the "final" PR early, even though it will take some refactoring as other PRs go in, so that we can see the broader picture now. There's also an option to create an ARM-specific pyconfig.h if necessary, but I don't believe it will be. I created https://bugs.python.org/issue35920 for this work. Cheers, Steve
The PR is here: https://github.com/python/cpython/pull/11774 Searching _M_ARM I see these #ifdef changes outside of ctypes: * Include\pyport.h - adds on to existing MSVC ifdef * Include\pythonrun.h - adds on to existing MSVC ifdef * Modules\_decimal\libmpdec\bits.h * Python\ceval.c - workaround compiler bug, could be replaced with #pragma optimize around entire function. -----Original Message----- From: Steve Dower <steve.dower@python.org> Sent: Wednesday, February 6, 2019 11:16 AM To: Terry Reedy <tjreedy@udel.edu>; python-dev@python.org; Paul Monson <Paul.Monson@microsoft.com> Subject: Re: [Python-Dev] CPython on Windows ARM32 On 06Feb2019 0054, Terry Reedy wrote:
On 2/5/2019 10:10 PM, Zachary Ware wrote:
I'm all for the first two changes (especially the second), and if 10 years of pledged corporate support for a new platform is the price we have to pay for them, I'm ok with that :).
I would expect that the main question should be the density of WinArm32-specific ifdefs in the main code and extensions other than ctypes.
Agreed. I've asked Paul to post the "final" PR early, even though it will take some refactoring as other PRs go in, so that we can see the broader picture now. There's also an option to create an ARM-specific pyconfig.h if necessary, but I don't believe it will be. I created https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.pytho... for this work. Cheers, Steve
On 06/02/2019 02.09, Paul Monson via Python-Dev wrote:
Updating OpenSSL and libffi are independent of ARM support but need to be done as prerequisites. OpenSSL 1.1.0 doesn't have support for ARM32 on Windows but OpenSSL 1.1.1 does.
I have OpenSSL 1.1.1a ready to check in to master with all tests passing on x86 and x64 on Windows. Since work has already been done on this for other platforms only very small changes were needed for Windows.
+1 for OpenSSL 1.1.1 from the maintainer of the ssl module. The new version also introduces TLS 1.3 support. Linux distributions have been switching to OpenSSL 1.1.1 for a while. If it's good enough for RHEL 8, then it's good enough for us, too. Do you want to update Python 3.8 (master) only or also 3.7? I'm not strictly against updating 3.7. However we have traditionally kept the OpenSSL version of each branch stable. 1.1.1 comes with new features, stricter security settings and some ciphers removed. Christian
On 06Feb2019 1423, Christian Heimes wrote:
Do you want to update Python 3.8 (master) only or also 3.7? I'm not strictly against updating 3.7. However we have traditionally kept the OpenSSL version of each branch stable. 1.1.1 comes with new features, stricter security settings and some ciphers removed.
I would prefer to stay on 1.1.0 for 3.7, but it's up to the release manager. Cheers, Steve
On Feb 6, 2019, at 18:28, Steve Dower <steve.dower@python.org> wrote:
On 06Feb2019 1423, Christian Heimes wrote:
Do you want to update Python 3.8 (master) only or also 3.7? I'm not strictly against updating 3.7. However we have traditionally kept the OpenSSL version of each branch stable. 1.1.1 comes with new features, stricter security settings and some ciphers removed. I would prefer to stay on 1.1.0 for 3.7, but it's up to the release manager.
Me, too. I am concerned that 1.1.1 support has not had a lot of exposure yet. Even the "What's New" document for 3.7 states: "The ssl module has preliminary and experimental support for TLS 1.3 and OpenSSL 1.1.1. " I am OK with fixes for 1.1.1 support but I think it would be premature to change the Windows and/or macOS installers from 1.1.0 to 1.1.1. -- Ned Deily nad@python.org -- []
On 07/02/2019 00.41, Ned Deily wrote:
On Feb 6, 2019, at 18:28, Steve Dower <steve.dower@python.org> wrote:
On 06Feb2019 1423, Christian Heimes wrote:
Do you want to update Python 3.8 (master) only or also 3.7? I'm not strictly against updating 3.7. However we have traditionally kept the OpenSSL version of each branch stable. 1.1.1 comes with new features, stricter security settings and some ciphers removed. I would prefer to stay on 1.1.0 for 3.7, but it's up to the release manager.
Me, too. I am concerned that 1.1.1 support has not had a lot of exposure yet. Even the "What's New" document for 3.7 states: "The ssl module has preliminary and experimental support for TLS 1.3 and OpenSSL 1.1.1. "
That's from the alpha and beta phase of OpenSSL. Support for 1.1.1 is as stable as it can get.
I am OK with fixes for 1.1.1 support but I think it would be premature to change the Windows and/or macOS installers from 1.1.0 to 1.1.1.
1.1.1a is a solid release. Debian testing, Fedora, and RHEL 8 beta have been shipping and testing 1.1.1 for a while. In my professional opinion it's less about stability but more about backwards compatibility issues. TLS 1.3 behaves slightly differently and 1.1.1 has dropped some weak ciphers. Christian
Here are the current OpenSSL 1.1.1a changes I have, in a seperate PR I did some additional testing and have some test failures to investigate tomorrows test_parse_cert_CVE_2019_5010 only fails win32 debug (access violation) works for amd64 debug/release and win32 release test_load_default_certs_env_windows fails on win32 and amd64 retail. skipped on debug -----Original Message----- From: Steve Dower <steve.dower@python.org> Sent: Wednesday, February 6, 2019 3:28 PM To: Christian Heimes <christian@python.org>; Paul Monson <Paul.Monson@microsoft.com>; python-dev@python.org; Ned Deily <nad@python.org> Subject: Re: [Python-Dev] CPython on Windows ARM32 On 06Feb2019 1423, Christian Heimes wrote:
Do you want to update Python 3.8 (master) only or also 3.7? I'm not strictly against updating 3.7. However we have traditionally kept the OpenSSL version of each branch stable. 1.1.1 comes with new features, stricter security settings and some ciphers removed.
I would prefer to stay on 1.1.0 for 3.7, but it's up to the release manager. Cheers, Steve
participants (6)
-
Christian Heimes -
Ned Deily -
Paul Monson -
Steve Dower -
Terry Reedy -
Zachary Ware