Mailman 3 February 2018 - Python-Dev

Re: [Python-Dev] Why is Python for Windows compiled with MSVC?
by Gregory P. Smith 01 Feb '18

01 Feb '18

TL;DR of Steve's post - MSVC is the compiler of choice for most serious software on Windows. So we use it to best integrate with the world. There is no compelling reason to change that. The free-as-in-beer MSVC community edition is finally non-sucky (their earlier efforts were crippled, they seem to have learned the lesson) There are other viable Windows compilers. If we want to support those in CPython someone needs to contribute the work to do so, ongoing maintenance, and buildbots. I'd love to see a Clang based Windows build (Google Chrome is built using that). But I have no motivating reason to do the work. I *believe* such a build could be made to integrate and inter-operate fully with MSVC builds and ABIs. We could *probably* even make cross-compilation of extensions from Linux -> Windows work that way. We're highly unlikely to ever stop shipping python.org Windows binaries built with anything other than MSVC unless Microsoft takes a turn toward the dark side again. -gps On Wed, Jan 31, 2018 at 3:07 PM Steve Dower <steve.dower(a)python.org> wrote: > Because every other supported platform builds using the native tools, so > why shouldn’t the one with the most users? > > > > I’m likely biased because I work there and I’m the main intermediary with > python-dev, but these days Microsoft is one of the strongest supporters of > CPython. We employ the most core developers of any private company and we > all are allowed work time to contribute, we provide full access to our > development tools and platforms to all core developers and some prominent > projects, we’ve made fixes, enhancements and releases or core products such > as the CRT, MSVC, Visual Studio, Visual Studio Code, and Azure SPECIFICALLY > to support CPython development and users. As far as I know, ALL the Windows > buildbots are running on Azure subscriptions that Microsoft provides > (though managed by some awesome volunteers). You’ll see us at PyCon US > under the biggest banner and we’ll have a booth filled with engineers and > not recruiters. Crash reports from thousands of opted-in users come into > our systems and have directly lead to both CPython and Windows bug fixes. > > > > Meanwhile, most of the MinGW contributions have been complaints and > drive-by patches. We (python-dev) are not opposed to supporting a second > compiler for Windows, and honestly I’d love for extensions built with other > compilers to be fully compatible with our main binary release, but the > sacrifice involved in switching is significant and there’s no apparent > commitment from the alternative options. > > > > (Note that I’m not saying Microsoft’s support is conditional on our > compiler being used. But our ability to contribute technically would be > greatly reduced if we didn’t have the inside access that we do.) > > > > And as has been mentioned, MSVC was selected before the other options were > feasible. Python is a much older tool than those others, and so uses the > tools that were best at the time. > > > > So in my opinion at least, the reasoning for selecting MSVC was perfectly > sound, and the reasoning for continuing with it is perfectly sound. > Unwillingness on the part of package developers to not even test on Windows > before releasing a wheel for it is not a compelling reason to change > anything. > > > > Cheers, > > Steve > > > > Top-posted from my Windows phone > > > > *From: *Oleg Sivokon <olegs(a)traiana.com> > *Sent: *Thursday, February 1, 2018 5:40 > *To: *python-dev(a)python.org > *Subject: *[Python-Dev] Why is Python for Windows compiled with MSVC? > > > > Hello list. > > > > I'll give some background before asking my question in more detail. > > > > I've been tasked with writing some infrastructure code that needs to talk > to Kubernetes. (Kubernetes is a popular software for managing and > automating virtualization / containerization of cloud services). One of > the requirements was that the code be written in Python 3.X. > > > > The tasks my code was supposed to perform on Kubernetes would be something > like cluster creation from specification, deletion of all or parts of the > cluster, providing realtime statistics of cluster usage etc. There were > few prototype scripts written in Bash using kubectl (official client > written in Go). > > > > My first reaction was to look for an official client for Kubernetes > written in Python. There is one official client for Kubernetes, with a > single maintainer, impossible to parse documentation, containing mostly > generated code. It is nigh impossible to use. Here I need to explain that > for whatever reason Kubernetes team decided to write their HTTP API in such > a way that the server is "dumb" and the client must be "smart" in order to > do anything useful. For instance, if you have a description of your > cluster, you cannot just send this description to the server and hope that > it will know how to create the cluster from description. You need to make > multiple API calls (perhaps hundreds of them) to arrange for the server to > create the cluster from description. > > > > Since the official client is no help (it really only mirrors the HTTP > API), I searched for other clients. There are two more. None is in good > shape, and none comes even close to being able to do what kubectl can. > > > > There is one more client that shells out calls to kubectl. It implements > only a small subset of kubectl commands, and... it's a lot of parsing of > standard output with regular expressions and magic. > > > > Well... I was given a lot of time to investigate other options for dealing > with this project, so I decided, what if I can compile kubectl into a > shared library and write a Python extension that links against that > library. And, indeed, after few days I came up with such an extension. It > worked!.. On Linux... > > > > Now all I had to do was to re-create my success on Windows (most of the > employees in my company use Windows). At first I thought that I'd > cross-compile on Linux using MinGW. I compiled Go shared library into a > DLL, then tried to compile my Python extension and... it didn't work. I > downloaded VirtualBox and some Windows images, etc... tried to compile on > Windows. It didn't work. I started asking around, and was told that even > though for some earlier versions of Python this was kind of possible, for > Python 3.5, 3.6 it is not. You must use MSVC to compile Python > extensions. No way around it. > > Now, since Go won't compile with MSVC, I'll have to scrap my project and > spend many weeks re-implementing kubectl. > > > > Here's my question: Why? > > > > Why did you choose to use non-free compiler, which also makes > cross-compilation impossible? There wasn't really a reason not to choose > MinGW as a way to compile extensions on Windows (Ruby does that, Go uses > MinGW, perhaps some others too). It would've made things like CI and > packaging so much easier... What do Python users / developers get from > using MSVC instead? > > > > Thank you. > > > > Oleg > > This communication and all information contained in or attached to it is > confidential, intended solely for the addressee, may be legally privileged > and is the intellectual property of one of the companies of NEX Group plc > ("NEX") or third parties. If you are not the intended addressee or receive > this message in error, please immediately delete all copies of it and > notify the sender. We have taken precautions to minimise the risk of > transmitting software viruses, but we advise you to carry out your own > virus checks on any attachments. We do not accept liability for any loss or > damage caused by software viruses. NEX reserves the right to monitor all > communications. We do not accept any legal responsibility for the content > of communications, and no communication shall be considered legally > binding. Furthermore, if the content of this communication is personal or > unconnected with our business, we accept no liability or responsibility for > it. NEX Group plc is a public limited company regi > > stered in England and Wales under number 10013770 and certain of its > affiliates are authorised and regulated by regulatory authorities. For > further regulatory information please see www.NEX.com. > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev(a)python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org > > > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org >

3 2

Why is Python for Windows compiled with MSVC?
by Oleg Sivokon 01 Feb '18

01 Feb '18

Hello list. I'll give some background before asking my question in more detail. I've been tasked with writing some infrastructure code that needs to talk to Kubernetes. (Kubernetes is a popular software for managing and automating virtualization / containerization of cloud services). One of the requirements was that the code be written in Python 3.X. The tasks my code was supposed to perform on Kubernetes would be something like cluster creation from specification, deletion of all or parts of the cluster, providing realtime statistics of cluster usage etc. There were few prototype scripts written in Bash using kubectl (official client written in Go). My first reaction was to look for an official client for Kubernetes written in Python. There is one official client for Kubernetes, with a single maintainer, impossible to parse documentation, containing mostly generated code. It is nigh impossible to use. Here I need to explain that for whatever reason Kubernetes team decided to write their HTTP API in such a way that the server is "dumb" and the client must be "smart" in order to do anything useful. For instance, if you have a description of your cluster, you cannot just send this description to the server and hope that it will know how to create the cluster from description. You need to make multiple API calls (perhaps hundreds of them) to arrange for the server to create the cluster from description. Since the official client is no help (it really only mirrors the HTTP API), I searched for other clients. There are two more. None is in good shape, and none comes even close to being able to do what kubectl can. There is one more client that shells out calls to kubectl. It implements only a small subset of kubectl commands, and... it's a lot of parsing of standard output with regular expressions and magic. Well... I was given a lot of time to investigate other options for dealing with this project, so I decided, what if I can compile kubectl into a shared library and write a Python extension that links against that library. And, indeed, after few days I came up with such an extension. It worked!.. On Linux... Now all I had to do was to re-create my success on Windows (most of the employees in my company use Windows). At first I thought that I'd cross-compile on Linux using MinGW. I compiled Go shared library into a DLL, then tried to compile my Python extension and... it didn't work. I downloaded VirtualBox and some Windows images, etc... tried to compile on Windows. It didn't work. I started asking around, and was told that even though for some earlier versions of Python this was kind of possible, for Python 3.5, 3.6 it is not. You must use MSVC to compile Python extensions. No way around it. Now, since Go won't compile with MSVC, I'll have to scrap my project and spend many weeks re-implementing kubectl. Here's my question: Why? Why did you choose to use non-free compiler, which also makes cross-compilation impossible? There wasn't really a reason not to choose MinGW as a way to compile extensions on Windows (Ruby does that, Go uses MinGW, perhaps some others too). It would've made things like CI and packaging so much easier... What do Python users / developers get from using MSVC instead? Thank you. Oleg This communication and all information contained in or attached to it is confidential, intended solely for the addressee, may be legally privileged and is the intellectual property of one of the companies of NEX Group plc ("NEX") or third parties. If you are not the intended addressee or receive this message in error, please immediately delete all copies of it and notify the sender. We have taken precautions to minimise the risk of transmitting software viruses, but we advise you to carry out your own virus checks on any attachments. We do not accept liability for any loss or damage caused by software viruses. NEX reserves the right to monitor all communications. We do not accept any legal responsibility for the content of communications, and no communication shall be considered legally binding. Furthermore, if the content of this communication is personal or unconnected with our business, we accept no liability or responsibility for it. NEX Group plc is a public limited company registered in England and Wales under number 10013770 and certain of its affiliates are authorised and regulated by regulatory authorities. For further regulatory information please see www.NEX.com.

6 5

[IMPORTANT] post 3.7.0b1 development now open
by Ned Deily 01 Feb '18

01 Feb '18

Here we are: 3.7.0b1 and feature code freeze! Congratulations and thanks to all of you who've contributed to the huge number of PEPs, features, bug fixes, and doc changes that have gone into 3.7 since feature development began back in September 2016, after 3.6.0b1, 3.6's feature freeze. Now that feature development for 3.7 is over, the challenge is to put the finishing touches on the features and documentation, squash bugs, and test test test. In the cpython repo, there is now a 3.7 branch. Starting now, all PRs destined for 3.7.0 should get cherry-picked from master to the 3.7 branch or just pushed to 3.7 if only applicable there. New features should continue to be pushed to the master branch for release in 3.8; no new features are now permitted in 3.7 unless you have contacted me and we have agreed on an extension (and all granted extensions will expire as of 3.7.0b2). As before, bug fixes appropriate for 3.6.x should continue to be cherry-picked to the 3.6 branch. I've updated the Developer's Guide to reflect the now current workflow. Let me know if you find any bugs in it. Likewise, please contact me if you have any questions about the workflow or about whether a change is appropriate for 3.7 beta. The cpython repo on Github has been updated. You should now find that builds on the master branch produce a Python 3.8, rather than 3.7; you may want to clean your build directory. And there is now a 3.7 branch that you will need to use for 3.7 builds and pushs. There were several PRs that were merged to master over the last couple of days since we started 3.7.0b1 release engineering. All but one of those have been cherry-picked into the new 3.7 branch and you should have seen messages for them. One was for a 3.8 feature and so was not backported. At the moment, the new 3.7 buildbots may not be fully operational but they should be soon. Likewise, the docs.python.org may take up to 24 hours to reflect all the changes. Note that is the first time we've done a feature freeze using our new git-based workflow, so there's likely that there might be a glitch or something overlooked. Please let us know if you suspect something or have a question. I'll be around here and or #python-dev. Also, don't forget to direct 3.8-related questions to Łukasz. Welcome on-board! To recap: 2018-01-31: 3.7 branch open for 3.7.0; 3.8.0 feature development begins 2018-01-31 to 2018-05-21: 3.7.0 beta phase (no new 3.7 features) - push PRs for new features, bugs, regressions, doc fixes to the master branch for release in 3.8 - cherry-pick or push PRs for 3.7.0 (bug/regression/doc fixes) to the new 3.7 branch - cherry-pick or push select PRs for important bug/regression/doc fixes to 3.6 and 2.7 branches as appropriate - propose PRs to 3.5 and 3.4 branches for any identified security issues 2018-02-26: 3.7.0 beta 2 (next beta preview) 2018-03-26: 3.7.0 beta 3 (3.7.0 ABI freeze) 2018-04-30: 3.7.0 beta 4 (only critical and urgent fixes after this point) 2018-05-21: 3.7.0 release candidate 1 (3.7.0 code freeze, only any emergency fixes afer this point) 2018-06-15: 3.7.0 release 2019-10-20: 3.8.0 release (next planned feature release, see PEP 569) Thank you all again for your great efforts so far on 3.7! --Ned

1 0