Python should be easily compilable on Windows with MinGW
Hi. I am currently working on adding some functionality on a standard library module (http://bugs.python.org/issue15873). The Python part went fine, but now I have to do the C counterpart, and I have ran into in several problems, which, stacked up, are a huge obstacle to easily contribute further. Currently, despite I could work, I can't go further on my patch. I am currently working in very limited network, CPU and time ressources* which are quite uncommon in the western world, but are much less in the rest of the world. I have a 2GB/month mobile data plan and a 100KB/s speed. For the C part of my patch, I should download Visual Studio. The Express Edition 2015 is roughly 9GB. I can't afford that. I downloaded Virtualbox and two Linux netinstall (Ubuntu 15.10 and Fedora 23). Shortly, I couldn't get something working quickly and simply (quickly = less than 2 hours, downloading time NOT included, which is anyway way too already much). What went wrong and why it went wrong could be a whole new thread and is outside of the scope of this message. Let me precise this : at my work I use many virtualbox instances automatically fired and run in parallel to test new deployments and run unittests. I like this tool, but despite its simple look, it (most of the time) can not be used simply by a profane. The concepts it requires you to understand are not intuitive at first sight and there is *always* a thing that go wrong (guest additions, mostly).(for example : Ubuntu and Virtualbox shipped for a moment a broken version of mount.vboxsf, preventing sharing folder to mount. Despite it's fixed, the broken releases spread everywhere and you may encounter them a lot in various Ubuntu and Virtualbox version. I downloaded the last versions of both and I am yet infected. https://www.virtualbox.org/ticket/12879). I could do whole new thread on why you can't ask newcomers to use Virtualbox (currently, at least). I ran into is a whole patch set to make CPython compile on MinGW (https://bugs.python.org/issue3871#msg199695). But it is not denying it's very experimental, and I know I would again spent useless hours trying to get it work rather than joyfully improving Python, and that's exactly what I do not want to happen. Getting ready to contribute to CPython pure python modules from an standard, average mr-everyone Windows PC for a beginner-to-medium contributor only require few megabytes of internet and few minutes of his time: getting a tarball of CPython sources (or cloning the github CPython mirror)**, a basic text editor and msys-git. The step further, if doing some -even basic- C code is required, implies downloading 9GB of Visual Studio and countless hours for it to be ready to use. I think downloading the whole Visual Studio suite is a huge stopper to contribute further for an average medium-or-below-contributor. I think (and I must not be the only one since CPython is to be moved to github), that barriers to contribute to CPython should be set to the lowest. Of course my situation is a bit special but I think it represents daily struggle of a *lot* of non-western programmer (at least for limited internet)(even here in Australia, landline limited internet connections are very common). It's not a big deal if the MinGW result build is twenty time slower or if some of the most advanced modules can't be build. But everyone programmer should be able to easily make some C hacks and get them to work. Hoping you'll be receptive to my pleas, Cheers * I am currently picking fruits in the regional Australia. I live in a van and have internet through with smartphone through an EDGE connection. I can plug the laptop in the farm but not in the van. ** No fresh programmer use mercurial unless he has a gun pointed on his head.
No. Visual Studio is a solid compiler suit, mingw is a jenky mess, especially when you try and move to 64bit (where I don't think there is one true version of mingw). I'm sorry that Visual Studio makes it very hard for you to contribute, but changing THE compiler of the distribution from the platform compiler, especially when we FINALLY got a stable abi with it, is going to be a non starter. Compiling on MinGW for your own edification is fine, but that's not the build platform for windows python, nor should it be. Contributions are, and should continue to be, tested against Visual Studio. On 2/26/2016 05:12, Mathieu Dupuy wrote:
But what do you really think? IMO, windows builds probably should do both visual studio and mingw. That is, there probably should be two builds on windows, since there's no clear consensus about which to use. I certainly prefer mingw over visual studio - and I have adequate bandwidth for either. On Fri, Feb 26, 2016 at 9:55 AM, Alexander Walters <tritium-list@sdamon.com> wrote:
Ok, fine. Bring a windows build bot online. And also take on the support burden of guiding people to which version of which compiler you use for each of the currently supported python versions. And go ahead and write the pep to change how wheel distributions work (which will effectively kill them, so yeah, good side benefit there.) Want to kill python on windows for anything that needs a c extension? go ahead, release one version of python with 2 ABIs. What do I know. On 2/26/2016 13:05, Dan Stromberg wrote:
On Fri, 26 Feb 2016 10:05:19 -0800, Dan Stromberg <drsalists@gmail.com> wrote:
I don't think there is much if any objection to the idea of making CPython compilable with mingw, we just need the official supported release to be the VS one for compatibility reasons. But, there has historically been a lack of a clear target in the mingw space for someone to actually produce a working generalized port (as opposed to, say, cygwin), much less generate a set of reviewable patches that could be incorporated in to the repository. (Among other things for the latter we would need a mingw buildbot, and no one has stepped forward on that front at all, as far as I know.) I think there has been some progress lately, but it is a hard problem and needs more volunteer time. Ideally we'd have someone who is all of passionate enough about it, knowledgeable enough about it, and patient enough to work with the others in the community who need to be involved. --David
You might be interested in a project funded by NumFOCUS + PSF to make the mingw ecosystem work with CPython. http://mingwpy.github.io/ -- Andy On Fri, Feb 26, 2016 at 12:24 PM, R. David Murray <rdmurray@bitdance.com> wrote:
On Feb 26, 2016 9:56 AM, "Alexander Walters" <tritium-list@sdamon.com> wrote:
No.
Visual Studio is a solid compiler suit, mingw is a jenky mess, especially
when you try and move to 64bit (where I don't think there is one true version of mingw). I see why you say this, but I think you're seriously underestimating mingw. There's the split between the legacy mingw and mingw-w64 fork, but that's a non-issue: legacy mingw is basically dead, everyone uses mingw-w64 for both 32- and 64-bits, no big deal. If you want a precompiled mingw-w64, then there's a ton of different distributions to choose from, but that's true of lots of F/OSS projects, including python itself. There are a few different official build configurations (e.g. "seh" vs "sjlj"), but they all have to do with the C++ abi so don't matter for python itself. The one real issue is that classically mingw-w64 has not been compatible with msvc -- mostly due to using different C runtimes, but with a few other corner case issues too. This, however, is fixable. The PSF- and NumFOCUS-sponsored mingwpy project is near to having builds of mingw-w64 that are abi compatible with all cpython releases from 2.6-3.4: http://mingwpy.github.io/ The bigger challenge is getting a version of mingw-w64 that is compatible with the latest MSVC used for cpython 3.5+, since they rearranged the runtimes (for the better, but...). Unfortunately this is also the case that's relevant for OP's use case of working on cpython itself. There's no technical obstacle to doing this, and it's a major priority for both the numerical python community and mingw-w64 upstream, but it's not currently clear who will do the work and if/how it will be funded. In any case though I agree that now is not a good time to start trying to support two incompatible windows ABIs in python upstream, just we're converging on a common abi across the different compilers. -n
On Fri, Feb 26, 2016 at 6:55 PM, Alexander Walters <tritium-list@sdamon.com> wrote:
My personal experience with psutil is that at some point I simply gave up trying to support mingw because it was too difficult and I couldn't keep up with it anymore. I had to hack through all sorts of missing stuff such as: https://github.com/giampaolo/psutil/blob/08d490d0d8fa60ee1d689cca30738ceb599..., I always wondered why mingw doesn't do it for me, plus the fact that I never really got it to work with 64-bit Python. I can only imagine what it would mean doing something similar in a much larger C code base such as Python's. The advantage of being able to use something else other than VS (which I despise as well) would undoubtedly be enormous, but from my experience mingw is an alternative which simply doesn't work. -- Giampaolo - http://grodola.blogspot.com
Hi Mathieu I just want to say that we are very aware of the concerns and issues faced here and people (including myself) are actively working to resolve them. For example, I am working with the Visual C++ team to encourage and support work such as https://blogs.msdn.microsoft.com/vcblog/2016/02/16/try-out-the-latest-c-comp..., which strips out most of that 9GB install. It currently is not sufficient for building Python or extension modules, and is likely to be 500MB+ by the time it is, but I am directly in contact with the team involved to achieve that. Once this is available, I will be looking into ways to make it easier to install the compiler. (I work at Microsoft, so I have a better ability than most to influence this sort of thing.) In parallel, there is plenty going on over on distutils-sig to improve the package ecosystem in this respect (which I know is not your immediate concern, but it always comes up as soon as someone mentions a compiler). Improved sdist and wheel capabilities are critical to being able to help non-Windows developers produce pre-compiled software for their users. Supporting a second compiler for CPython is a fairly significant task. We are willing to do it when someone volunteers to actively maintain it and manage a buildbot for it, but so far the best offer has been patches (not all of which are ready to be committed, though I haven't gone through them recently so that may have been improved). Releasing builds from a second compiler is a big compatibility concern, as it is highly likely (right now) that existing wheels will simply crash when used. Dealing with that - probably by treating wheels as incompatible - is essential before we want other distros thinking they can release a MinGW build. There's also plenty of distutils work required to support it properly, and I'm not sure how much of that should/could be done anyway. So basically, smaller packages for MSVC are on their way, and MinGW support is blocked until someone commits to supporting it for an extended period of time. Hopefully that helps clarify our position. Cheers, Steve On 26Feb2016 0212, Mathieu Dupuy wrote:
One alternative to consider is using Cygwin. A complete Cygwin environment, including a GCC toolchain, is pretty small. And it can build a *nix-style CPython that works inside the Cygwin environment. That may not be sufficient for a lot of uses, but for your purpose, it should be. Another alternative, as crazy as it may sound, is to get an AWS-Free-Tier EC2 instance and develop on that. Or, of course, buy an ancient used laptop and install linux natively. Obviously none of these are ideal, but they may still be better for you than waiting for a complete MinGW port of Python or a smaller MSVC toolchain.
The 9 gig initial download is not the only problem. Visual studio is very bandwidth hungry in day to day operations (between polling websites and vcs remotes, near constant updating, integration with the VS web service, etc.). You can of course shut all of that off, but it's a pain. It's my understanding from Steve's post that a leaner, meaner edition of VS is in the works, so waiting for that might just be an overall better solution. On 2/27/2016 16:27, Franklin? Lee wrote:
No. Visual Studio is a solid compiler suit, mingw is a jenky mess, especially when you try and move to 64bit (where I don't think there is one true version of mingw). I'm sorry that Visual Studio makes it very hard for you to contribute, but changing THE compiler of the distribution from the platform compiler, especially when we FINALLY got a stable abi with it, is going to be a non starter. Compiling on MinGW for your own edification is fine, but that's not the build platform for windows python, nor should it be. Contributions are, and should continue to be, tested against Visual Studio. On 2/26/2016 05:12, Mathieu Dupuy wrote:
But what do you really think? IMO, windows builds probably should do both visual studio and mingw. That is, there probably should be two builds on windows, since there's no clear consensus about which to use. I certainly prefer mingw over visual studio - and I have adequate bandwidth for either. On Fri, Feb 26, 2016 at 9:55 AM, Alexander Walters <tritium-list@sdamon.com> wrote:
Ok, fine. Bring a windows build bot online. And also take on the support burden of guiding people to which version of which compiler you use for each of the currently supported python versions. And go ahead and write the pep to change how wheel distributions work (which will effectively kill them, so yeah, good side benefit there.) Want to kill python on windows for anything that needs a c extension? go ahead, release one version of python with 2 ABIs. What do I know. On 2/26/2016 13:05, Dan Stromberg wrote:
On Fri, 26 Feb 2016 10:05:19 -0800, Dan Stromberg <drsalists@gmail.com> wrote:
I don't think there is much if any objection to the idea of making CPython compilable with mingw, we just need the official supported release to be the VS one for compatibility reasons. But, there has historically been a lack of a clear target in the mingw space for someone to actually produce a working generalized port (as opposed to, say, cygwin), much less generate a set of reviewable patches that could be incorporated in to the repository. (Among other things for the latter we would need a mingw buildbot, and no one has stepped forward on that front at all, as far as I know.) I think there has been some progress lately, but it is a hard problem and needs more volunteer time. Ideally we'd have someone who is all of passionate enough about it, knowledgeable enough about it, and patient enough to work with the others in the community who need to be involved. --David
You might be interested in a project funded by NumFOCUS + PSF to make the mingw ecosystem work with CPython. http://mingwpy.github.io/ -- Andy On Fri, Feb 26, 2016 at 12:24 PM, R. David Murray <rdmurray@bitdance.com> wrote:
On Feb 26, 2016 9:56 AM, "Alexander Walters" <tritium-list@sdamon.com> wrote:
No.
Visual Studio is a solid compiler suit, mingw is a jenky mess, especially
when you try and move to 64bit (where I don't think there is one true version of mingw). I see why you say this, but I think you're seriously underestimating mingw. There's the split between the legacy mingw and mingw-w64 fork, but that's a non-issue: legacy mingw is basically dead, everyone uses mingw-w64 for both 32- and 64-bits, no big deal. If you want a precompiled mingw-w64, then there's a ton of different distributions to choose from, but that's true of lots of F/OSS projects, including python itself. There are a few different official build configurations (e.g. "seh" vs "sjlj"), but they all have to do with the C++ abi so don't matter for python itself. The one real issue is that classically mingw-w64 has not been compatible with msvc -- mostly due to using different C runtimes, but with a few other corner case issues too. This, however, is fixable. The PSF- and NumFOCUS-sponsored mingwpy project is near to having builds of mingw-w64 that are abi compatible with all cpython releases from 2.6-3.4: http://mingwpy.github.io/ The bigger challenge is getting a version of mingw-w64 that is compatible with the latest MSVC used for cpython 3.5+, since they rearranged the runtimes (for the better, but...). Unfortunately this is also the case that's relevant for OP's use case of working on cpython itself. There's no technical obstacle to doing this, and it's a major priority for both the numerical python community and mingw-w64 upstream, but it's not currently clear who will do the work and if/how it will be funded. In any case though I agree that now is not a good time to start trying to support two incompatible windows ABIs in python upstream, just we're converging on a common abi across the different compilers. -n
On Fri, Feb 26, 2016 at 6:55 PM, Alexander Walters <tritium-list@sdamon.com> wrote:
My personal experience with psutil is that at some point I simply gave up trying to support mingw because it was too difficult and I couldn't keep up with it anymore. I had to hack through all sorts of missing stuff such as: https://github.com/giampaolo/psutil/blob/08d490d0d8fa60ee1d689cca30738ceb599..., I always wondered why mingw doesn't do it for me, plus the fact that I never really got it to work with 64-bit Python. I can only imagine what it would mean doing something similar in a much larger C code base such as Python's. The advantage of being able to use something else other than VS (which I despise as well) would undoubtedly be enormous, but from my experience mingw is an alternative which simply doesn't work. -- Giampaolo - http://grodola.blogspot.com
Hi Mathieu I just want to say that we are very aware of the concerns and issues faced here and people (including myself) are actively working to resolve them. For example, I am working with the Visual C++ team to encourage and support work such as https://blogs.msdn.microsoft.com/vcblog/2016/02/16/try-out-the-latest-c-comp..., which strips out most of that 9GB install. It currently is not sufficient for building Python or extension modules, and is likely to be 500MB+ by the time it is, but I am directly in contact with the team involved to achieve that. Once this is available, I will be looking into ways to make it easier to install the compiler. (I work at Microsoft, so I have a better ability than most to influence this sort of thing.) In parallel, there is plenty going on over on distutils-sig to improve the package ecosystem in this respect (which I know is not your immediate concern, but it always comes up as soon as someone mentions a compiler). Improved sdist and wheel capabilities are critical to being able to help non-Windows developers produce pre-compiled software for their users. Supporting a second compiler for CPython is a fairly significant task. We are willing to do it when someone volunteers to actively maintain it and manage a buildbot for it, but so far the best offer has been patches (not all of which are ready to be committed, though I haven't gone through them recently so that may have been improved). Releasing builds from a second compiler is a big compatibility concern, as it is highly likely (right now) that existing wheels will simply crash when used. Dealing with that - probably by treating wheels as incompatible - is essential before we want other distros thinking they can release a MinGW build. There's also plenty of distutils work required to support it properly, and I'm not sure how much of that should/could be done anyway. So basically, smaller packages for MSVC are on their way, and MinGW support is blocked until someone commits to supporting it for an extended period of time. Hopefully that helps clarify our position. Cheers, Steve On 26Feb2016 0212, Mathieu Dupuy wrote:
One alternative to consider is using Cygwin. A complete Cygwin environment, including a GCC toolchain, is pretty small. And it can build a *nix-style CPython that works inside the Cygwin environment. That may not be sufficient for a lot of uses, but for your purpose, it should be. Another alternative, as crazy as it may sound, is to get an AWS-Free-Tier EC2 instance and develop on that. Or, of course, buy an ancient used laptop and install linux natively. Obviously none of these are ideal, but they may still be better for you than waiting for a complete MinGW port of Python or a smaller MSVC toolchain.
The 9 gig initial download is not the only problem. Visual studio is very bandwidth hungry in day to day operations (between polling websites and vcs remotes, near constant updating, integration with the VS web service, etc.). You can of course shut all of that off, but it's a pain. It's my understanding from Steve's post that a leaner, meaner edition of VS is in the works, so waiting for that might just be an overall better solution. On 2/27/2016 16:27, Franklin? Lee wrote:
participants (10)
-
Alexander Walters -
Andrew Barnert -
Andy Ray Terrel -
Dan Stromberg -
Franklin? Lee -
Giampaolo Rodola' -
Mathieu Dupuy -
Nathaniel Smith -
R. David Murray -
Steve Dower