Re: [Python-Dev] Status of C compilers for Python on Windows
Stephen J. Turnbull:
Python is open source. Nobody is objecting to "somebody else" doing this.[1] The problem here is simply that some "somebody elses" are trying to throw future work over the wall into python-dev space.
If that's how it's seen at this point, then it sounds like the logical course of action for CPython-with-MinGW is to demonstrate feasibility in a fork. Get what currently works as a set of 80-something patches and PKGBUILD instructions into a single repository that is usable by a wider variety of people, in more distributions, etc. Set up as much CI as possible so every patch gets tested in as many configurations as we can. Experiment with extension compatibility and find out what is actually achievable, with or without needing to modify MinGW-w64 in the process. There are people, though evidently not much of python-dev, who have a need and desire to make this happen. It seems python-dev would rather have us spend zero time proposing changes that allow CPython itself to be built differently than today, and all of our time on MinGW extensions. I personally find 3 of the 4 combinations of how one could build CPython and how one could build extensions interesting and worth looking into for different reasons (the one that's uninteresting to me is the currently used all-MSVC method, due to its many limitations I listed earlier). Ray for example may only care about using MinGW for everything. Python's a big community with lots of room for different people to work on different aspects of the same set of problems. For the combination of MSVC Python and MinGW extensions that most of you have recommended we focus on, it would be more productive to engage with setuptools, distutils-sig, and likely numpy as well, instead of python-dev. My experience lies more in getting troublesome C codebases to build with MinGW than it does with the messy intricacies and backwards-compatibility requirements of Python extensions and package management however, so my ability to contribute on that side of things is more limited. I'll just note that every project I've ever had a patch for which improved functionality with a new compiler (whether GCC, MSVC, clang, icc or ifort, etc.) reacted with review and thanks for the patches, not "why do you want to do this?" pushback. If potential contributors have a desire to get it working in the first place, then they will also be invested in helping keep it working on an ongoing basis. Sincerely, Tony
Tony Kelman writes:
If potential contributors have a desire to get it working in the first place, then they will also be invested in helping keep it working on an ongoing basis.
Sure -- as long as it works for them, though, such potential contributors don't necessarily care if it works for anybody else. My experience (in other projects) is that allowing that level of commitment to be sufficient for inclusion in the maintained code base frequently results in bug reports from third parties who try to use the new feature and discover it *doesn't* work for them. The core maintainers then have a very unpleasant choice: to say "we don't support that usage", or to deal with the problem on a continuing basis (because the same issues tend to regress repeatedly as the said "invested contributors" continue to modify their code on the same "works for us" basis).
If that's how it's seen at this point, then it sounds like the logical course of action for CPython-with-MinGW is to demonstrate feasibility in a fork. Get what currently works as a set of 80-something patches and PKGBUILD instructions into a single repository that is usable by a wider variety of people, in more distributions, etc. Set up as much CI as possible so every patch gets tested in as many configurations as we can. Experiment with extension compatibility and find out what is actually achievable, with or without needing to modify MinGW-w64 in the process.
Sounds good to me. You seem to think that's excessive, though:
There are people, though evidently not much of python-dev, who have a need and desire to make this happen.
Well, for starters, most of python-dev would rather avoid any contact whatsoever with Windows. I think part of the problem is that Windows developers *of* Python are *very* rare (fingers of one hand rare). Sure, there are many Windows-based Python developers, and quite of few of them do a fair amount to contribute to Python on Windows -- but they do that work in *Python*, not at a level where they deal with the extension ABI. Even those who do work on C extensions have so far been happy to work with the VC build (except for the recurrent issue of getting one's hands on the toolchain). So why should python-dev have a desire to do that work? It should be evident by now that our belief is that the large majority of Windows users is well-served by the current model (produce an official binary and a plethora of compatible extensions, which provides strong incentive to third parties to ensure that their extensions and alternative builds of core are also compatible). I think the repeated query, "why isn't this model good enough for you?" is being misinterpreted. I suppose that some who ask that really want to know, because if you have what they consider a good reason, they'd be willing to help. Others are asking but by "you" they mean the world at large, in particular, "what benefit is there to the large number of users well-served by the current model?"
It seems python-dev would rather have us spend zero time proposing changes that allow CPython itself to be built differently than today, and all of our time on MinGW extensions.
Of course they would. (Third person because I'm not competent to do the work anyway.) It's quite clear that one thing the two sides agree on is that ensuring that MinGW and VC interpreter and extension builds can mix and match is quite a bit of work. They quite naturally don't want to do that work, and don't see much point in discussing it if the (apparently) few people who need it aren't going to supply the resources. That's quite a reasonable solution, really: *both* communities avoid spending effort on mix and match, and because it's a fork, it's a different name, and people won't expect them to mix and match.
I personally find 3 of the 4 combinations of how one could build CPython and how one could build extensions interesting and worth looking into for different reasons (the one that's uninteresting to me is the currently used all-MSVC method, due to its many limitations I listed earlier). Ray for example may only care about using MinGW for everything. Python's a big community with lots of room for different people to work on different aspects of the same set of problems.
There's a reason we call it "core". One of python-dev's more important responsibilities is to ensure that the "aspects" work and play well together. "Aspects" people tend to deprecate that responsibility (until somebody else's "aspect" treads on their blue suede shoes). That's as it should be, IMO -- but so is python-dev's reluctance to admit new "aspects" until their impact on core responsibilities is made clear.
Stephen J. Turnbull:
Sure -- as long as it works for them, though, such potential contributors don't necessarily care if it works for anybody else. My experience (in other projects) is that allowing that level of commitment to be sufficient for inclusion in the maintained code base frequently results in bug reports from third parties who try to use the new feature and discover it *doesn't* work for them.
Good point. I definitely care whether patches work for everyone else. Patches should be done well and accompanied with proper documentation so new functionality is fully reproducible. If that's what's holding up review, comments in the bug threads indicating as much would be helpful. Any fork will also have to be thoroughly documented if it's to have any chance of working.
Sounds good to me. You seem to think that's excessive, though:
No, just hearing the words come out of my mouth they sound a little nuts. Maybe not, there are after all half a dozen or more incompatible alternate Python implementations floating around. I think most of them started as from-scratch rewrites rather than source forks, but maybe that's irrelevant.
Well, for starters, most of python-dev would rather avoid any contact whatsoever with Windows. I think part of the problem is that Windows developers *of* Python are *very* rare (fingers of one hand rare).
In my opinion the MSVC toolchain makes that problem worse, as it's far harder for unix developers to have any familiarity with how things work. But you do have involvement and core developers from Microsoft which is obviously incredibly important. Maybe even mandatory for Python on Windows to be viable in your eyes.
Even those who do work on C extensions have so far been happy to work with the VC build (except for the recurrent issue of getting one's hands on the toolchain).
It should be evident by now that our belief is that the large majority of Windows users is well-served by the current model
This is not the case at all in the scientific community. NumPy and SciPy put in a lot of extra work to come up with something that is compatible with the MSVC build of CPython because they have to, not because they're "happy to" jump through the necessary hoops. The situation today with NumPy AIUI is they already have to build with a custom hacked MinGW toolchain that only one person knows how to use. Evidently until very recently they were using a many-years-old 32-bit-only build of GCC 3.x. Do python-dev and numpy-discussion not talk to one another? I get that not everyone uses numpy, or Windows, but it sounds like there's very little understanding, or even awareness, of the issues they face.
They quite naturally don't want to do that work, and don't see much point in discussing it if the (apparently) few people who need it aren't going to supply the resources.
I'm going to move the "extensions with MinGW-w64" part of this conversation over to numpy-discussion, since they have a need for this today and are already using workarounds and processes that I don't think anyone is fully satisfied with. I do find this combination interesting, worth working on, and possible to make work well, but not completely in isolation as it does not address my embedding use case.
but so is python-dev's reluctance to admit new "aspects" until their impact on core responsibilities is made clear.
Okay. I'll table the discussion with python-dev for now then. -Tony
On 28 October 2014 14:46, Tony Kelman <kelman@berkeley.edu> wrote:
Patches should be done well and accompanied with proper documentation so new functionality is fully reproducible. If that's what's holding up review, comments in the bug threads indicating as much would be helpful.
Typically that tends to be expressed as a terse "I can't get this working". That's not ideal, but is an indication. When the response is "but it's easy" (as it sometimes is) communication degenerates quite fast. There's an example here in this thread - it took me a *long* time, with help from a couple of people, to work out how to get a mingw toolchain I could use to try things out. Even though the premise of the whole discussion was "it's easy to set up a mingw toolchain"...
In my opinion the MSVC toolchain makes that problem worse, as it's far harder for unix developers to have any familiarity with how things work. But you do have involvement and core developers from Microsoft which is obviously incredibly important. Maybe even mandatory for Python on Windows to be viable in your eyes.
One problem I've seen a lot on other projects is that when Unix developers set up a comfortable, Unix-like environment on Windows using something like mingw, they frequently aren't aware of crucial differences between Unix and Windows and tend to write software that even though it's Windows-native, *feels* Unixy to Windows users (don't ask me to be more specific :-)). That's always going to happen, and the people with Windows experience have to take up the slack by keeping an eye out for such things, but setting the bar marginally higher, to "you have to at least be willing to download and use a native Windows compiler" can at least remind said Unix developers that they are in a different environment. That's *not* a criticism of anyone in the Python community, btw. Typically the experience of Windows users is very well respected by python core and package developers. But I tend to think that's partly *because* we didn't take the "Unix-like toolchain" approach by default.
This is not the case at all in the scientific community. NumPy and SciPy put in a lot of extra work to come up with something that is compatible with the MSVC build of CPython because they have to, not because they're "happy to" jump through the necessary hoops. The situation today with NumPy AIUI is they already have to build with a custom hacked MinGW toolchain that only one person knows how to use. Evidently until very recently they were using a many-years-old 32-bit-only build of GCC 3.x. Do python-dev and numpy-discussion not talk to one another? I get that not everyone uses numpy, or Windows, but it sounds like there's very little understanding, or even awareness, of the issues they face.
Sadly, no. The numpy developers have always been a pretty much separate world. We're seeing a bit more interaction these days, but it's very limited and far from the level that's needed. The fault (if that's the right word) probably lies on both sides. It's certainly not purely the responsibility of the core team to build communication links. As this thread has demonstrated, python-dev (and distutils-sig, which is another place that desparately needs more numpy interaction) is not the most welcoming of places for someone who is 100% focused on the scientific stack, but conversely the scientific types do tend to do their own thing, and sometimes when they do surface, they can dive in with little appreciation of the wider picture. But we can get along, and we can make progress (albeit not always as fast as either party might like). If all this thread has done is raise awareness of each others' concerns, it's still been a win IMO.
I'm going to move the "extensions with MinGW-w64" part of this conversation over to numpy-discussion, since they have a need for this today and are already using workarounds and processes that I don't think anyone is fully satisfied with. I do find this combination interesting, worth working on, and possible to make work well, but not completely in isolation as it does not address my embedding use case.
Please keep distutils-sig in the loop as well. I can't promise we'll be able to help, but we should at least make sure we're not hindering you, and we can make you aware of when your solutions won't work outside your area of interest. Now the door is open, let's not close it again. Paul
Tony Kelman writes:
No, just hearing the words come out of my mouth they sound a little nuts. Maybe not, there are after all half a dozen or more incompatible alternate Python implementations floating around. I think most of them started as from-scratch rewrites rather than source forks, but maybe that's irrelevant.
Well, they have different names and they clearly are not intended to be ABI compatible, so noone expects that. OTOH, there clearly is an expectation among many (and not just in the Windows world, cf. all of the distros that provide whole stacks of everything for each version of Python) that downloaded packages will just work without incompatibility.
Well, for starters, most of python-dev would rather avoid any contact whatsoever with Windows. I think part of the problem is that Windows developers *of* Python are *very* rare (fingers of one hand rare).
In my opinion the MSVC toolchain makes that problem worse, as it's far harder for unix developers to have any familiarity with how things work.
I've used Cygwin, I've used MinGW, and I've used VC. Sure, the former two are GCC-based so I have a lot of muscle memory for command-line switches. But that's not very important; the pain of using Windows is what drives me away from all of them.
But you do have involvement and core developers from Microsoft which is obviously incredibly important. Maybe even mandatory for Python on Windows to be viable in your eyes.
No, I don't think that's true. What I think *is* true is that most developers on Windows do have access to Microsoft tools, so we do need to provide compatibility with them. As you say, the VC toolchain is not all things to all men, but what's visible to python-dev makes it more important than Cygwin or MinGW. See Paul Moore's post about communications between the scientific Python community and python-dev for what I mean by "visible".
It should be evident by now that our belief is that the large majority of Windows users is well-served by the current model
This is not the case at all in the scientific community. NumPy and SciPy put in a lot of extra work to come up with something that is compatible with the MSVC build of CPython because they have to, not because they're "happy to" jump through the necessary hoops.
Agreed. This is well-known to python-dev, and AFAICS it *is* a concern for us. However, as Paul points out, a bridge needs to be built. Your posts have been a contribution to building that bridge, for sure, but more work on the bridge is needed.
Do python-dev and numpy-discussion not talk to one another?
Exactly the issue here. To resolve this, we need to talk more. Unfortunately, I'm not one to help build the bridge as I haven't developed on Windows at all since about 2003.
I'm going to move the "extensions with MinGW-w64" part of this conversation over to numpy-discussion,
As far as I can tell, that's a good idea right now. They have the need, they have the expertise, both of which are somewhat lacking here.
Okay. I'll table the discussion with python-dev for now then.
I hope you'll be able to come pick it back up at some point. You might want to interact with Steve Dower off-list, as he's spearheading the effort to move the official builds to the "stable ABI" version of MSVC. Once that's in place, the MinGW guys will have a stationary target which is up to date to shoot at.
"You might want to interact with Steve Dower off-list" FWIW, I'm happy to talk specifics off list, and have already been involved in a number of discussions with the numpy and Scipy guys wrt figuring out specific technical challenges or clarifying non obvious parts of dealing with Windows. (As far as coding goes, practically all my spare time is taken up already, which is why I haven't contributed directly to those projects, much as I'd like to.) Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Stephen J. Turnbull<mailto:stephen@xemacs.org> Sent: 10/28/2014 20:59 To: Tony Kelman<mailto:kelman@berkeley.edu> Cc: python-dev@python.org<mailto:python-dev@python.org> Subject: Re: [Python-Dev] Status of C compilers for Python on Windows Tony Kelman writes:
No, just hearing the words come out of my mouth they sound a little nuts. Maybe not, there are after all half a dozen or more incompatible alternate Python implementations floating around. I think most of them started as from-scratch rewrites rather than source forks, but maybe that's irrelevant.
Well, they have different names and they clearly are not intended to be ABI compatible, so noone expects that. OTOH, there clearly is an expectation among many (and not just in the Windows world, cf. all of the distros that provide whole stacks of everything for each version of Python) that downloaded packages will just work without incompatibility.
Well, for starters, most of python-dev would rather avoid any contact whatsoever with Windows. I think part of the problem is that Windows developers *of* Python are *very* rare (fingers of one hand rare).
In my opinion the MSVC toolchain makes that problem worse, as it's far harder for unix developers to have any familiarity with how things work.
I've used Cygwin, I've used MinGW, and I've used VC. Sure, the former two are GCC-based so I have a lot of muscle memory for command-line switches. But that's not very important; the pain of using Windows is what drives me away from all of them.
But you do have involvement and core developers from Microsoft which is obviously incredibly important. Maybe even mandatory for Python on Windows to be viable in your eyes.
No, I don't think that's true. What I think *is* true is that most developers on Windows do have access to Microsoft tools, so we do need to provide compatibility with them. As you say, the VC toolchain is not all things to all men, but what's visible to python-dev makes it more important than Cygwin or MinGW. See Paul Moore's post about communications between the scientific Python community and python-dev for what I mean by "visible".
It should be evident by now that our belief is that the large majority of Windows users is well-served by the current model
This is not the case at all in the scientific community. NumPy and SciPy put in a lot of extra work to come up with something that is compatible with the MSVC build of CPython because they have to, not because they're "happy to" jump through the necessary hoops.
Agreed. This is well-known to python-dev, and AFAICS it *is* a concern for us. However, as Paul points out, a bridge needs to be built. Your posts have been a contribution to building that bridge, for sure, but more work on the bridge is needed.
Do python-dev and numpy-discussion not talk to one another?
Exactly the issue here. To resolve this, we need to talk more. Unfortunately, I'm not one to help build the bridge as I haven't developed on Windows at all since about 2003.
I'm going to move the "extensions with MinGW-w64" part of this conversation over to numpy-discussion,
As far as I can tell, that's a good idea right now. They have the need, they have the expertise, both of which are somewhat lacking here.
Okay. I'll table the discussion with python-dev for now then.
I hope you'll be able to come pick it back up at some point. You might want to interact with Steve Dower off-list, as he's spearheading the effort to move the official builds to the "stable ABI" version of MSVC. Once that's in place, the MinGW guys will have a stationary target which is up to date to shoot at. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/28/2014 11:59 PM, Stephen J. Turnbull wrote:
most developers on Windows do have access to Microsoft tool
I assume you mean python-dev folks who work on Windows: it is certainly not true for the vast majority of develoeprs who use Python on Windows, who don't have the toolchain build their own C extensions. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAlRQ+BYACgkQ+gerLs4ltQ7L8QCeJ4NYs+//39O0dUUNqG1lXy1Z 7GMAniUCjmodCfKAVBF/yeOv3GrR03Fm =n4bm -----END PGP SIGNATURE-----
On Wed, 29 Oct 2014 10:22:14 -0400, Tres Seaver <tseaver@palladion.com> wrote:
On 10/28/2014 11:59 PM, Stephen J. Turnbull wrote:
most developers on Windows do have access to Microsoft tool
I assume you mean python-dev folks who work on Windows: it is certainly not true for the vast majority of develoeprs who use Python on Windows, who don't have the toolchain build their own C extensions.
I'm pretty sure he meant "most people who develop software for Windows", even though that's not how he phrased it. But this does not include, as you point out, people who develop Python software that *also* works on Windows. If you are writing code targeted for Windows, I think you are very likely to have an MSDN subscription of some sort if your package includes C code. I'm sure it's not 100%, though. --David
On Wed, 29 Oct 2014 10:31:50 -0400 "R. David Murray" <rdmurray@bitdance.com> wrote:
On Wed, 29 Oct 2014 10:22:14 -0400, Tres Seaver <tseaver@palladion.com> wrote:
On 10/28/2014 11:59 PM, Stephen J. Turnbull wrote:
most developers on Windows do have access to Microsoft tool
I assume you mean python-dev folks who work on Windows: it is certainly not true for the vast majority of develoeprs who use Python on Windows, who don't have the toolchain build their own C extensions.
I'm pretty sure he meant "most people who develop software for Windows", even though that's not how he phrased it. But this does not include, as you point out, people who develop Python software that *also* works on Windows.
If you are writing code targeted for Windows, I think you are very likely to have an MSDN subscription of some sort if your package includes C code. I'm sure it's not 100%, though.
You can use Express editions of Visual Studio. Regards Antoine.
On 30 October 2014 00:46, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Wed, 29 Oct 2014 10:31:50 -0400 "R. David Murray" <rdmurray@bitdance.com> wrote:
On Wed, 29 Oct 2014 10:22:14 -0400, Tres Seaver <tseaver@palladion.com> wrote:
On 10/28/2014 11:59 PM, Stephen J. Turnbull wrote:
most developers on Windows do have access to Microsoft tool
I assume you mean python-dev folks who work on Windows: it is certainly not true for the vast majority of develoeprs who use Python on Windows, who don't have the toolchain build their own C extensions.
I'm pretty sure he meant "most people who develop software for Windows", even though that's not how he phrased it. But this does not include, as you point out, people who develop Python software that *also* works on Windows.
If you are writing code targeted for Windows, I think you are very likely to have an MSDN subscription of some sort if your package includes C code. I'm sure it's not 100%, though.
You can use Express editions of Visual Studio.
And the PSF can help out if folks working on Python software for Windows need an MSDN subscription. The objections general aren't about the availability (or lack thereof) of Windows build tools (especially now the Python 2.7 compiler availability issue has been addressed via http://www.microsoft.com/en-us/download/details.aspx?id=44266), it's more about: * wanting to build for Windows within the scope of an existing POSIX based build automation system * folks that prefer to use only open source software themselves wanting to make their projects available to Windows users "Python as first language" is still a relatively novel phenomenon, and the existing distribution infrastructure certainly wasn't originally designed for that use case. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On 29 Oct 2014 14:47, "Antoine Pitrou" <solipsis@pitrou.net> wrote:
On Wed, 29 Oct 2014 10:31:50 -0400 "R. David Murray" <rdmurray@bitdance.com> wrote:
On Wed, 29 Oct 2014 10:22:14 -0400, Tres Seaver <tseaver@palladion.com>
wrote:
On 10/28/2014 11:59 PM, Stephen J. Turnbull wrote:
most developers on Windows do have access to Microsoft tool
I assume you mean python-dev folks who work on Windows: it is certainly not true for the vast majority of develoeprs who use Python on Windows, who don't have the toolchain build their own C extensions.
I'm pretty sure he meant "most people who develop software for Windows", even though that's not how he phrased it. But this does not include, as you point out, people who develop Python software that *also* works on Windows.
If you are writing code targeted for Windows, I think you are very likely to have an MSDN subscription of some sort if your package includes C code. I'm sure it's not 100%, though.
You can use Express editions of Visual Studio.
IIUC, the express edition compilers are 32-bit only, and what you actually want are the "SDK compilers": https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows These are freely downloadable by anyone, no msdn subscription required, but only if you know where to find them! AFAICT the main obstacle to using MSVC to build python extensions (assuming it can handle your code at all) is not anything technical, but rather that there's no clear and correct tutorial on how to do it, and lots of confusion and misinformation circulating. -n
On 29 October 2014 15:31, Nathaniel Smith <njs@pobox.com> wrote:
You can use Express editions of Visual Studio.
IIUC, the express edition compilers are 32-bit only, and what you actually want are the "SDK compilers": https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
These are freely downloadable by anyone, no msdn subscription required, but only if you know where to find them!
AFAICT the main obstacle to using MSVC to build python extensions (assuming it can handle your code at all) is not anything technical, but rather that there's no clear and correct tutorial on how to do it, and lots of confusion and misinformation circulating.
Would it help if I wrote a document explaining how to set up the MS compilers (free and paid for) to allow building of Python extensions? There are a few provisos: 1. A lot of it will be pretty trivial: "If you have Vistal Studio (full edition), install it. Done." 2. It will be out of date very fast as the situation for Python 3.5+ will be trivial across the board. 3. I don't have anywhere particularly authoritative to host it (maybe the Python Wiki?) and it could easily get lost in the huge swamp of variously outdated, over-complicated, or otherwise alternative documents available. Ideally I'd like someone to suggest an "official" location I could use. I don't want to do this if it won't be useful, as it'll take me a bit of effort to confirm the process for the only non-trivial scenario (64-bit Python 3.3/3.4 with free tools). But if people think it would help, that's fine, I volunteer. Paul PS Even if I don't get positive feedback, I may just say "to heck with it" and do it anyway, because it *is* so trivial :-) I just won't promise.
This sounds like something good for packaging.python.org
On Oct 29, 2014, at 4:05 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 29 October 2014 15:31, Nathaniel Smith <njs@pobox.com> wrote:
You can use Express editions of Visual Studio.
IIUC, the express edition compilers are 32-bit only, and what you actually want are the "SDK compilers": https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
These are freely downloadable by anyone, no msdn subscription required, but only if you know where to find them!
AFAICT the main obstacle to using MSVC to build python extensions (assuming it can handle your code at all) is not anything technical, but rather that there's no clear and correct tutorial on how to do it, and lots of confusion and misinformation circulating.
Would it help if I wrote a document explaining how to set up the MS compilers (free and paid for) to allow building of Python extensions?
There are a few provisos:
1. A lot of it will be pretty trivial: "If you have Vistal Studio (full edition), install it. Done." 2. It will be out of date very fast as the situation for Python 3.5+ will be trivial across the board. 3. I don't have anywhere particularly authoritative to host it (maybe the Python Wiki?) and it could easily get lost in the huge swamp of variously outdated, over-complicated, or otherwise alternative documents available. Ideally I'd like someone to suggest an "official" location I could use.
I don't want to do this if it won't be useful, as it'll take me a bit of effort to confirm the process for the only non-trivial scenario (64-bit Python 3.3/3.4 with free tools). But if people think it would help, that's fine, I volunteer.
Paul
PS Even if I don't get positive feedback, I may just say "to heck with it" and do it anyway, because it *is* so trivial :-) I just won't promise. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io
On 29 October 2014 20:26, Donald Stufft <donald@stufft.io> wrote:
This sounds like something good for packaging.python.org
Yeah, I wondered about that. I'll work up a patch for that. But the more I think about it, it really is trivial: - For non-free MSVC, install the appropriate version, and everything just works. - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7 package and everything just works as long as you're using setuptools. - For 32 bit Python 3.2-3.4, install Visual Studio Express and everything just works. - For 64 bit Python 3.2-3.4, install the SDK, set some environment variables, and everything just works. - For Python 3.5+, install the current Visual Studion Express and everything just works. (I propose to deem Python 2.7 without setuptools as "unsupported") The only things I can see that need expansion are: 1. The precise versions to use (trivial to add, I'm just to lazy to check right now). 2. Where to get VS 2010 Express (as it's no longer supported or available from MS). 3. How to set up the SDK environment for 64-bit Python 3.2-3.4. Before I do a write-up, I want to set up clean VMs with these configurations, so that I can confirm the details. But I don't expect any problems, as I've done them all before. Paul.
On Oct 29, 2014, at 6:09 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 29 October 2014 20:26, Donald Stufft <donald@stufft.io> wrote:
This sounds like something good for packaging.python.org
Yeah, I wondered about that. I'll work up a patch for that. But the more I think about it, it really is trivial:
- For non-free MSVC, install the appropriate version, and everything just works. - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7 package and everything just works as long as you're using setuptools. - For 32 bit Python 3.2-3.4, install Visual Studio Express and everything just works. - For 64 bit Python 3.2-3.4, install the SDK, set some environment variables, and everything just works. - For Python 3.5+, install the current Visual Studion Express and everything just works.
(I propose to deem Python 2.7 without setuptools as "unsupported")
The only things I can see that need expansion are:
1. The precise versions to use (trivial to add, I'm just to lazy to check right now). 2. Where to get VS 2010 Express (as it's no longer supported or available from MS). 3. How to set up the SDK environment for 64-bit Python 3.2-3.4.
Before I do a write-up, I want to set up clean VMs with these configurations, so that I can confirm the details. But I don't expect any problems, as I've done them all before.
Paul.
I think it’d be good even if considered trivial. I know I certainly have no idea which pieces I needed to do that. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 10/29/2014 03:09 PM, Paul Moore wrote:
On 29 October 2014 20:26, Donald Stufft <donald@stufft.io> wrote:
This sounds like something good for packaging.python.org
Yeah, I wondered about that. I'll work up a patch for that. But the more I think about it, it really is trivial:
I am reminded of an interview question I was once asked which was prefaced with: "Here's an easy one..." My reply was, if you know the answer, it's easy!
- For non-free MSVC, install the appropriate version, and everything just works. - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7 package and everything just works as long as you're using setuptools. - For 32 bit Python 3.2-3.4, install Visual Studio Express and everything just works. - For 64 bit Python 3.2-3.4, install the SDK, set some environment variables, and everything just works. - For Python 3.5+, install the current Visual Studion Express and everything just works.
I would suggest - where to get these packages - where to get any dependencies - any options to [not] specify during install - what environment variables to what value - where one should be at when one starts the compile process and, of course, a gotchas section for uncommon but frustrating things that might go wrong. And thanks for doing this! -- ~Ethan~
On 29 October 2014 22:19, Ethan Furman <ethan@stoneleaf.us> wrote:
Yeah, I wondered about that. I'll work up a patch for that. But the more I think about it, it really is trivial:
I am reminded of an interview question I was once asked which was prefaced with: "Here's an easy one..."
My reply was, if you know the answer, it's easy!
Yeah, I know what you mean. My take on this is that I agree it's not easy if you don't know and can't get access to the information, but if you can, there's very little to it.
- For non-free MSVC, install the appropriate version, and everything just works. - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7 package and everything just works as long as you're using setuptools. - For 32 bit Python 3.2-3.4, install Visual Studio Express and everything just works. - For 64 bit Python 3.2-3.4, install the SDK, set some environment variables, and everything just works. - For Python 3.5+, install the current Visual Studion Express and everything just works.
I would suggest - where to get these packages
Conceded. Working out how to point people at stuff on MSDN is a challenge, things seem to move around. Maybe Steve Dower could help here with canonical URLs for some of them (IIRC, he provided one for the VS compilers for Python 2.7 package when it was announced). For the paid versions, I'm going to assume that anyone who paid for a compiler and doesn't know where their copy is, probably can't be helped ;-) And of course there's the awkward problem that VS 2010 Express is unsupported and therefore no longer available from *any* official location. I can't solve that, sadly.
- where to get any dependencies
There are none. I could state that explicitly, I guess.
- any options to [not] specify during install
I'll have to go through the installs again just to be sure, but I'm pretty certain "Select the default for everything" is correct.
- what environment variables to what value
None, except for the SDK which I did say I needed to test out and cover in more detail.
- where one should be at when one starts the compile process
I don't understand this. It's just "pip wheel foo" to build a wheel for foo (which will be downloaded), or "pip wheel ." or "python setup.py bdist_wheel" as you prefer for a local package. That's standard distutils/setuptools/pip extension building. I don't propose to cover that, just how to *set up* the environment. With the sole exception of the SDK case, once installed, everything just works everywhere, nothing to set up, no special environment to configure. Start up a new cmd or powershell console (or use the one you're already in) and go. Maybe Unix users expect more complexity because it's not that simple on Unix? But I thought it was - install the appropriate OS packages and that's it?
and, of course, a gotchas section for uncommon but frustrating things that might go wrong.
Hmm, I see your point here, but I'm not sure what I might include. You *can* get in a mess [1] but generally I don't as I'm an experienced Windows user. I also don't want to offer to debug and fix everyone's problems in getting things set up, so offering to collect and document "common issues" that I've seen is impractical. Maybe a section entitled "Common Issues and Their Solutions", with some placeholder text saying "If you have any issues installing any of the compiler packages mentioned, please document what went wrong and how to fix it, and submit a PR!" would do?
And thanks for doing this!
No problem! Paul [1] I once spent a *long* time fighting failed installs of the Windows SDK. Turns out it was because I was trying to install a 32-bit SDK on a 64-bit machine (doh!), and the installer really doesn't like that :-( About all I could say to document that is "Read the instructions properly" and "I'm sorry that the MS installers fail really badly when faced with relatively obvious idiot-user errors, but I can't do anything about it :-("
On 10/29/2014 03:46 PM, Paul Moore wrote:
On 29 October 2014 22:19, Ethan Furman <ethan@stoneleaf.us> wrote:
- where one should be at when one starts the compile process
I don't understand this. It's just "pip wheel foo" to build a wheel for foo (which will be downloaded), or "pip wheel ." or "python setup.py bdist_wheel" as you prefer for a local package.
Hmmm... That looks like it's for installing/compiling somebody else's package. Is that last command sufficient to prepare one's own wheel for uploading to PyPI, or there something else to do? -- ~Ethan~
On Oct 29, 2014, at 7:02 PM, Ethan Furman <ethan@stoneleaf.us> wrote:
On 10/29/2014 03:46 PM, Paul Moore wrote:
On 29 October 2014 22:19, Ethan Furman <ethan@stoneleaf.us> wrote:
- where one should be at when one starts the compile process
I don't understand this. It's just "pip wheel foo" to build a wheel for foo (which will be downloaded), or "pip wheel ." or "python setup.py bdist_wheel" as you prefer for a local package.
Hmmm... That looks like it's for installing/compiling somebody else's package. Is that last command sufficient to prepare one's own wheel for uploading to PyPI, or there something else to do?
Generally for uploading to PyPI you do ``python setup.py bdist_wheel``, though I don’t think there’d be any bad thing if you used pip wheel. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 29 October 2014 23:02, Ethan Furman <ethan@stoneleaf.us> wrote:
On 10/29/2014 03:46 PM, Paul Moore wrote:
On 29 October 2014 22:19, Ethan Furman <ethan@stoneleaf.us> wrote:
- where one should be at when one starts the compile process
I don't understand this. It's just "pip wheel foo" to build a wheel for foo (which will be downloaded), or "pip wheel ." or "python setup.py bdist_wheel" as you prefer for a local package.
Hmmm... That looks like it's for installing/compiling somebody else's package. Is that last command sufficient to prepare one's own wheel for uploading to PyPI, or there something else to do?
Oh, I see what you're thinking. I explicitly *don't* want to get into general packaging issues - they are covered elsewhere. The point here is that if you have the right compiler set up, you can read any generic packaging instructions and it doesn't matter whether there's C code involved. That's it. But yes, "python setup.py bdist_wheel" is how you build a wheel for upload. That's true whether your project includes C extensions or not. I'm also not expecting to explain to people how to build any dependencies (for example, PyYAML depends on the C compiler being able to find libyaml). That *is* harder on Windows than on Linux, because there's no "system location" equivalent to /usr/local/include and /usr/local/lib. But again, I consider this to be outside the scope of the document, because it's specific to the particular project you're building. Paul
On Wed, Oct 29, 2014 at 10:46 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 29 October 2014 22:19, Ethan Furman <ethan@stoneleaf.us> wrote:
Yeah, I wondered about that. I'll work up a patch for that. But the more I think about it, it really is trivial:
I am reminded of an interview question I was once asked which was prefaced with: "Here's an easy one..."
My reply was, if you know the answer, it's easy!
Yeah, I know what you mean. My take on this is that I agree it's not easy if you don't know and can't get access to the information, but if you can, there's very little to it.
That's great, but yeah. In case it helps as a data point, I consider myself a reasonably technical guy, probably more than your average python package author -- 15 years of free software hacking, designed a pretty popular X remote display protocol, helped invent DVCS, have written patches to gcc and the kernel, numpy co-maintainer, etc. etc. For the last ~12 months I've had an underlined and circled todo item saying "make wheels for https://pypi.python.org/pypi/zs/", and every time I've sat down and spent a few hours trying I've ended up utterly defeated with a headache. Distutils is spaghetti, and Windows is spaghetti (esp. if you've never used it for more than checking email on a friend's laptop), and the toolchain setup is spaghetti, and then suddenly people are talking about vcvarsall.bats and I don't know what all. I honestly would totally benefit from one of those talk-your-grandparents-through-it tutorials ("here's a screenshot of the dialogue box, and I've drawn an arrow pointing at the 'ok' button. You should press the 'ok' button.")
- For non-free MSVC, install the appropriate version, and everything just works. - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7 package and everything just works as long as you're using setuptools. - For 32 bit Python 3.2-3.4, install Visual Studio Express and everything just works. - For 64 bit Python 3.2-3.4, install the SDK, set some environment variables, and everything just works.
I think the SDK covers both 32 and 64 bit? If so then leaving visual studio out of things entirely is probably simpler. I'm also unsure how you even get both 32- and 64-bit versions of python to coexist on the same system. -n -- Nathaniel J. Smith Postdoctoral researcher - Informatics - University of Edinburgh http://vorpus.org
On 29 October 2014 23:22, Nathaniel Smith <njs@pobox.com> wrote:
Yeah, I know what you mean. My take on this is that I agree it's not easy if you don't know and can't get access to the information, but if you can, there's very little to it.
That's great, but yeah. In case it helps as a data point, I consider myself a reasonably technical guy, probably more than your average python package author -- 15 years of free software hacking, designed a pretty popular X remote display protocol, helped invent DVCS, have written patches to gcc and the kernel, numpy co-maintainer, etc. etc.
Yeah, that counts :-)
For the last ~12 months I've had an underlined and circled todo item saying "make wheels for https://pypi.python.org/pypi/zs/", and every time I've sat down and spent a few hours trying I've ended up utterly defeated with a headache. Distutils is spaghetti, and Windows is spaghetti (esp. if you've never used it for more than checking email on a friend's laptop), and the toolchain setup is spaghetti, and then suddenly people are talking about vcvarsall.bats and I don't know what all. I honestly would totally benefit from one of those talk-your-grandparents-through-it tutorials ("here's a screenshot of the dialogue box, and I've drawn an arrow pointing at the 'ok' button. You should press the 'ok' button.")
OK. That needs to be fixed. How about this. I'll work with you to get the wheel build process set up in such a way that you can maintain it, and we'll document what you needed to know in order to do it. Then I can put that documentation into the packaging user guide for other people to work with. It can wait till you have the time, but when you do, shout and I'll help. I know nothing about zs, but I just tried building it. MSVC 2010 doesn't support "static inline" (used twice in zs\pycrc-crc64xz.h) but when I removed the "inline" from that it built fine for me. So it should be a good example, as there are no particular complexities to deal with.
- For non-free MSVC, install the appropriate version, and everything just works. - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7 package and everything just works as long as you're using setuptools. - For 32 bit Python 3.2-3.4, install Visual Studio Express and everything just works. - For 64 bit Python 3.2-3.4, install the SDK, set some environment variables, and everything just works.
I think the SDK covers both 32 and 64 bit? If so then leaving visual studio out of things entirely is probably simpler.
The SDK is the most complex environment to set up, so avoiding it unless you need it is a good thing. But maybe it's worth explaining that if you need it anyway, this is how you set it up to cover some of the other cases as well.
I'm also unsure how you even get both 32- and 64-bit versions of python to coexist on the same system.
Just install them into different directories. The default is the same for 32 and 64 bit, so you need to override the default for (at least) one of them, but that's all. Paul
"For the paid versions, I'm going to assume that anyone who paid for a compiler and doesn't know where their copy is, probably can't be helped ;-)" You could link to visualstudio.com for the trial versions, and maybe to a page/post about the PSF's grants process if such a page exists. "And of course there's the awkward problem that VS 2010 Express is unsupported and therefore no longer available from *any* official location. I can't solve that, sadly" These are still at http://www.visualstudio.com/en-us/downloads/download-visual-studio-vs#Downlo..., which is the main download page. Hopefully they don't go away before 3.5, but I have no control over that unfortunately. The link I posted for 2.7 was aka.ms/vcpython27, which is a redirect that I own and can update if necessary. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Paul Moore<mailto:p.f.moore@gmail.com> Sent: 10/29/2014 15:48 To: Ethan Furman<mailto:ethan@stoneleaf.us> Cc: Python Dev<mailto:python-dev@python.org> Subject: Re: [Python-Dev] Status of C compilers for Python on Windows On 29 October 2014 22:19, Ethan Furman <ethan@stoneleaf.us> wrote:
Yeah, I wondered about that. I'll work up a patch for that. But the more I think about it, it really is trivial:
I am reminded of an interview question I was once asked which was prefaced with: "Here's an easy one..."
My reply was, if you know the answer, it's easy!
Yeah, I know what you mean. My take on this is that I agree it's not easy if you don't know and can't get access to the information, but if you can, there's very little to it.
- For non-free MSVC, install the appropriate version, and everything just works. - For Python 2.7 (32 or 64 bit), install the compiler for Python 2.7 package and everything just works as long as you're using setuptools. - For 32 bit Python 3.2-3.4, install Visual Studio Express and everything just works. - For 64 bit Python 3.2-3.4, install the SDK, set some environment variables, and everything just works. - For Python 3.5+, install the current Visual Studion Express and everything just works.
I would suggest - where to get these packages
Conceded. Working out how to point people at stuff on MSDN is a challenge, things seem to move around. Maybe Steve Dower could help here with canonical URLs for some of them (IIRC, he provided one for the VS compilers for Python 2.7 package when it was announced). For the paid versions, I'm going to assume that anyone who paid for a compiler and doesn't know where their copy is, probably can't be helped ;-) And of course there's the awkward problem that VS 2010 Express is unsupported and therefore no longer available from *any* official location. I can't solve that, sadly.
- where to get any dependencies
There are none. I could state that explicitly, I guess.
- any options to [not] specify during install
I'll have to go through the installs again just to be sure, but I'm pretty certain "Select the default for everything" is correct.
- what environment variables to what value
None, except for the SDK which I did say I needed to test out and cover in more detail.
- where one should be at when one starts the compile process
I don't understand this. It's just "pip wheel foo" to build a wheel for foo (which will be downloaded), or "pip wheel ." or "python setup.py bdist_wheel" as you prefer for a local package. That's standard distutils/setuptools/pip extension building. I don't propose to cover that, just how to *set up* the environment. With the sole exception of the SDK case, once installed, everything just works everywhere, nothing to set up, no special environment to configure. Start up a new cmd or powershell console (or use the one you're already in) and go. Maybe Unix users expect more complexity because it's not that simple on Unix? But I thought it was - install the appropriate OS packages and that's it?
and, of course, a gotchas section for uncommon but frustrating things that might go wrong.
Hmm, I see your point here, but I'm not sure what I might include. You *can* get in a mess [1] but generally I don't as I'm an experienced Windows user. I also don't want to offer to debug and fix everyone's problems in getting things set up, so offering to collect and document "common issues" that I've seen is impractical. Maybe a section entitled "Common Issues and Their Solutions", with some placeholder text saying "If you have any issues installing any of the compiler packages mentioned, please document what went wrong and how to fix it, and submit a PR!" would do?
And thanks for doing this!
No problem! Paul [1] I once spent a *long* time fighting failed installs of the Windows SDK. Turns out it was because I was trying to install a 32-bit SDK on a 64-bit machine (doh!), and the installer really doesn't like that :-( About all I could say to document that is "Read the instructions properly" and "I'm sorry that the MS installers fail really badly when faced with relatively obvious idiot-user errors, but I can't do anything about it :-(" _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...
On 29 October 2014 23:49, Steve Dower <Steve.Dower@microsoft.com> wrote:
"For the paid versions, I'm going to assume that anyone who paid for a compiler and doesn't know where their copy is, probably can't be helped ;-)"
You could link to visualstudio.com for the trial versions, and maybe to a page/post about the PSF's grants process if such a page exists.
That doesn't help for VS 2010 and VS 2008, which are the relevant ones right now, unfortunately. In thew brave new world of Python 3.5+ that might well do, though. (Although the feedback from Unix users is that they really want a direct link to the exact edition they need - i.e., a permalink to VC++ Express Edition <latest> for Python 3.5+. I'm not sure how well the MS website structure does that, though.
"And of course there's the awkward problem that VS 2010 Express is unsupported and therefore no longer available from *any* official location. I can't solve that, sadly"
These are still at http://www.visualstudio.com/en-us/downloads/download-visual-studio-vs#Downlo..., which is the main download page. Hopefully they don't go away before 3.5, but I have no control over that unfortunately.
Wow! I didn't know that. That's great, solves the biggest issue I had.
The link I posted for 2.7 was aka.ms/vcpython27, which is a redirect that I own and can update if necessary.
Cool. As long as it's stable and reliable, it's exactly what I want. Many thanks, Paul
On 10/29/2014 4:05 PM, Paul Moore wrote:
On 29 October 2014 15:31, Nathaniel Smith <njs@pobox.com> wrote:
You can use Express editions of Visual Studio.
IIUC, the express edition compilers are 32-bit only, and what you actually want are the "SDK compilers": https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
These are freely downloadable by anyone, no msdn subscription required, but only if you know where to find them!
AFAICT the main obstacle to using MSVC to build python extensions (assuming it can handle your code at all) is not anything technical, but rather that there's no clear and correct tutorial on how to do it, and lots of confusion and misinformation circulating.
Would it help if I wrote a document explaining how to set up the MS compilers (free and paid for) to allow building of Python extensions?
There are a few provisos:
1. A lot of it will be pretty trivial: "If you have Vistal Studio (full edition), install it. Done." 2. It will be out of date very fast as the situation for Python 3.5+ will be trivial across the board. 3. I don't have anywhere particularly authoritative to host it (maybe the Python Wiki?) and it could easily get lost in the huge swamp of variously outdated, over-complicated, or otherwise alternative documents available. Ideally I'd like someone to suggest an "official" location I could use.
There is already a section in the devguide for building Python itself, with mostly the same info, except it may not be up to date.
I don't want to do this if it won't be useful, as it'll take me a bit of effort to confirm the process for the only non-trivial scenario (64-bit Python 3.3/3.4 with free tools). But if people think it would help, that's fine, I volunteer.
The devguide needs to be kept up to date. If you open a tracker issue, put me as nosy to review and commit. -- Terry Jan Reedy
On Wed, 29 Oct 2014 23:33:06 -0400, Terry Reedy <tjreedy@udel.edu> wrote:
On 10/29/2014 4:05 PM, Paul Moore wrote:
On 29 October 2014 15:31, Nathaniel Smith <njs@pobox.com> wrote:
You can use Express editions of Visual Studio.
IIUC, the express edition compilers are 32-bit only, and what you actually want are the "SDK compilers": https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
These are freely downloadable by anyone, no msdn subscription required, but only if you know where to find them!
AFAICT the main obstacle to using MSVC to build python extensions (assuming it can handle your code at all) is not anything technical, but rather that there's no clear and correct tutorial on how to do it, and lots of confusion and misinformation circulating.
Would it help if I wrote a document explaining how to set up the MS compilers (free and paid for) to allow building of Python extensions?
There are a few provisos:
1. A lot of it will be pretty trivial: "If you have Vistal Studio (full edition), install it. Done." 2. It will be out of date very fast as the situation for Python 3.5+ will be trivial across the board. 3. I don't have anywhere particularly authoritative to host it (maybe the Python Wiki?) and it could easily get lost in the huge swamp of variously outdated, over-complicated, or otherwise alternative documents available. Ideally I'd like someone to suggest an "official" location I could use.
There is already a section in the devguide for building Python itself, with mostly the same info, except it may not be up to date.
I don't want to do this if it won't be useful, as it'll take me a bit of effort to confirm the process for the only non-trivial scenario (64-bit Python 3.3/3.4 with free tools). But if people think it would help, that's fine, I volunteer.
The devguide needs to be kept up to date. If you open a tracker issue, put me as nosy to review and commit.
The devguide is about building python itself. Paul is talking about building extensions. That should go in the packaging docs. I don't *think* Paul is volunteering to explain how to build python itself, that's pretty much our job :) --David
On 10/30/2014 8:59 AM, R. David Murray wrote:
On Wed, 29 Oct 2014 23:33:06 -0400, Terry Reedy <tjreedy@udel.edu> wrote:
The devguide needs to be kept up to date. If you open a tracker issue, put me as nosy to review and commit.
The devguide is about building python itself. Paul is talking about building extensions. That should go in the packaging docs. I don't *think* Paul is volunteering to explain how to build python itself, that's pretty much our job :)
I was thinking that building Python and extensions used the same tools, but then I read that the new compiler for 2.7 was extensions only and ditto for something else. If VC Express 2010 is in a new location, that *should* be recorded in the devguide. -- Terry Jan Reedy
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/29/2014 10:31 AM, R. David Murray wrote:
If you are writing code targeted for Windows, I think you are very likely to have an MSDN subscription of some sort if your package includes C code. I'm sure it's not 100%, though.
My experience with distributing distributions-with-extensions indicates that the vast majority of Windows developers who are "downstream" users for those distributions *cannot* build them from source: if there is no MSI / bdist_win (maybe now wheel), they won't use the project. (Note that "having an MSDN subscription" is not the same as "knowing how to configure which compiler such that it can bulid extensions against an installed Python binary"). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAlRRAskACgkQ+gerLs4ltQ7ILQCfbFCmgZqH+mZa28bQwjNuZruK 6BcAoLG/fxhi4LBkAgZoXNaxq6gi+Pbx =8OvV -----END PGP SIGNATURE-----
On Wed, 29 Oct 2014 11:07:53 -0400 Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 10/29/2014 10:31 AM, R. David Murray wrote:
If you are writing code targeted for Windows, I think you are very likely to have an MSDN subscription of some sort if your package includes C code. I'm sure it's not 100%, though.
My experience with distributing distributions-with-extensions indicates that the vast majority of Windows developers who are "downstream" users for those distributions *cannot* build them from source: if there is no MSI / bdist_win (maybe now wheel), they won't use the project.
(Note that "having an MSDN subscription" is not the same as "knowing how to configure which compiler such that it can bulid extensions against an installed Python binary").
I don't think you have to configure anything. Just install the right Visual Studio version and it's done. Regards Antoine.
Antoine Pitrou wrote:
On Wed, 29 Oct 2014 11:07:53 -0400 Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 10/29/2014 10:31 AM, R. David Murray wrote:
If you are writing code targeted for Windows, I think you are very likely to have an MSDN subscription of some sort if your package includes C code. I'm sure it's not 100%, though.
My experience with distributing distributions-with-extensions indicates that the vast majority of Windows developers who are "downstream" users for those distributions *cannot* build them from source: if there is no MSI / bdist_win (maybe now wheel), they won't use the project.
(Note that "having an MSDN subscription" is not the same as "knowing how to configure which compiler such that it can bulid extensions against an installed Python binary").
I don't think you have to configure anything. Just install the right Visual Studio version and it's done.
The tricky part here is the *right* Visual Studio version. As many have noted, there were bugs/missing components in some of the older Express editions (like the 64-bit compiler integration), and even the compiler pack we released recently doesn't actually help building Python itself, though it's good for extensions. In general, VS 2008 Professional and VS 2010 Professional are easiest to set up if you can get them, the C++ Express editions are fairly easy to set up, and the Windows SDK is difficult but possible (and won't currently build CPython either, as the build depends on VS - I'm also fixing this for 3.5). My ideal target (Utopia refined to be achievable) is for python-dev to handle the Python binaries themselves (already done) and to make them easy to bundle with applications/etc (I'm working on this for 3.5), and for PyPI to include pre-built wheels for Windows where necessary. Note that I don't require package developers to build their own wheels, though I think that they are generally the right people to be doing it. It would be nice to create a culture of delegation, so that "someone" could be a trusted builder for a range of packages (for example, I'd love it if Continuum/Enthought/similar could provide their builds of numpy/scipy as wheels and those projects would be willing to have them be the official PyPI downloads). This is roughly how the python.org installers are handled, and while it may cause some lag between source and binary releases, I think the release cycles are slow enough that most users would only notice that "pip install numpy" now works. Cheers, Steve
Regards
Antoine.
On Oct 29, 2014, at 11:37 AM, Steve Dower <Steve.Dower@microsoft.com> wrote:
Antoine Pitrou wrote:
On Wed, 29 Oct 2014 11:07:53 -0400 Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 10/29/2014 10:31 AM, R. David Murray wrote:
If you are writing code targeted for Windows, I think you are very likely to have an MSDN subscription of some sort if your package includes C code. I'm sure it's not 100%, though.
My experience with distributing distributions-with-extensions indicates that the vast majority of Windows developers who are "downstream" users for those distributions *cannot* build them from source: if there is no MSI / bdist_win (maybe now wheel), they won't use the project.
(Note that "having an MSDN subscription" is not the same as "knowing how to configure which compiler such that it can bulid extensions against an installed Python binary").
I don't think you have to configure anything. Just install the right Visual Studio version and it's done.
The tricky part here is the *right* Visual Studio version. As many have noted, there were bugs/missing components in some of the older Express editions (like the 64-bit compiler integration), and even the compiler pack we released recently doesn't actually help building Python itself, though it's good for extensions. In general, VS 2008 Professional and VS 2010 Professional are easiest to set up if you can get them, the C++ Express editions are fairly easy to set up, and the Windows SDK is difficult but possible (and won't currently build CPython either, as the build depends on VS - I'm also fixing this for 3.5).
My ideal target (Utopia refined to be achievable) is for python-dev to handle the Python binaries themselves (already done) and to make them easy to bundle with applications/etc (I'm working on this for 3.5), and for PyPI to include pre-built wheels for Windows where necessary.
Note that I don't require package developers to build their own wheels, though I think that they are generally the right people to be doing it. It would be nice to create a culture of delegation, so that "someone" could be a trusted builder for a range of packages (for example, I'd love it if Continuum/Enthought/similar could provide their builds of numpy/scipy as wheels and those projects would be willing to have them be the official PyPI downloads). This is roughly how the python.org installers are handled, and while it may cause some lag between source and binary releases, I think the release cycles are slow enough that most users would only notice that "pip install numpy" now works.
For the record, a long term “down the road” kind of thing I want to do is have PyPI have a build farm which can build Wheels. So that people upload a source distribution to PyPI and it kicks off Wheel builds on the various platforms automatically. What this looks like is a complete unknown, all I have is the general idea. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
On 10/29/2014 11:37 AM, Steve Dower wrote:
My ideal target (Utopia refined to be achievable) is for python-dev to handle the Python binaries themselves (already done) and to make them easy to bundle with applications/etc (I'm working on this for 3.5), and for PyPI to include pre-built wheels for Windows where necessary.
Note that I don't require package developers to build their own wheels, though I think that they are generally the right people to be doing it. It would be nice to create a culture of delegation, so that "someone" could be a trusted builder for a range of packages
Christoph Gohlke, Laboratory for Fluorescence Dynamics, University of California, Irvine now provides 32 and 64 bit builds for 2.6/7, 3.2/3/4 (10 total) for about 300 'scientific open-source extension packages' an his site. http://www.lfd.uci.edu/~gohlke/pythonlibs/ Perhaps the PyPI folks could discuss incorporating wheel versions on PyPI (I have no idea what is involved, but I presume it could be antomated.) -- Terry Jan Reedy
(Paul Moore already covered most of this, but I'll go into a bit more detail in a couple of areas) On 29 October 2014 00:46, Tony Kelman <kelman@berkeley.edu> wrote:
Stephen J. Turnbull:
It should be evident by now that our belief is that the large majority of Windows users is well-served by the current model
This is not the case at all in the scientific community. NumPy and SciPy put in a lot of extra work to come up with something that is compatible with the MSVC build of CPython because they have to, not because they're "happy to" jump through the necessary hoops.
Lots of folks are happy with POSIX emulation layers on Windows, as they're OK with "basically works" rather than "works like any other native application". "Basically works" isn't sufficient for many Python-on-Windows use cases though, so the core ABI is a platform native one, rather than a POSIX emulation. This makes Python fit in more cleanly with other Windows applications, but makes it harder to write Python applications that span both POSIX and Windows. The degree to which a language runtime abstracts away the details of the underlying operating system is a complex trade-off, from C (where the C/C++ ABI is a core part of what *defines* an operating system), through Python (which lets the developer largely choose whether to use high level abstractions or lower level platform specific APIs) to the JVM and CLR (which both largely abstract away many of the details of the underlying operating system, aside from carefully contained "native" code snippets). This approach *does* fragment the community a bit, into at least "Python-on-POSIX-only", "Python-on-Windows-only", "platform-independent-Python" and "Python-on-POSIX-and-Windows" (where the latter code has to deal directly with the differences between Windows and POSIX because it needs the low level platform specific APIs). I personally consider that trade-off worth it, as it gives Python a scope of applicability that more isolated runtimes lack, as well as providing access to sophisticated platform level capabilities far more cheaply than would be possible with a higher level of default isolation.
The situation today with NumPy AIUI is they already have to build with a custom hacked MinGW toolchain that only one person knows how to use. Evidently until very recently they were using a many-years-old 32-bit-only build of GCC 3.x. Do python-dev and numpy-discussion not talk to one another? I get that not everyone uses numpy, or Windows, but it sounds like there's very little understanding, or even awareness, of the issues they face.
There have been major ongoing communication problems stemming from the wildly different assumptions that go into "programming as a tool for building applications" (what python-dev and distutils-sig mostly do) and "programming as a tool for thinking" (what scientists and data analysts mostly do). Adding FORTRAN dependencies to the mix (which *none* of the platform vendors really support properly) then makes the whole problem of dealing with scientific tools even more alien to most professional developers. Those differing assumptions then meant that the two groups end up frequently talk past each other because the worldviews are too different. While that's starting to improve of late (as the above distinction becomes better recognised), there's still a long way to go. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, 30 Oct 2014 01:09:45 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Lots of folks are happy with POSIX emulation layers on Windows, as they're OK with "basically works" rather than "works like any other native application". "Basically works" isn't sufficient for many Python-on-Windows use cases though, so the core ABI is a platform native one, rather than a POSIX emulation.
This makes Python fit in more cleanly with other Windows applications, but makes it harder to write Python applications that span both POSIX and Windows.
I don't really understanding why that's the case. Only the building and packaging may be more difficult, and that assumes you're familiar with mingw32. But mingw32, AFAIK, doesn't make the Windows runtime magically POSIX-compatible (Cygwin does, to some extent). Regards Antoine.
On Wed, Oct 29, 2014 at 3:25 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 30 Oct 2014 01:09:45 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Lots of folks are happy with POSIX emulation layers on Windows, as they're OK with "basically works" rather than "works like any other native application". "Basically works" isn't sufficient for many Python-on-Windows use cases though, so the core ABI is a platform native one, rather than a POSIX emulation.
This makes Python fit in more cleanly with other Windows applications, but makes it harder to write Python applications that span both POSIX and Windows.
I don't really understanding why that's the case. Only the building and packaging may be more difficult, and that assumes you're familiar with mingw32. But mingw32, AFAIK, doesn't make the Windows runtime magically POSIX-compatible (Cygwin does, to some extent).
mingw32 is a more compliant C compiler (VS2008 does not implement much from C89), and it does implement quite a few things not implemented in the C runtime, especially for math. But TBH, those are not compelling cases to build python itself on mingw, only to better support C extensions with mingw. David
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/cournape%40gmail.com
On Wed, Oct 29, 2014 at 5:17 PM, David Cournapeau <cournape@gmail.com> wrote:
On Wed, Oct 29, 2014 at 3:25 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:
On Thu, 30 Oct 2014 01:09:45 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:
Lots of folks are happy with POSIX emulation layers on Windows, as they're OK with "basically works" rather than "works like any other native application". "Basically works" isn't sufficient for many Python-on-Windows use cases though, so the core ABI is a platform native one, rather than a POSIX emulation.
This makes Python fit in more cleanly with other Windows applications, but makes it harder to write Python applications that span both POSIX and Windows.
I don't really understanding why that's the case. Only the building and packaging may be more difficult, and that assumes you're familiar with mingw32. But mingw32, AFAIK, doesn't make the Windows runtime magically POSIX-compatible (Cygwin does, to some extent).
mingw32 is a more compliant C compiler (VS2008 does not implement much from C89)
That should read much C99, of course, otherwise VS 2008 would have been a completely useless C compiler ! David
On Thu, 30 Oct 2014 01:09:45 +1000, Nick Coghlan <ncoghlan@gmail.com> wrote:
(Paul Moore already covered most of this, but I'll go into a bit more detail in a couple of areas)
On 29 October 2014 00:46, Tony Kelman <kelman@berkeley.edu> wrote:
Stephen J. Turnbull:
It should be evident by now that our belief is that the large majority of Windows users is well-served by the current model
This is not the case at all in the scientific community. NumPy and SciPy put in a lot of extra work to come up with something that is compatible with the MSVC build of CPython because they have to, not because they're "happy to" jump through the necessary hoops.
Lots of folks are happy with POSIX emulation layers on Windows, as they're OK with "basically works" rather than "works like any other native application". "Basically works" isn't sufficient for many Python-on-Windows use cases though, so the core ABI is a platform native one, rather than a POSIX emulation.
Since some of the context here is scientific use of Python, it might be a useful bit of perspective to know that, while there are doubtless many scientists using windows and using the windows native interfaces happily, the Software Carpentry bootcamps that aim to give scientists the basic framework for making better use of computers and software and programming have as one foundation the bash shell, taught on Windows via git-bash. That is, the common toolset being taught to scientists (by Software Carpentry) is the posix one, even on Windows. --David
Stephen J. Turnbull:
the pain of using Windows is what drives me away from all of them.
Enough that you are not able to make the software you write usable on Windows? I see your point and agree with it - I don't even like Windows much at all, but supporting it is important for plenty of reasons. Steve Dower:
FWIW, I'm happy to talk specifics off list
Off-list, on-list, either way. I've read the blog post about the CRT refactoring http://blogs.msdn.com/b/vcblog/archive/2014/06/10/the-great-crt-refactoring.... but have not yet experimented with the early CTP versions of Visual Studio "14." Is there any plan to promote these runtimes to installed-by-default components of the operating system in future service packs or new releases of Windows? The fact that the redistributables are a separate download and not always there by default means MinGW developers are not quite so optimistic about them suddently solving all problems. In the embedded- Python-for-IJulia pull request on github for example, it turns out the Python msi installer does not include the necessary runtimes and wouldn't work by itself on a brand new computer. It would also help if there were a preview version of just the runtime libraries available. More far-fetched and not likely to happen would be a smaller "command line tools for Visual Studio" type package, like Apple provides for Xcode, with only the compiler, tools, and libraries needed to build C/C++ libraries or Python extensions but not the whole IDE. Nick Coghlan:
* wanting to build for Windows within the scope of an existing POSIX based build automation system * folks that prefer to use only open source software themselves wanting to make their projects available to Windows users
Don't forget: * MSVC is not capable of compiling the code I need to use. That's the crux of the scientific-software problem. If it were possible to use MSVC for everything, I'd probably suck it up and spend my time writing CMake build systems for every project I wanted to get to work on Windows. Most software I work with is a library first, Python extension second, if at all. Nick Coghlan:
Lots of folks are happy with POSIX emulation layers on Windows, as they're OK with "basically works" rather than "works like any other native application". "Basically works" isn't sufficient for many Python-on-Windows use cases though, so the core ABI is a platform native one, rather than a POSIX emulation.
To clarify here: MinGW does not use a POSIX layer at all. Cygwin does, MSYS does (it's based on Cygwin with relatively minor changes), but when we're discussing the MinGW.org or MinGW-w64 compilers, there is no POSIX. There is a GCC runtime library, which can be linked statically if you choose, though it's not always a good idea to do so. Cygwin or MSYS provide a POSIX build environment where you can run bash, make, etc, but with MinGW you are not targeting or depending on that build environment with the compiled binaries. The distinction in runtime libraries is that MinGW links to msvcrt.dll, which is from an old version of Visual Studio but is installed by default on every version of Windows since XP. The python.org installers are compiled with a particular more recent version of Visual Studio, so they (and hence all extensions used with them) depend on the specific corresponding msvcr##.dll. Most MinGW developers and users would rather avoid the dll hell and issues stemming from choosing a particular MSVC runtime and having to stick with it for everything. NumPy was able to coax a custom MinGW-w64 build into linking to CPython's designated runtime and statically link all the GCC libraries (these are 2 separate issues however). I do think some sensitivity to the requirements of the existing Windows CPython/PyPI environment from the MinGW-w64 side would be valuable. If NumPy's needs can be met with fewer modifications and without having to rebuild a custom GCC from source, I think that would be beneficial for everyone. We'll see how many of the MinGW-w64 developers I can get to agree with me on that. Any changes that may need to be upstreamed to GCC will have their own set of challenges.
Adding FORTRAN dependencies to the mix (which *none* of the platform vendors really support properly)
I'm not sure what you mean by platform vendors? Are you talking Conda, Enthought, etc? To the best of my knowledge, they're using the Intel toolchain for Fortran, at least on Windows and probably elsewhere. So I suspect you'll find a libifcoremd.dll somewhere in those distributions. That imposes some cost requirements on reproducing their processes that are a bit high for average users to meet. Donald Stufft:
So that people upload a source distribution to PyPI and it kicks off Wheel builds on the various platforms automatically.
What this looks like is a complete unknown, all I have is the general idea.
I do something similar today for C, C++, and Fortran libraries using the MinGW-w64 cross-compiler. It looks like this https://build.opensuse.org/project/show/windows:mingw:win64 I write a spec file and upload source, and get back dll's and exe's - compressed in RPM's in this case, along with the RPM dependency metadata. For OSX I would look at what Homebrew does with their "bottle" infrastructure. A build farm with Vagrant (or similar) VM's could even be made to do the same basic thing on Windows with MSVC, at least for binaries that MSVC is capable of compiling. -Tony
On 10/28/2014 6:45 AM, Stephen J. Turnbull wrote:
because it's a fork, it's a different name I think this is an important point, and first brought to this discussion here. A fork is _not_ called Python, but something else... but if it is kept extremely compatible and up-to-date in the hopes of reintegration once it proves that it solves a problem, and that there are sufficient users, then such hopes seem reasonable.
And targeting the new "future compatible" MSVCRT sounds like the right approach, although it won't solve today's problems today, but it may solve tomorrow's problems for a long time into the future... if the MinGW people can be convinced to support that new MSVCRT as well. In addition to all the components that are enabled by MinGW (particularly Fortran based extensions), one must remember that the current Windows Python also has extensions that interface to MSVC libraries that have never been ported to MinGW or Linux, and may never be. So an incompatible MinGW-built fork will lose some of those extensions... they may not be important to the folks that need MinGW, but that would be where & why the community would be split if the MinGW fork is not compatible with (some version of MSVC). Of course, the current MSVC-based community is _already_ having issues with incompatible versions of MSVC (not limiting that community to Python, but broader users of MSVC)... enough problems that even M$ has noticed that their incompatibilities are problematical, and are attempting to address... not just for Python, but for many other systems and libraries as well. So gathering around and supporting their attempts to achieve that, by using their new system early, when feedback can still have a chance of improving that new "future compatible" system, will benefit everyone... but that time is limited, from what Steve Dower reports of the schedule... hoping to be ready for Python 3.5. So now is an excellent time for this discussion to be happening, and if some work can be done to fork, pull the patches together, and tweak them to work with the new MSVC, in the Python 3.5 timeframe, you can have a phenomenal result, even if it is still a fork at that time.
participants (14)
-
Antoine Pitrou -
David Cournapeau -
Donald Stufft -
Ethan Furman -
Glenn Linderman -
Nathaniel Smith -
Nick Coghlan -
Paul Moore -
R. David Murray -
Stephen J. Turnbull -
Steve Dower -
Terry Reedy -
Tony Kelman -
Tres Seaver