As I understand it, the MS VS2010 compiler is required (or at least best practice) for compiling Python extensions for the python.org Windows builds of py 3.4 and ?[1] However, MS now makes it very hard (impossible?) to download VS2010 Express ( or Community, or whatever the free as in beer version is called). I realize that this is not python-dev's responsibility, but if there is any way to either document where it can be found, or put a bit of pressure on MS to make it available, as they have for VS2008 and py2.7, that would be great. Sorry to bug this list, I didn't know where else to reach out to. -Chris [1] it's actually prefer hard to find out which compiler version is used for which python version. And has been for years. Would a patch to the docs, probably here: https://docs.python.org/3.4/using/windows.html#compiling-python-on-windows Be considered?
You can use "Windows SDK for Windows 7 and .NET Framework 4". http://www.microsoft.com/en-us/download/details.aspx?id=8279 On Sat, Sep 26, 2015 at 12:24 AM, Chris Barker - NOAA Federal < chris.barker@noaa.gov> wrote:
As I understand it, the MS VS2010 compiler is required (or at least best practice) for compiling Python extensions for the python.org Windows builds of py 3.4 and ?[1]
However, MS now makes it very hard (impossible?) to download VS2010 Express ( or Community, or whatever the free as in beer version is called).
I realize that this is not python-dev's responsibility, but if there is any way to either document where it can be found, or put a bit of pressure on MS to make it available, as they have for VS2008 and py2.7, that would be great.
Sorry to bug this list, I didn't know where else to reach out to.
-Chris
[1] it's actually prefer hard to find out which compiler version is used for which python version. And has been for years. Would a patch to the docs, probably here:
https://docs.python.org/3.4/using/windows.html#compiling-python-on-windows
Be considered? _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
-- INADA Naoki <songofacandy@gmail.com>
You can use "Windows SDK for Windows 7 and .NET Framework 4". http://www.microsoft.com/en-us/download/details.aspx?id=8279 Thanks. Last time I tried that route, it was for 64 bit py2.7. And it required some kludging of environment variables, and registry acces I don't have permission for. But it still may be the best bet. I'll give it a try when I have a chance. And this should be in the "official" docs... -CHB On Sat, Sep 26, 2015 at 12:24 AM, Chris Barker - NOAA Federal < chris.barker@noaa.gov> wrote:
As I understand it, the MS VS2010 compiler is required (or at least best practice) for compiling Python extensions for the python.org Windows builds of py 3.4 and ?[1]
However, MS now makes it very hard (impossible?) to download VS2010 Express ( or Community, or whatever the free as in beer version is called).
I realize that this is not python-dev's responsibility, but if there is any way to either document where it can be found, or put a bit of pressure on MS to make it available, as they have for VS2008 and py2.7, that would be great.
Sorry to bug this list, I didn't know where else to reach out to.
-Chris
[1] it's actually prefer hard to find out which compiler version is used for which python version. And has been for years. Would a patch to the docs, probably here:
https://docs.python.org/3.4/using/windows.html#compiling-python-on-windows
Be considered? _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
-- INADA Naoki <songofacandy@gmail.com>
On Fri, Sep 25, 2015 at 6:27 PM, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
You can use "Windows SDK for Windows 7 and .NET Framework 4".
http://www.microsoft.com/en-us/download/details.aspx?id=8279
Thanks. Last time I tried that route, it was for 64 bit py2.7. And it required some kludging of environment variables, and registry acces I don't have permission for.
But it still may be the best bet. I'll give it a try when I have a chance.
And this should be in the "official" docs...
For what it's worth, I've had good luck compiling *most* extension modules on Windows using the gcc from MinGW-w64. The big notable exception was that last time I tried compiling Numpy with it I got a segfaulting Numpy. But I never had a chance to investigate why or if it's fixable. My own extension modules work fine on Windows when compiled in MinGW though. Erik B.
On Sat, Sep 26, 2015 at 12:24 AM, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
As I understand it, the MS VS2010 compiler is required (or at least best practice) for compiling Python extensions for the python.org Windows builds of py 3.4 and ?[1]
However, MS now makes it very hard (impossible?) to download VS2010 Express ( or Community, or whatever the free as in beer version is called).
I realize that this is not python-dev's responsibility, but if there is any way to either document where it can be found, or put a bit of pressure on MS to make it available, as they have for VS2008 and py2.7, that would be great.
Sorry to bug this list, I didn't know where else to reach out to.
-Chris
[1] it's actually prefer hard to find out which compiler version is used for which python version. And has been for years. Would a patch to the docs, probably here:
https://docs.python.org/3.4/using/windows.html#compiling-python-on-windows
Be considered? _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
-- INADA Naoki <songofacandy@gmail.com>
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/erik.m.bray%40gmail.com
On 2015-09-28 19:00, Erik Bray wrote:
On Fri, Sep 25, 2015 at 6:27 PM, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
You can use "Windows SDK for Windows 7 and .NET Framework 4".
http://www.microsoft.com/en-us/download/details.aspx?id=8279
Thanks. Last time I tried that route, it was for 64 bit py2.7. And it required some kludging of environment variables, and registry acces I don't have permission for.
But it still may be the best bet. I'll give it a try when I have a chance.
And this should be in the "official" docs...
For what it's worth, I've had good luck compiling *most* extension modules on Windows using the gcc from MinGW-w64.
The big notable exception was that last time I tried compiling Numpy with it I got a segfaulting Numpy. But I never had a chance to investigate why or if it's fixable. My own extension modules work fine on Windows when compiled in MinGW though.
Erik B.
Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5, both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet that wasn't due to a bug in the source code.
On Sat, Sep 26, 2015 at 12:24 AM, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
As I understand it, the MS VS2010 compiler is required (or at least best practice) for compiling Python extensions for the python.org Windows builds of py 3.4 and ?[1]
However, MS now makes it very hard (impossible?) to download VS2010 Express ( or Community, or whatever the free as in beer version is called).
I realize that this is not python-dev's responsibility, but if there is any way to either document where it can be found, or put a bit of pressure on MS to make it available, as they have for VS2008 and py2.7, that would be great.
Sorry to bug this list, I didn't know where else to reach out to.
-Chris
[1] it's actually prefer hard to find out which compiler version is used for which python version. And has been for years. Would a patch to the docs, probably here:
https://docs.python.org/3.4/using/windows.html#compiling-python-on-windows
Be considered? _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com
-- INADA Naoki <songofacandy@gmail.com>
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/erik.m.bray%40gmail.com
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/python%40mrabarnett.plus....
On 28 September 2015 at 21:18, MRAB <python@mrabarnett.plus.com> wrote:
Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5, both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet that wasn't due to a bug in the source code.
Interesting. With Python 3.5, what CRT does the module use? With Python 3.4 and earlier, distutils arranged for mingw builds to link to the CRT that matched the CPython build, but I don't think that happened for 3.5 (because mingw doesn't support ucrt yet). So you'd have a CRT that doesn't match the core. Obviously this isn't causing you issues, but I'm not sure if it could (CRT problems seem to be mostly about "might cause issues" type problems...) Paul
On 2015-09-28 21:38, Paul Moore wrote:
On 28 September 2015 at 21:18, MRAB <python@mrabarnett.plus.com> wrote:
Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5, both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet that wasn't due to a bug in the source code.
Interesting. With Python 3.5, what CRT does the module use? With Python 3.4 and earlier, distutils arranged for mingw builds to link to the CRT that matched the CPython build, but I don't think that happened for 3.5 (because mingw doesn't support ucrt yet). So you'd have a CRT that doesn't match the core. Obviously this isn't causing you issues, but I'm not sure if it could (CRT problems seem to be mostly about "might cause issues" type problems...)
I can't remember the details off-hand, but here's the issue: http://bugs.python.org/issue24385
On Sep 28, 2015 1:39 PM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 28 September 2015 at 21:18, MRAB <python@mrabarnett.plus.com> wrote:
Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5, both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet that wasn't due to a bug in the source code.
Interesting. With Python 3.5, what CRT does the module use? With Python 3.4 and earlier, distutils arranged for mingw builds to link to the CRT that matched the CPython build,
This doesn't sound right to me. Maybe if you're talking about the original mingw, but that's been abandoned upstream for years. Nowadays everyone uses the mingw-w64 fork, for both 32 and 64 bits. And in mingw-w64, the only way to select a non-default CRT is to hack up the toolchain's source tree and cross your fingers. It's definitely not a supported configuration upstream. (Source: we've been talking to mingw-w64 upstream and MS about how to potentially make building free software less painful on Windows.)
but I don't think that happened for 3.5 (because mingw doesn't support ucrt yet). So you'd have a CRT that doesn't match the core. Obviously this isn't causing you issues, but I'm not sure if it could (CRT problems seem to be mostly about "might cause issues" type problems...)
CRT issues are indeed tricky, because they only bite in certain circumstances -- so long as you never pass a FILE* or a fileno across the dll boundary, or call malloc in one module and free in the other, or ... then it works fine. Except when it doesn't :-). Example of a subtle issue: msvc and mingw-w64 disagree about whether the x87 fpu should be configured in extended precision mode or not. So simply loading a mingw-w64-compiled dll can potentially cause math functions in other dlls to start returning incorrect results. Or maybe not -- it depends on the details of how MS implemented things, and we can't exactly check. Or whether you get correct math result in mingw-w64 compiled code can depend on which runtime was used to spawn the thread that's running the code (!!). And even if things work today, it could change without warning in any update, because no one is testing this configuration upstream. It's totally plausible that the regex module happens to avoid all these issues. But you can't assume that it's a general solution. -n
On 28 September 2015 at 22:18, Nathaniel Smith <njs@pobox.com> wrote:
On Sep 28, 2015 1:39 PM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 28 September 2015 at 21:18, MRAB <python@mrabarnett.plus.com> wrote:
Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5, both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet that wasn't due to a bug in the source code.
Interesting. With Python 3.5, what CRT does the module use? With Python 3.4 and earlier, distutils arranged for mingw builds to link to the CRT that matched the CPython build,
This doesn't sound right to me. Maybe if you're talking about the original mingw, but that's been abandoned upstream for years. Nowadays everyone uses the mingw-w64 fork, for both 32 and 64 bits. And in mingw-w64, the only way to select a non-default CRT is to hack up the toolchain's source tree and cross your fingers. It's definitely not a supported configuration upstream. (Source: we've been talking to mingw-w64 upstream and MS about how to potentially make building free software less painful on Windows.)
Interesting, it's been a long time since I've done work on the mingw support, so I hadn't realised this had happened. I guess that means that nowadays all mingw-compiled extensions link to msvcrt (I assume that's still the CRT that mingw-w64 uses by default)?
but I don't think that happened for 3.5 (because mingw doesn't support ucrt yet). So you'd have a CRT that doesn't match the core. Obviously this isn't causing you issues, but I'm not sure if it could (CRT problems seem to be mostly about "might cause issues" type problems...)
CRT issues are indeed tricky, because they only bite in certain circumstances -- so long as you never pass a FILE* or a fileno across the dll boundary, or call malloc in one module and free in the other, or ... then it works fine. Except when it doesn't :-).
Example of a subtle issue: msvc and mingw-w64 disagree about whether the x87 fpu should be configured in extended precision mode or not. So simply loading a mingw-w64-compiled dll can potentially cause math functions in other dlls to start returning incorrect results. Or maybe not -- it depends on the details of how MS implemented things, and we can't exactly check. Or whether you get correct math result in mingw-w64 compiled code can depend on which runtime was used to spawn the thread that's running the code (!!). And even if things work today, it could change without warning in any update, because no one is testing this configuration upstream.
It's totally plausible that the regex module happens to avoid all these issues. But you can't assume that it's a general solution.
Agreed. That was essentially what I was asking - has someone found a solution to the issue of avoiding mixed CRTs when using mingw as a compiler (or at least, found a better solution than using the original mingw -lmsvcr100 approach combined with the distutils code that supports this). Looks like the answer is that nobody has. But it's certainly (still) true that for straightforward cases things seem pretty much OK, especially if you avoid the "obvious" issues. (The non-obvious ones sound terrifying, and I'm glad I don't have to write code that encounters those!!!) Cheers, Paul
On Mon, Sep 28, 2015 at 2:55 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 28 September 2015 at 22:18, Nathaniel Smith <njs@pobox.com> wrote:
On Sep 28, 2015 1:39 PM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 28 September 2015 at 21:18, MRAB <python@mrabarnett.plus.com> wrote:
Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5, both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet that wasn't due to a bug in the source code.
Interesting. With Python 3.5, what CRT does the module use? With Python 3.4 and earlier, distutils arranged for mingw builds to link to the CRT that matched the CPython build,
This doesn't sound right to me. Maybe if you're talking about the original mingw, but that's been abandoned upstream for years. Nowadays everyone uses the mingw-w64 fork, for both 32 and 64 bits. And in mingw-w64, the only way to select a non-default CRT is to hack up the toolchain's source tree and cross your fingers. It's definitely not a supported configuration upstream. (Source: we've been talking to mingw-w64 upstream and MS about how to potentially make building free software less painful on Windows.)
Interesting, it's been a long time since I've done work on the mingw support, so I hadn't realised this had happened.
I guess that means that nowadays all mingw-compiled extensions link to msvcrt (I assume that's still the CRT that mingw-w64 uses by default)?
Right, linking to msvcrt.dll is the only standard/supported configuration for upstream mingw-w64.
but I don't think that happened for 3.5 (because mingw doesn't support ucrt yet). So you'd have a CRT that doesn't match the core. Obviously this isn't causing you issues, but I'm not sure if it could (CRT problems seem to be mostly about "might cause issues" type problems...)
CRT issues are indeed tricky, because they only bite in certain circumstances -- so long as you never pass a FILE* or a fileno across the dll boundary, or call malloc in one module and free in the other, or ... then it works fine. Except when it doesn't :-).
Example of a subtle issue: msvc and mingw-w64 disagree about whether the x87 fpu should be configured in extended precision mode or not. So simply loading a mingw-w64-compiled dll can potentially cause math functions in other dlls to start returning incorrect results. Or maybe not -- it depends on the details of how MS implemented things, and we can't exactly check. Or whether you get correct math result in mingw-w64 compiled code can depend on which runtime was used to spawn the thread that's running the code (!!). And even if things work today, it could change without warning in any update, because no one is testing this configuration upstream.
It's totally plausible that the regex module happens to avoid all these issues. But you can't assume that it's a general solution.
Agreed. That was essentially what I was asking - has someone found a solution to the issue of avoiding mixed CRTs when using mingw as a compiler (or at least, found a better solution than using the original mingw -lmsvcr100 approach combined with the distutils code that supports this). Looks like the answer is that nobody has. But it's certainly (still) true that for straightforward cases things seem pretty much OK, especially if you avoid the "obvious" issues. (The non-obvious ones sound terrifying, and I'm glad I don't have to write code that encounters those!!!)
It's not *quite* "nobody has". Carl Kleffner has been doing a lot of work on trying to set up Python-compatible gcc-based toolchains for Windows. (This is still pretty much the only hope for getting a usable free fortran compiler on Windows, and there's still lots of fortran down at the bottom of the scientific/numerical stack). It's still rather beta and the approach probably isn't sustainable without more funding, but right now it's probably the closest thing you can get to what you're asking about. I believe these are the latest builds: https://binstar.org/carlkl/mingwpy/files (Yes, those are .whl files that contain compiler toolchains.) (Also, if making this *actually* work reliably would make a difference to anyone, then feel free to send me heartfelt testimonials and I'll forward them on to MS. Or I guess you could just send them to Steve Dower directly ;-).) -n -- Nathaniel J. Smith -- http://vorpus.org
On 29/09/2015 00:52, Nathaniel Smith wrote:
On Mon, Sep 28, 2015 at 2:55 PM, Paul Moore <p.f.moore@gmail.com> wrote:
On 28 September 2015 at 22:18, Nathaniel Smith <njs@pobox.com> wrote:
On Sep 28, 2015 1:39 PM, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 28 September 2015 at 21:18, MRAB <python@mrabarnett.plus.com> wrote:
Same here. I compile the regex module for Python 2.5-2.7 and 3.1-3.5, both 32-bit and 64-bit, using MinGW-w64, and I haven't had a problem yet that wasn't due to a bug in the source code.
Interesting. With Python 3.5, what CRT does the module use? With Python 3.4 and earlier, distutils arranged for mingw builds to link to the CRT that matched the CPython build,
This doesn't sound right to me. Maybe if you're talking about the original mingw, but that's been abandoned upstream for years. Nowadays everyone uses the mingw-w64 fork, for both 32 and 64 bits. And in mingw-w64, the only way to select a non-default CRT is to hack up the toolchain's source tree and cross your fingers. It's definitely not a supported configuration upstream. (Source: we've been talking to mingw-w64 upstream and MS about how to potentially make building free software less painful on Windows.)
Interesting, it's been a long time since I've done work on the mingw support, so I hadn't realised this had happened.
I guess that means that nowadays all mingw-compiled extensions link to msvcrt (I assume that's still the CRT that mingw-w64 uses by default)?
Right, linking to msvcrt.dll is the only standard/supported configuration for upstream mingw-w64.
but I don't think that happened for 3.5 (because mingw doesn't support ucrt yet). So you'd have a CRT that doesn't match the core. Obviously this isn't causing you issues, but I'm not sure if it could (CRT problems seem to be mostly about "might cause issues" type problems...)
CRT issues are indeed tricky, because they only bite in certain circumstances -- so long as you never pass a FILE* or a fileno across the dll boundary, or call malloc in one module and free in the other, or ... then it works fine. Except when it doesn't :-).
Example of a subtle issue: msvc and mingw-w64 disagree about whether the x87 fpu should be configured in extended precision mode or not. So simply loading a mingw-w64-compiled dll can potentially cause math functions in other dlls to start returning incorrect results. Or maybe not -- it depends on the details of how MS implemented things, and we can't exactly check. Or whether you get correct math result in mingw-w64 compiled code can depend on which runtime was used to spawn the thread that's running the code (!!). And even if things work today, it could change without warning in any update, because no one is testing this configuration upstream.
It's totally plausible that the regex module happens to avoid all these issues. But you can't assume that it's a general solution.
Agreed. That was essentially what I was asking - has someone found a solution to the issue of avoiding mixed CRTs when using mingw as a compiler (or at least, found a better solution than using the original mingw -lmsvcr100 approach combined with the distutils code that supports this). Looks like the answer is that nobody has. But it's certainly (still) true that for straightforward cases things seem pretty much OK, especially if you avoid the "obvious" issues. (The non-obvious ones sound terrifying, and I'm glad I don't have to write code that encounters those!!!)
It's not *quite* "nobody has". Carl Kleffner has been doing a lot of work on trying to set up Python-compatible gcc-based toolchains for Windows. (This is still pretty much the only hope for getting a usable free fortran compiler on Windows, and there's still lots of fortran down at the bottom of the scientific/numerical stack). It's still rather beta and the approach probably isn't sustainable without more funding, but right now it's probably the closest thing you can get to what you're asking about. I believe these are the latest builds:
https://binstar.org/carlkl/mingwpy/files
(Yes, those are .whl files that contain compiler toolchains.)
(Also, if making this *actually* work reliably would make a difference to anyone, then feel free to send me heartfelt testimonials and I'll forward them on to MS. Or I guess you could just send them to Steve Dower directly ;-).)
-n
For anyone interested although this http://bugs.python.org/issue3871 mingw issue is closed, it points to four other issues. In total there are around 25 mingw issues open. Maybe some of the work done on them can be tied up with the work that Carl Kleffner has done? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence
On Mon, Sep 28, 2015 at 5:14 PM, Mark Lawrence <breamoreboy@yahoo.co.uk> wrote:
For anyone interested although this http://bugs.python.org/issue3871 mingw issue is closed, it points to four other issues. In total there are around 25 mingw issues open. Maybe some of the work done on them can be tied up with the work that Carl Kleffner has done?
There are two different topics here: building extensions with mingw-w64, and building CPython itself with mingw-w64. That issue is about the latter, and I think everyone is agreed at this point that Python.org isn't going to support that so long as there are incompatible CRT/ABIs between mingw-w64 and MSVC, because that creates all kinds of problems (e.g. now you have to distinguish between wheels targeting win32-using-msvc versus win32-using-mingw-w64). So the first thing to solve is to get the mingw-w64 CRT/ABI incompatibilities sorted out and extensions working, and then once that's solid we could think about whether CPython upstream is interested in carrying mingw-w64 build patches for CPython itself. (It might not even be that interesting at that point... the main reason for wanting mingw-w64-built CPython is so that it can interoperate with the mingw-w64 ecosystem -- e.g. supporting Python plugins in mingw-w64's gdb. If MSVC and mingw-w64 python become compatible with each other, then this becomes much less urgent...) The msys2 folks who are maintaining the CPython-on-mingw-w64 patch set are definitely aware of Carl's work. -n -- Nathaniel J. Smith -- http://vorpus.org
And in mingw-w64, the only way to select a non-default CRT
what is a "default" CRT in this case??
CRT issues are indeed tricky, because they only bite in certain circumstances -- so long as you never pass a FILE* or a fileno across the dll boundary, or call malloc in one module and free in the other, or ...
Which happens to be EXACTLY what I'm needing to do right now.... then it works fine. Except when it doesn't :-).
yup -- a new developer on my team spent yesterday bangin his head agaist this -- then asked "I'm compiling the lib with VS2013, the the python extension with VS2008, could that be the problem? -- uh, yes? The Python core developers made the decision a long time ago to use the MS compilers for the "official" distributions -- as they are the "platform" compilers. And as far as I know, everyone else has followed that path for compatibility (Enthought Canopy, Anaconda, ActiveState) (except cygWin, but that's not really the Windows platform, IFAIC). Given MS decision to both: tie the compiler tightly to the run-time, AND make it hard to get old versions of the compiler, that's turned out to be a pain. But I suspect there have been real advantages over, for instance, building the whole thing on MinGW. But in any case, that's what we've got -- and it would be really nice to support teh community as best we can, as has been done fro Python2.7 with updates to setuptools and the MS download of the old VS2008. It sounds like we don't expect MS t help out in this case, but at least we can better document what users need to do, and how to do it -- and NOT with a "kludge together an open-source compiler and cross your fingers" approach. OK -- I'm going to get off my soap box now -- time to actually suggest doc patches.... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 29Sep2015 0820, Chris Barker wrote:
OK -- I'm going to get off my soap box now -- time to actually suggest doc patches....
Just bear in mind that you're suggesting patches for Python 3.3 and 3.4, which means that 3.4.4 is the only real chance to get them onto people's machines. http://docs.python.org/ already points to the 3.5 docs by default, which means only people who explicitly look for the earlier versions will see the changes. You don't really have a huge audience, so I wouldn't invest large amounts of time in the official docs. A couple of well-publicised blog posts will reach more people. Cheers, Steve
On Tue, 29 Sep 2015 17:20 Steve Dower <steve.dower@python.org> wrote: On 29Sep2015 0820, Chris Barker wrote:
OK -- I'm going to get off my soap box now -- time to actually suggest doc patches....
Just bear in mind that you're suggesting patches for Python 3.3 and 3.4, which means that 3.4.4 is the only real chance to get them onto people's machines. http://docs.python.org/ already points to the 3.5 docs by default, which means only people who explicitly look for the earlier versions will see the changes. You don't really have a huge audience, so I wouldn't invest large amounts of time in the official docs. A couple of well-publicised blog posts will reach more people. Maybe but I think it would be good to have some documentation somewhere (perhaps under the pypa banner) that can advise extension module authors who want to provide Windows binaries for multiple CPython versions. It's not obvious how to go about setting things up to make python.org compatible binaries for say Python 2.7 and 3.2-3.5. Before 3.5 the simplest solution was to use MinGW but that doesn't work for 3.5 (hence numpy are currently uploading rcs without 3.5 binaries for Windows). Just an explanation of what it takes in terms of MS compilers and which free as in beer ones are available where would be helpful which could go in a blog post as you say. IMO It would be more helpful though if it were a continuously improved document that will be updated as new CPython releases come out and the availability of MS compilers changes. -- Oscar
On Sep 29, 2015, at 9:20 AM, Steve Dower <steve.dower@python.org> wrote:
On 29Sep2015 0820, Chris Barker wrote: OK -- I'm going to get off my soap box now -- time to actually suggest doc patches....
Just bear in mind that you're suggesting patches for Python 3.3 and 3.4, which means that 3.4.4 is the only real chance to get them onto people's machines.
I don't expect to get it onto anyone's machine -- does anyone download docs anymore?
http://docs.python.org/ already points to the 3.5 docs by default, which means only people who explicitly look for the earlier versions will see the changes.
Well, if your trying to build an extension for 3.4, hopefully you'll think to look at 3.4 docs. But good point -- the 3.5 docs should have a note saying that the requirements are different than older releases.
You don't really have a huge audience, so I wouldn't invest large amounts of time in the official docs. A couple of well-publicised blog posts will reach more people.
Not on my blog..... At this point, I think the Cython Wiki us the best place to point people, but really, less than ideal. -CHB
Cheers, Steve
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
On Sep 29, 2015 8:22 AM, "Chris Barker" <chris.barker@noaa.gov> wrote:
And in mingw-w64, the only way to select a non-default CRT
what is a "default" CRT in this case??
To answer the question: mingw-w64's standard/default/only-officially-supported CRT is msvcrt.dll (the version that has no version number in the name). Wikipedia tells me that this is the runtime used by msvc 4.2 - 6.0 (the later of which was released in 1998). The reason for this decision is that for whatever historical reason, this particular CRT dll is shipped with Windows by default, and thus using it avoids certain license issues that can otherwise arise, esp. with GPL software. (Fortunately not an issue for python extensions, but since it is an issue for other programs they just made the decision to support a single configuration that works for everybody. Except when it doesn't ;-).) In practice mingw-w64 ships its own runtime as a layer on top of msvcrt.dll -- it gets used as a OS services layer, but mingw-w64 users aren't just getting dropped directly into a wonderland of 15+ year old code. (In fact, mingw-w64 upstream would like to make their runtime clever enough to bind the underlying "OS services layer" at load time rather than compile time, so that mingw-w64 compiled dll's would be "runtime agile" and could be safely loaded into any process. The main thing we care about is just that this would give upstream a way to officially support targeting newer CRTs, but it would have a number of other benefits too: one modern toolchain that could build extension modules for all versions of python, the ability when combined with Python's "stable abi" to target 3.2+ with a single binary distribution, better interoperability between python and other windows ecosystems that went the mingw-w64 route like R... but, this all requires funding someone to do the work.) -n
On 25 September 2015 at 16:35, INADA Naoki <songofacandy@gmail.com> wrote:
You can use "Windows SDK for Windows 7 and .NET Framework 4".
http://www.microsoft.com/en-us/download/details.aspx?id=8279
On Sat, Sep 26, 2015 at 12:24 AM, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
As I understand it, the MS VS2010 compiler is required (or at least best practice) for compiling Python extensions for the python.org Windows builds of py 3.4 and ?[1]
However, MS now makes it very hard (impossible?) to download VS2010 Express ( or Community, or whatever the free as in beer version is called).
I realize that this is not python-dev's responsibility, but if there is any way to either document where it can be found, or put a bit of pressure on MS to make it available, as they have for VS2008 and py2.7, that would be great.
Sorry to bug this list, I didn't know where else to reach out to.
On 29 September 2015 at 16:20, Chris Barker <chris.barker@noaa.gov> wrote:
It sounds like we don't expect MS t help out in this case, but at least we can better document what users need to do, and how to do it -- and NOT with a "kludge together an open-source compiler and cross your fingers" approach.
I'm not sure why INADA Naoki's answer above wasn't sufficient for you? You need to have the SDK compilers on your PATH (by running the relevant environment setting command) and you need to set the environment variable DISTUTILS_USE_SDK=1 (this latter bit of information is hard to find documented, admittedly, but pretty well known). That basically gives you a Visual Studio 2010 equivalent environment. You'll still have all sorts of *other* problems building extensions on Windows, but unless I'm missing something important in what you were saying, getting a compiler isn't the issue. Paul
I'm not sure why INADA Naoki's answer above wasn't sufficient for you?
It may be -- I've still have to test. Neither Windows nor py3 are my primary production versions. But the last time I tried the SDK approach was for py2.7 on Win64, and it was both a pain, and impossible to set up to "just work" in my highly secure environment. So at best, it needs some serious documentation. And no, it shouldn't be just a blog post somewhere (unless maybe Steve Dowers's blog). Thus whole issue has been dogging python on windows for many years -- it never has been easy to find out what to do. In fact, there are all sorts of Blog posts out there, with various levels of out of date, conflicting, or just plain wrong advice. A definitive version in the official docs would be a very good thing. BTW, I think this is a good discussion: https://github.com/cython/cython/wiki/CythonExtensionsOnWindows But not everyone is going to think to look at the Cython Wiki... -Chris
You need to have the SDK compilers on your PATH (by running the relevant environment setting command) and you need to set the environment variable DISTUTILS_USE_SDK=1 (this latter bit of information is hard to find documented, admittedly, but pretty well known).
That basically gives you a Visual Studio 2010 equivalent environment. You'll still have all sorts of *other* problems building extensions on Windows, but unless I'm missing something important in what you were saying, getting a compiler isn't the issue.
Paul
On 30 September 2015 at 01:31, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
It may be -- I've still have to test. Neither Windows nor py3 are my primary production versions.
Part of the problem here is that people put up postings saying "do this" having checked that it works for them, and then others say "but it doesn't work", without giving any information as to why or providing any means to improve the instructions. I'm not suggesting here that you're going to do this - but those instructions are in essence the "official" (not well documented, conceded) advice at present, and they have been tested by a number of people for whom they work. They have also been reported to fail, but (as noted above) never in a way that allows us to improve them :-( They will remain the official advice, I suspect, until we get better feedback.
But the last time I tried the SDK approach was for py2.7 on Win64, and it was both a pain, and impossible to set up to "just work" in my highly secure environment.
Well, I'd need more explanation of the issues. Was "install the SDK" a problem? Or was "set the environment variable DISTUTILS_USE_SDK"? Or did the instructions to do those two things not work? (I appreciate we're talking about 2 different things here, and I think Python 3 is a much less painful case than 2.7 on win64 was prior to the Visual C for Python 2.7 package). I suspect (from your mention of a highly secure environment) that installing the SDK was the main pain point. That's been the case for a number of people, I believe. But I don't think advice on how to install the Windows SDK in various environments is something we'd want to maintain in the Python documentation... This of course is another aspect of the problem - the official answer is "use a supported version of Windows, ensure you have the necessary permissions to install the needed software, and off you go". But in real life people want to build on an old laptop they had lying around, or on a VM they created on Linux, or on their Linux box using Wine, or on the box they use for a load of other development that has more SDKs than you want to imagine already installed, some of which got broken in the past, ... Or they need to use mingw, because a key library can't be compiled with Visual Studio. These are all real problems, and the people fighting them are doing a huge amount of good work building binary distributions for the community. They document what they are doing to solve their issues and those documents show up on Google, causing people who just want to build a simple extension to find them and get hopelessly confused.
So at best, it needs some serious documentation. And no, it shouldn't be just a blog post somewhere (unless maybe Steve Dowers's blog).
OK, without wishing to minimise what I fully agree is a serious problem, at a very high level the instructions are: For Python 2.7: 1. Install "Visual C++ Compiler Package for Python 2.7" 2. Use (a recent version of) setuptools in your setup.py (pip will do this for you, in any case). 3. Done. For Python 3.4 1. Install "Windows SDK for Windows 7 and .NET Framework 4" (v7.1) 2. Work from an SDK command prompt (with the environment variables set, and the SDK on PATH). 3. Set DISTUTILS_USE_SDK=1 4. Done. For Python 3.5 1. Install Visual Studio 2015 Community Edition (or any later version, when these are released). 2. Done.
Thus whole issue has been dogging python on windows for many years -- it never has been easy to find out what to do.
In fact, there are all sorts of Blog posts out there, with various levels of out of date, conflicting, or just plain wrong advice.
And this is why it's hard to fix the problem - people don't look at the documentation, they don't try to find "official" advice, they simply google and get a load of confusing advice. See above as to why some of those documents are there, and are more complex than the average user needs.
A definitive version in the official docs would be a very good thing.
Well, I'd happily put the above instructions in the packaging user guide (packaging.python.org). Would that be sufficient for you? Paul
TL;DR -- yes, documenting this in packaging.python.org would be great!
Part of the problem here is that people put up postings saying "do this" having checked that it works for them, and then others say "but it doesn't work", without giving any information as to why or providing any means to improve the instructions.
Exactly. Which is why a Blog post is not the answer. If I find a blog post, I have no idea how well vetted and tested it is. If you look at the history of the Cython Wiki page about this, it becomes very clear how much confusion there is. (Though I think it's in pretty god shape now-- wikis have the advantage of being able to evolve and getter better).
I'm not suggesting here that you're going to do this -
Absolutely not -- I would not post anything anywhere without review -- this's kind of the point. If I find it in an official source, I know it's been reviewed.
us to improve them :-( They will remain the official advice, I suspect, until we get better feedback.
Which is my point -- how do I currently know what the official advise is? (Yes, I know now -- thanks!) But posting to python-dev is exactly the wrong way to find out!
But the last time I tried the SDK approach was for py2.7 on Win64, and it was both a pain, and impossible to set up to "just work" in my highly secure environment.
Well, I'd need more explanation of the issues. Was "install the SDK" a problem? Or was "set the environment variable DISTUTILS_USE_SDK"? Or did the instructions to do those two things not work?
The issue was not having permissions to alter the registry the way we wanted. But again, I'll need to do some proper testing with up to date everything and see what happens.
I think Python 3 is a much less painful case than 2.7 on win64 was prior to the Visual C for Python 2.7 package).
Good to know.
But I don't think advice on how to install the Windows SDK in various environments is something we'd want to maintain in the Python documentation...
Agreed -- if you can't install it you can't install it. And if it can't be installed in user-space, that's between MS and your sysadmins...
This of course is another aspect of the problem - the official answer is "use a supported version of Windows, ensure you have the necessary permissions to install the needed software, and off you go".
Well, I for one think that's fine. Supporting open source software on a closed-source OS is going to have these pain points.
OK, without wishing to minimise what I fully agree is a serious problem, at a very high level the instructions are:
For Python 2.7:
1. Install "Visual C++ Compiler Package for Python 2.7" 2. Use (a recent version of) setuptools in your setup.py (pip will do this for you, in any case). 3. Done.
That new MS installer is really really great.
For Python 3.4
1. Install "Windows SDK for Windows 7 and .NET Framework 4" (v7.1) 2. Work from an SDK command prompt (with the environment variables set, and the SDK on PATH). 3. Set DISTUTILS_USE_SDK=1 4. Done.
This, unfortunately is non-trivial, and really a pain if you want to automate builds. Apparently, conda-build has got this automated under the hood. Maybe that could be kludge do into setuptools?
- people don't look at the documentation, they don't try to find "official" advice, they simply google and get a load of confusing advice.
Well, yes, but when there is no official advice, looking for it doesn't help.... I, for one, have looked for docs and official advise over and over again for many years ( back when kept trying to remember if py2.6 was built with VS2003 or VS2008, or ??
Well, I'd happily put the above instructions in the packaging user guide (packaging.python.org). Would that be sufficient for you?
That would be great, thanks! An "official" source, but easier to update that the regular docs. Thanks, -Chris
On 30 September 2015 at 16:57, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
1. Install "Windows SDK for Windows 7 and .NET Framework 4" (v7.1) 2. Work from an SDK command prompt (with the environment variables set, and the SDK on PATH). 3. Set DISTUTILS_USE_SDK=1 4. Done.
This, unfortunately is non-trivial, and really a pain if you want to automate builds.
Please clarify. What is non-trivial? Installing the SDK? I know, but we said that's out of scope. Using an SDK command prompt? It is, sort of, particularly if (like me) you use powershell. But again, not our issue. I assume setting the environment variable isn't an issue - you can do it for the session rather than globally, so even restrictive permissions aren't a problem. I appreciate you mightn't be intending this as criticism of the instructions, but many people do criticise in exactly this sort of way. Unix developers, in particular, who have limited Windows knowledge, find this level of instruction really frustrating to deal with. That's not a complaint - I have *huge* appreciation for non-Windows users who bother to make builds for Windows users - but it is an acknowledgement that often the audience for this sort of instruction are stumped by Microsoft's less than intuitive install processes... For context, installing mingw is just as messy, complicated and error prone (I speak from experience :-)) so it's unfair to complain that the above is a non-trivial pain. I know of no install option that's *less* straightforward than this (except of course for "install any version of Visual Studio 2010, even the free ones" - if you have access to those, use them!) For automation, why not use Appveyor? See https://packaging.python.org/en/latest/appveyor/ Unless you meant setting up a local build machine. If you want a simple "install a Python build environment" process, you could look at https://github.com/pfmoore/pybuild - I haven't used it in a while (as it's of no relevance to me, because I have VS2010) but it does work. I never publicised or distributed it, because I got too much pushback in terms of "but it doesn't work right on my system" (typically because the system in question usually *wasn't* a clean build of Windows) that I didn't have time or energy to address. But if it works for you, go for it. I'll push an addition to packaging.python.org, probably tomorrow. Paul
Concerning the claims that mingw is difficult: The mingwpy package is a sligthly modified mingw-w64 based gcc toolchain, that is in development. It is designed for simple use and for much better compatibility to the standard MSVC python builds. It should work out of the box, as long as the <Python>\Scripts folder is in the PATH. It is not 'officially' released and announced, due to the fact that some features are missing, the documentation has to be written and the build scripts for the toolchain are not (yet) published. Install a prerelease of mingwpy with pip: pip install -i https://pypi.anaconda.org/carlkl/simple mingwpy or with conda: (thanks to omnia-md) conda install -c https://conda.anaconda.org/omnia mingwpy and use it at usual with pip install or python setup.py You may need to configure %USERPROFILE%\pydistutils.cfg to use mingwpy if you have an MSVC compiler installed: [config] compiler=mingw32 [build] compiler=mingw32 [build_ext] compiler=mingw32 Or you install the latest portable winpython distribution https://winpython.github.io that contains the toolchain as well and works out of the box. Future releases of mingwpy will be deployed on Pypi. That has to be said: the main emphasis of the toolchain is building python binary extension (C, C++, GFORTRAN) on windows, not building python itself. Carl 2015-09-30 21:15 GMT+02:00 Paul Moore <p.f.moore@gmail.com>:
On 30 September 2015 at 16:57, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
1. Install "Windows SDK for Windows 7 and .NET Framework 4" (v7.1) 2. Work from an SDK command prompt (with the environment variables set, and the SDK on PATH). 3. Set DISTUTILS_USE_SDK=1 4. Done.
This, unfortunately is non-trivial, and really a pain if you want to automate builds.
Please clarify. What is non-trivial? Installing the SDK? I know, but we said that's out of scope. Using an SDK command prompt? It is, sort of, particularly if (like me) you use powershell. But again, not our issue. I assume setting the environment variable isn't an issue - you can do it for the session rather than globally, so even restrictive permissions aren't a problem.
I appreciate you mightn't be intending this as criticism of the instructions, but many people do criticise in exactly this sort of way. Unix developers, in particular, who have limited Windows knowledge, find this level of instruction really frustrating to deal with. That's not a complaint - I have *huge* appreciation for non-Windows users who bother to make builds for Windows users - but it is an acknowledgement that often the audience for this sort of instruction are stumped by Microsoft's less than intuitive install processes...
For context, installing mingw is just as messy, complicated and error prone (I speak from experience :-)) so it's unfair to complain that the above is a non-trivial pain. I know of no install option that's *less* straightforward than this (except of course for "install any version of Visual Studio 2010, even the free ones" - if you have access to those, use them!)
For automation, why not use Appveyor? See https://packaging.python.org/en/latest/appveyor/ Unless you meant setting up a local build machine. If you want a simple "install a Python build environment" process, you could look at https://github.com/pfmoore/pybuild - I haven't used it in a while (as it's of no relevance to me, because I have VS2010) but it does work. I never publicised or distributed it, because I got too much pushback in terms of "but it doesn't work right on my system" (typically because the system in question usually *wasn't* a clean build of Windows) that I didn't have time or energy to address. But if it works for you, go for it.
I'll push an addition to packaging.python.org, probably tomorrow.
Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com
Hi Carl, Looks promising. Any chance the effort would consider cross-compiling (from Linux) as a possible objective ? Best, Laurent On Wed, Sep 30, 2015, 3:58 PM Carl Kleffner <cmkleffner@gmail.com> wrote:
Concerning the claims that mingw is difficult:
The mingwpy package is a sligthly modified mingw-w64 based gcc toolchain, that is in development. It is designed for simple use and for much better compatibility to the standard MSVC python builds. It should work out of the box, as long as the <Python>\Scripts folder is in the PATH.
It is not 'officially' released and announced, due to the fact that some features are missing, the documentation has to be written and the build scripts for the toolchain are not (yet) published.
Install a prerelease of mingwpy with pip:
pip install -i https://pypi.anaconda.org/carlkl/simple mingwpy
or with conda: (thanks to omnia-md)
conda install -c https://conda.anaconda.org/omnia mingwpy
and use it at usual with pip install or python setup.py
You may need to configure %USERPROFILE%\pydistutils.cfg to use mingwpy if you have an MSVC compiler installed:
[config] compiler=mingw32 [build] compiler=mingw32 [build_ext] compiler=mingw32
Or you install the latest portable winpython distribution https://winpython.github.io that contains the toolchain as well and works out of the box.
Future releases of mingwpy will be deployed on Pypi.
That has to be said: the main emphasis of the toolchain is building python binary extension (C, C++, GFORTRAN) on windows, not building python itself.
Carl
2015-09-30 21:15 GMT+02:00 Paul Moore <p.f.moore@gmail.com>:
On 30 September 2015 at 16:57, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
1. Install "Windows SDK for Windows 7 and .NET Framework 4" (v7.1) 2. Work from an SDK command prompt (with the environment variables set, and the SDK on PATH). 3. Set DISTUTILS_USE_SDK=1 4. Done.
This, unfortunately is non-trivial, and really a pain if you want to automate builds.
Please clarify. What is non-trivial? Installing the SDK? I know, but we said that's out of scope. Using an SDK command prompt? It is, sort of, particularly if (like me) you use powershell. But again, not our issue. I assume setting the environment variable isn't an issue - you can do it for the session rather than globally, so even restrictive permissions aren't a problem.
I appreciate you mightn't be intending this as criticism of the instructions, but many people do criticise in exactly this sort of way. Unix developers, in particular, who have limited Windows knowledge, find this level of instruction really frustrating to deal with. That's not a complaint - I have *huge* appreciation for non-Windows users who bother to make builds for Windows users - but it is an acknowledgement that often the audience for this sort of instruction are stumped by Microsoft's less than intuitive install processes...
For context, installing mingw is just as messy, complicated and error prone (I speak from experience :-)) so it's unfair to complain that the above is a non-trivial pain. I know of no install option that's *less* straightforward than this (except of course for "install any version of Visual Studio 2010, even the free ones" - if you have access to those, use them!)
For automation, why not use Appveyor? See https://packaging.python.org/en/latest/appveyor/ Unless you meant setting up a local build machine. If you want a simple "install a Python build environment" process, you could look at https://github.com/pfmoore/pybuild - I haven't used it in a while (as it's of no relevance to me, because I have VS2010) but it does work. I never publicised or distributed it, because I got too much pushback in terms of "but it doesn't work right on my system" (typically because the system in question usually *wasn't* a clean build of Windows) that I didn't have time or energy to address. But if it works for you, go for it.
I'll push an addition to packaging.python.org, probably tomorrow.
Paul
_______________________________________________
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com
It should be possible with Wine. Am 30.09.2015 22:50 schrieb "Laurent Gautier" <lgautier@gmail.com>:
Hi Carl,
Looks promising.
Any chance the effort would consider cross-compiling (from Linux) as a possible objective ?
Best,
Laurent
On Wed, Sep 30, 2015, 3:58 PM Carl Kleffner <cmkleffner@gmail.com> wrote:
Concerning the claims that mingw is difficult:
The mingwpy package is a sligthly modified mingw-w64 based gcc toolchain, that is in development. It is designed for simple use and for much better compatibility to the standard MSVC python builds. It should work out of the box, as long as the <Python>\Scripts folder is in the PATH.
It is not 'officially' released and announced, due to the fact that some features are missing, the documentation has to be written and the build scripts for the toolchain are not (yet) published.
Install a prerelease of mingwpy with pip:
pip install -i https://pypi.anaconda.org/carlkl/simple mingwpy
or with conda: (thanks to omnia-md)
conda install -c https://conda.anaconda.org/omnia mingwpy
and use it at usual with pip install or python setup.py
You may need to configure %USERPROFILE%\pydistutils.cfg to use mingwpy if you have an MSVC compiler installed:
[config] compiler=mingw32 [build] compiler=mingw32 [build_ext] compiler=mingw32
Or you install the latest portable winpython distribution https://winpython.github.io that contains the toolchain as well and works out of the box.
Future releases of mingwpy will be deployed on Pypi.
That has to be said: the main emphasis of the toolchain is building python binary extension (C, C++, GFORTRAN) on windows, not building python itself.
Carl
2015-09-30 21:15 GMT+02:00 Paul Moore <p.f.moore@gmail.com>:
On 30 September 2015 at 16:57, Chris Barker - NOAA Federal <chris.barker@noaa.gov> wrote:
1. Install "Windows SDK for Windows 7 and .NET Framework 4" (v7.1) 2. Work from an SDK command prompt (with the environment variables set, and the SDK on PATH). 3. Set DISTUTILS_USE_SDK=1 4. Done.
This, unfortunately is non-trivial, and really a pain if you want to automate builds.
Please clarify. What is non-trivial? Installing the SDK? I know, but we said that's out of scope. Using an SDK command prompt? It is, sort of, particularly if (like me) you use powershell. But again, not our issue. I assume setting the environment variable isn't an issue - you can do it for the session rather than globally, so even restrictive permissions aren't a problem.
I appreciate you mightn't be intending this as criticism of the instructions, but many people do criticise in exactly this sort of way. Unix developers, in particular, who have limited Windows knowledge, find this level of instruction really frustrating to deal with. That's not a complaint - I have *huge* appreciation for non-Windows users who bother to make builds for Windows users - but it is an acknowledgement that often the audience for this sort of instruction are stumped by Microsoft's less than intuitive install processes...
For context, installing mingw is just as messy, complicated and error prone (I speak from experience :-)) so it's unfair to complain that the above is a non-trivial pain. I know of no install option that's *less* straightforward than this (except of course for "install any version of Visual Studio 2010, even the free ones" - if you have access to those, use them!)
For automation, why not use Appveyor? See https://packaging.python.org/en/latest/appveyor/ Unless you meant setting up a local build machine. If you want a simple "install a Python build environment" process, you could look at https://github.com/pfmoore/pybuild - I haven't used it in a while (as it's of no relevance to me, because I have VS2010) but it does work. I never publicised or distributed it, because I got too much pushback in terms of "but it doesn't work right on my system" (typically because the system in question usually *wasn't* a clean build of Windows) that I didn't have time or energy to address. But if it works for you, go for it.
I'll push an addition to packaging.python.org, probably tomorrow.
Paul
_______________________________________________
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
https://mail.python.org/mailman/options/python-dev/cmkleffner%40gmail.com
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/lgautier%40gmail.com
On Wed, Sep 30, 2015 at 3:57 PM, Carl Kleffner <cmkleffner@gmail.com> wrote:
Concerning the claims that mingw is difficult:
The mingwpy package is a sligthly modified mingw-w64 based gcc toolchain, that is in development. It is designed for simple use and for much better compatibility to the standard MSVC python builds. It should work out of the box, as long as the <Python>\Scripts folder is in the PATH.
Indeed, I tested it with 2.7.9 x86/x64, and wheres before I had to apply at least http://bugs.python.org/issue4709, and http://bugs.python.org/issue16472 and then generate the .a files to get it to work, this worked out of the box! (except of course still needing to change distutils.cfg). I do wonder though, which may be slightly off topic here, whether putting the mingw binary stub files (?) in python/Scripts, rather than directly having python/shared/mingwpy/bin be on the path, is the best approach? I suppose it's no worse than having the other stuff in python/scripts on the path (although with e.g. pip you could do the python -m pip trick). Are there guidelines about this (using python/scripts) for Windows? I didn't see anything mentioned when python -m pip install ... vs. pip install is discussed. Although I hope a 7z version of that mingw will be separately available for those that want to share a copy between versions of python, since the files in share/mingwpy is presumably the same between python versions for a specific bitness? Matt
On 30 September 2015 at 20:15, Paul Moore <p.f.moore@gmail.com> wrote:
I'll push an addition to packaging.python.org, probably tomorrow.
https://github.com/pypa/python-packaging-user-guide/pull/175 Unless there's a discussion on the PR, I'll probably commit it in a day or so. Paul
On Wed, Sep 30, 2015 at 12:15 PM, Paul Moore <p.f.moore@gmail.com> wrote:
This, unfortunately is non-trivial, and really a pain if you want to automate builds.
Please clarify.
First point -- that was intended to be a lament, not a criticism. And certainly not a criticism of anything Python devs are doing / have done. It's more difficult than it "should" be because of how MS distributes and configures the compiler and SDK, and that they are no longer distributing the same thing they used to distribute (i.e. VS2010 Express). I'm not at all sure that there is anything we can do to help.... But once upon a time, it really was as simple as: 1) Install this freely available compiler from MS 2) $ python setup.py build and that is the case now with py2.7 and (I assume) py3.5 Compared to that, setting environment variables, having to use a particular command prompt, etc. is non trivial. Part of the problem is that there are multiple audiences here: 1) experienced Windows devs that are writing and building their own packages: - not a big deal there, though it's still hard to figure out what is the "right" way to do it -- adding this to the packaging docs will help, I hope. 2) developers on other platforms that want to be able to build and distribute their package for Windows users. - this is a semi-big deal for them -- they need to learn a bit about how Windows works, and will find that annoying and frustrating because "why can't this be as easy as Linux?" (or really "the same as Linux") ;-) 3) users that want to be able to "pip install", and find that there is no binary wheel available, and they get the dreaded "can't find vcvarsall.bat" message. - these are the hardest people to support -- they may know very, very little about Windows command lines and environment variables etc. 4) users that are not particularly familiar with command lines or development tools, but need a little C extension -- Cython, in particular, makes it really easy to do that -- but then you need to buld the darn thing. distutils (setuptools) makes it possible to do that with very, very little understanding of build tools, but you have to get set up first... The whole process (including finding these instructions) is non-trivial for everyone of these use cases other than (1)[1]. What is non-trivial? Installing the SDK? I know, but
we said that's out of scope. Using an SDK command prompt? It is, sort of, particularly if (like me) you use powershell.
or gitBash :-)
But again, not our issue. I assume setting the environment variable isn't an issue - you can do it for the session rather than globally, so even restrictive permissions aren't a problem.
but both of these add steps to the "$python setup.py install" -- that really des make a difference to many users (see above) I but it
is an acknowledgement that often the audience for this sort of instruction are stumped by Microsoft's less than intuitive install processes...
yup !
For context, installing mingw is just as messy, complicated and error prone (I speak from experience :-))
God yes!
so it's unfair to complain that the above is a non-trivial pain. I know of no install option that's *less* straightforward than this (except of course for "install any version of Visual Studio 2010, even the free ones" - if you have access to those, use them!)
exactly! but that is the standard I'm comparing against. When it comes to usability, there really is no argument about whether something is "easy to use" -- As we've discussed, this comes up over an over again on StackOverflow, various mailing lists, assorted Blogs, -- there really is no debate about this being "easy". What can be done about it is a different question. My argument is that docs can help, and maybe once this is in the packaging docs, all those mailing list questions can be answered with a link to that.
For automation, why not use Appveyor? See https://packaging.python.org/en/latest/appveyor/
yup -- very cool, I've been meaning to get that set up for distributing some stuff.
Unless you meant setting up a local build machine.
Exactly -- you need that while developing, anyway. And at the moment, I have a non-sophisticated user-base that needs to be working with a frequently updated gitHub repo -- so needs to build themselves. (OK, if I had Appveyor doing updated binary packages every push, they wouldn't need to build themselves -- maybe someday)
If you want a simple "install a Python build environment" process, you could look at https://github.com/pfmoore/pybuild -
nice! I'll checdk that out. But I"m confused -- right in there, you write: "setting up a Windows build environment can be a significant challenge" then you ask me what is "non trivial" :-) I haven't used it in a while (as
it's of no relevance to me, because I have VS2010) but it does work. I never publicised or distributed it, because I got too much pushback in terms of "but it doesn't work right on my system" (typically because the system in question usually *wasn't* a clean build of Windows) that I didn't have time or energy to address. But if it works for you, go for it.
I'll give it a shot, thanks! I'll push an addition to packaging.python.org, probably tomorrow. I'll look at that, too -- thanks much for your attention to this. -CHB [1] Actually, I've had some of the larger struggles with folks that ARE experienced Windows devs -- they tend to think "I know how to use Visual Studio; I know how to build libs", and then they try to go off and do it by hand, and/or with their favorite verson of the compiler... I've had to pound it home again and again that you REALLY DO want to use distutils (or setuptools) and the version of VS that Python was compiled with. :-) -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 1 October 2015 at 18:18, Chris Barker <chris.barker@noaa.gov> wrote:
If you want a simple "install a Python build environment" process, you could look at https://github.com/pfmoore/pybuild -
nice! I'll checdk that out. But I"m confused -- right in there, you write:
"setting up a Windows build environment can be a significant challenge"
then you ask me what is "non trivial" :-)
You caught me out :-) Paul
I don't know that anyone disagrees with my point, but, less than an hour ago, this on the wxPython list: """ Microsoft no longer sells the compiler that's needed to build Python 3.4, and the needed compiler for Python 3.5 is free. """ To be fair, if you are trying to build wxWidgets, you may well want the whole IDE, but still... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On 25Sep2015 0824, Chris Barker - NOAA Federal wrote:
As I understand it, the MS VS2010 compiler is required (or at least best practice) for compiling Python extensions for the python.org Windows builds of py 3.4 and ?[1]
However, MS now makes it very hard (impossible?) to download VS2010 Express ( or Community, or whatever the free as in beer version is called).
This is the main motivation for moving to VS 2015 for Python 3.5. See https://docs.python.org/3.5/using/windows.html#compiling-python-on-windows (same link as [1] with the version changed) and also https://docs.python.org/3.5/whatsnew/3.5.html
I realize that this is not python-dev's responsibility, but if there is any way to either document where it can be found, or put a bit of pressure on MS to make it available, as they have for VS2008 and py2.7, that would be great.
I've applied that pressure already and it's not going to happen :( The best "solution" is to move to Py 3.5 asap. If you're writing an extension and are being careful, you can use a later compiler to build for 3.4 with Py_LIMITED_API set and a lot of care (did I mention you need to be careful? It's not just FILE* that will break...). Cheers, Steve
Sorry to bug this list, I didn't know where else to reach out to.
-Chris
[1] it's actually prefer hard to find out which compiler version is used for which python version. And has been for years. Would a patch to the docs, probably here:
https://docs.python.org/3.4/using/windows.html#compiling-python-on-windows
Be considered?
This is the main motivation for moving to VS 2015 for Python 3.5.
Yes,I've followed that work to the degree I understand it, and it looks like a great move. Thanks so much for your (continuing) efforts with this.
put a bit of
pressure on MS to make it available, as they have for VS2008 and py2.7, that would be great.
I've applied that pressure already and it's not going to happen :(
Darn. Makes no sense, but I work for almost as big a bureaucracy as MS, so I know how that goes... :-)
The best "solution" is to move to Py 3.5 asap.
Really not a solution at all, I'm afraid, and there are still growing pains with compiling for it as well (see Numpy/SciPy). But thanks. -Chris
participants (14)
-
Carl Kleffner -
Chris Barker -
Chris Barker - NOAA Federal
-
Erik Bray -
INADA Naoki -
Laurent Gautier -
Mark Lawrence -
Matthew Einhorn -
MRAB -
Nathaniel Smith -
Nathaniel Smith -
Oscar Benjamin -
Paul Moore -
Steve Dower