Mailman 3 March 2014 - Python-ideas

A python bridge between versions
by ian o March 2, 2014

March 2, 2014

The most quoted reason for staying with python 2 is that some required library is not available to support python 3. All it takes is one legacy library module in python 2 to keep an entire project in python 2. Many significant projects (e.g web2py) continue in python 2 for this reason. Even projects that produce code that can work under either python 2 or python 3, are themselves trapped into only using python 2 features. This is really holding back python! You cannot just import python2 code into the python 3 interpreter as it is not compatible. But could this be solved by a new approach which is to treat python 2 as another language and not directly import the code but handle python 2 code with an an extension module, which acts as bridge between versions. I mean, python three program can access C/C++ code (which is not python3), why can't the access python 2 code by treating it like C/C++? This would require an extension C module called from python 3. How to do this is well known. This extension C module would itself call the required module library(s) from python2. Calling python from C is also a well known technique. Ptyhon can call C. C can call python. So python3 call C which calls python2. So in python3 imput <python2-module> # this usually won't work because of language changes becomes import python2 <python2-module> = python2.import('<python2-module>') The resultant imported module would appear as a c++ extension to python3 and in effect be a wrapper for the python2 module using the python2 extension to execute the code This would mean programs using python2 imports have the overhead of both interpreters running in memory and calls across the python3/python2 boundary are 'wrapped' as c++ calls that then activate the required code within python 2. In reality quite a small overhead on modern computers. python2.sys.path (within the 'python2' module) or similar would be needed for the separate import space running inside the python2 boundary. Callbacks back across the python3/python2 boundary back into python3 are a further complication but several possible solutions exist to deliver this. Not many library modules require callback functions to the library module itself so this not the central issue. There is some work for the person coding the python 3 app in terms of a different syntax for legacy module imports and potentially setting python 2 environment values (like python2.sys.path), but not a lot and the python3 is the new program being worked on as opposed to the legacy library which can remain untouched. I think this type of approach could change the recommendation on 'python 2 or python 3' to a simple 'get the latest available to you and here is how to use legacy libraries if you have the need'. Thoughts or criticisms?

14 29

Re: [Python-ideas] Update the required C compiler for Windows to a supported version.
by Steve Dower March 2, 2014

March 2, 2014

Stephen J. Turnbull wrote: > Vernon D. Cole writes: > >> I cannot compile a Python extension module with any Microsoft compiler >> I can obtain. > > Your pain is understood, but it's not simple to address it. FWIW, I'm working on making the compiler easily obtainable. The VS 2008 link that was posted is unofficial, and could theoretically disappear at any time (I'm not in control of that), but the Windows SDK for Windows 7 and .NET 3.5 SP1 (http://www.microsoft.com/en-us/download/details.aspx?id=3138) should be around for as long as Windows 7 is supported. The correct compiler (VC9) is included in this SDK, but unfortunately does not install the vcvarsall.bat file that distutils expects. (Though it's pretty simple to add one that will switch on %1 and call the correct vcvars(86|64|...).bat.) The SDK needed for Python 3.3 and 3.4 (VC10) is even worse - there are many files missing. I'm hoping we'll be able to set up some sort of downloadable package/tool that will fix this. While we'd obviously love to move CPython onto our latest compilers, it's simply not possible (for good reason). Python 3.4 is presumably locked to VC10, but hopefully 3.5 will be able to use whichever version is current when that decision is made. > The basic problem is that the ABI changes. Therefore it's going to require > a complete new set of *all* C extensions for Windows, and the duplication > of download links for all those extensions from quite a few different vendors > is likely to confuse a lot of users. Specifically, the CRT changes. The CRT is an interesting mess of data structures that are exposed in header files, which means while you can have multiple CRTs loaded, they cannot touch each other's data structures at all or things will go bad/crash, and there's no nice way to set it up to avoid this (my colleague who currently owns MSVCRT suggested a not-very-nice way to do it, but I don't think it's going to be reliable enough). Python's stable ABI helps, but does not solve this problem. The file APIs are the worst culprits. The layout of FILE* objects can and do change between CRT versions, and file descriptors are simply indices into an array of these objects that is exposed through macros rather than function calls. As a result, you cannot mix either FILE pointers or file descriptors between CRTs. The only safe option is to build with the matching CRT, and for MSVCRT, this means with the matching compiler. It's unfortunate, and the responsible teams are well aware of the limitation, but it's history at this point, so we have no choice but to work with it. Cheers, Steve

4 3

Re: [Python-ideas] Update the required C compiler for Windows to a supported version.
by Steve Dower March 1, 2014

March 1, 2014

Not at my desk, so I can only get you started (didn't want to keep you waiting until Monday). It'll look something like this: IF %1 eq x86 CALL bin\vcvars32.bat %* IF %1 eq amd64 CALL bin\vcvars64.bat %* (similar for x86_amd64) The file should go at C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat If you want to redistribute something that will work for others, it's possible to monkey patch distutils from your setup.py. I can't remember the specifics right now, so I can't help with that until I'm back at work. I think that's the best long term solution for 2.7 packages that want to build on Windows. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Vernon D. Cole<mailto:vernondcole@gmail.com> Sent: ‎3/‎1/‎2014 2:24 To: Steve Dower<mailto:Steve.Dower@microsoft.com> Cc: Stephen J. Turnbull<mailto:stephen@xemacs.org>; python-ideas<mailto:python-ideas@python.org> Subject: Re: [Python-ideas] Update the required C compiler for Windows to a supported version. Sorry to resurrect this thread after so long a time, but I am starting to work on a project where I need this again... Any chance that I can get a copy (or a link to) " the vcvarsall.bat file that distutils expects. (Though it's pretty simple to add one" ? It's only simple to add one if you know what changes to make. -- VC On Wed, Nov 27, 2013 at 7:24 PM, Steve Dower <Steve.Dower(a)microsoft.com<mailto:Steve.Dower@microsoft.com>> wrote: Stephen J. Turnbull wrote: > Vernon D. Cole writes: > >> I cannot compile a Python extension module with any Microsoft compiler >> I can obtain. > > Your pain is understood, but it's not simple to address it. FWIW, I'm working on making the compiler easily obtainable. The VS 2008 link that was posted is unofficial, and could theoretically disappear at any time (I'm not in control of that), but the Windows SDK for Windows 7 and .NET 3.5 SP1 (http://www.microsoft.com/en-us/download/details.aspx?id=3138) should be around for as long as Windows 7 is supported. The correct compiler (VC9) is included in this SDK, but unfortunately does not install the vcvarsall.bat file that distutils expects. (Though it's pretty simple to add one that will switch on %1 and call the correct vcvars(86|64|...).bat.) The SDK needed for Python 3.3 and 3.4 (VC10) is even worse - there are many files missing. I'm hoping we'll be able to set up some sort of downloadable package/tool that will fix this. While we'd obviously love to move CPython onto our latest compilers, it's simply not possible (for good reason). Python 3.4 is presumably locked to VC10, but hopefully 3.5 will be able to use whichever version is current when that decision is made. > The basic problem is that the ABI changes. Therefore it's going to require > a complete new set of *all* C extensions for Windows, and the duplication > of download links for all those extensions from quite a few different vendors > is likely to confuse a lot of users. Specifically, the CRT changes. The CRT is an interesting mess of data structures that are exposed in header files, which means while you can have multiple CRTs loaded, they cannot touch each other's data structures at all or things will go bad/crash, and there's no nice way to set it up to avoid this (my colleague who currently owns MSVCRT suggested a not-very-nice way to do it, but I don't think it's going to be reliable enough). Python's stable ABI helps, but does not solve this problem. The file APIs are the worst culprits. The layout of FILE* objects can and do change between CRT versions, and file descriptors are simply indices into an array of these objects that is exposed through macros rather than function calls. As a result, you cannot mix either FILE pointers or file descriptors between CRTs. The only safe option is to build with the matching CRT, and for MSVCRT, this means with the matching compiler. It's unfortunate, and the responsible teams are well aware of the limitation, but it's history at this point, so we have no choice but to work with it. Cheers, Steve

1 0