Unable to build regex module against Python 3.5 32-bit

As the subject says, I've been unable to build the regex module against Python 3.5b1 for 32-bit. MingGW says: skipping incompatible .../libpython35.a when searching for -lpython35 It builds without a problem against Python 3.5 for 64-bit. Any ideas? Should I just wait until beta 2?

Try building the module with -m32. The error message basically means: "../libpython35.a is 32-bit, but what you're building is 64-bit." Gotta love ld! On May 25, 2015 3:06:01 PM CDT, MRAB <python@mrabarnett.plus.com> wrote:
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

On 25 May 2015 at 21:06, MRAB <python@mrabarnett.plus.com> wrote:
MinGW is (and always has been) only marginally supported, unfortunately. I'd rather it didn't break totally for 3.5, but I am anticipating some difficulties (there have been a lot of compiler-related changes with 3.5). Could you raise a bug, including details of precisely how you tried to build the module (presumably https://pypi.python.org/pypi/regex/2015.05.10) and assign it to me? I'll try to take a look and reproduce the issue. With luck, it may be as simple as the wrong version of libpython35.a being picked up somewhere. Just to check the obvious - you *are* using 32-bit Python 3.5b1 and a 32-bit Mingw to build the 32-bit version, and 64-bit Python 3.5b1 and a 64-bit Mingw to build the 64-bit one? (I.e., two installations of Python and two of Mingw) Paul

On 2015-05-25 22:59, Paul Moore wrote:
I'm not sure what happened, but I'm now getting this for Python 3.5 (32-bit): C:\Python35(32-bit)\libs/libpython35.a(dsxbs01290.o):(.idata$7+0x0): undefined reference to `_head_C__build_cpython_PCBuild_win32_libpython35_a' C:\Python35(32-bit)\libs/libpython35.a(dsxbs00283.o):(.idata$7+0x0): undefined reference to `_head_C__build_cpython_PCBuild_win32_libpython35_a' C:\Python35(32-bit)\libs/libpython35.a(dsxbs00291.o):(.idata$7+0x0): undefined reference to `_head_C__build_cpython_PCBuild_win32_libpython35_a' C:\Python35(32-bit)\libs/libpython35.a(dsxbs00273.o):(.idata$7+0x0): undefined reference to `_head_C__build_cpython_PCBuild_win32_libpython35_a' C:\Python35(32-bit)\libs/libpython35.a(dsxbs00255.o):(.idata$7+0x0): undefined reference to `_head_C__build_cpython_PCBuild_win32_libpython35_a' C:\Python35(32-bit)\libs/libpython35.a(dsxbs01280.o):(.idata$7+0x0): more undefined references to `_head_C__build_cpython_PCBuild_win32_libpython35_a' follow collect2: ld returned 1 exit status All other builds, from Python 2.5 to Python 3.4, both 32-bit and 64-bit, and also Python 3.5 (64-bit), work. The 32-bit Python says it's 32-bit and the 64-bit Python says it's 64-bit. ---8<--- C: rem Compile for Python 3.5 (64-bit) [works] cd C:\MinGW64\bin "C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35\include" -c "D:\projects\mrab-regex\regex_3\regex\_regex_unicode.c" -o "D:\projects\mrab-regex\regex_3\Release(3.5)\_regex_unicode.o" "C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35\include" -c "D:\projects\mrab-regex\regex_3\regex\_regex.c" -o "D:\projects\mrab-regex\regex_3\Release(3.5)\_regex.o" "C:\MinGW64\bin\gcc.exe" -m64 -shared -s "D:\projects\mrab-regex\regex_3\Release(3.5)\_regex_unicode.o" "D:\projects\mrab-regex\regex_3\Release(3.5)\_regex.o" -L"C:\Python35\libs" -lpython35 -o "D:\projects\mrab-regex\regex_3\Release(3.5)\_regex.pyd" rem Compile for Python 3.5 (32-bit) [fails] cd C:\MinGW\bin "C:\MinGW\bin\gcc.exe" -mdll -m32 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35(32-bit)\include" -c "D:\projects\mrab-regex\regex_3\regex\_regex_unicode.c" -o "D:\projects\mrab-regex\regex_3\Release(3.5)(32-bit)\_regex_unicode.o" "C:\MinGW\bin\gcc.exe" -mdll -m32 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35(32-bit)\include" -c "D:\projects\mrab-regex\regex_3\regex\_regex.c" -o "D:\projects\mrab-regex\regex_3\Release(3.5)(32-bit)\_regex.o" "C:\MinGW\bin\gcc.exe" -m32 -shared -s "D:\projects\mrab-regex\regex_3\Release(3.5)(32-bit)\_regex_unicode.o" "D:\projects\mrab-regex\regex_3\Release(3.5)(32-bit)\_regex.o" -L"C:\Python35(32-bit)\libs" -lpython35 -o "D:\projects\mrab-regex\regex_3\Release(3.5)(32-bit)\_regex.pyd" ---8<---

On 26 May 2015 at 00:34, MRAB <python@mrabarnett.plus.com> wrote:
Do you get the same failure when using distutils to build the extension? Paul PS This discussion should probably be moved to bugs.python.org, as I mentioned...

On 26 May 2015 at 07:49, Paul Moore <p.f.moore@gmail.com> wrote:
Do you get the same failure when using distutils to build the extension?
Hmm, I just checked and it seems that only Python 3.5 ships libpythonXY.a by default - 3.4 and earlier (at least on my machine) don't have it. Presumably you generated it yourself for the previous versions? So I wonder if this is a difference between how you built your version previously and how the shipped version is now built. The code to build the shipped version is in Tools\msi\dev\dev.wixproj. It runs gendef - "$(BuildPath)$(PyDllName).dll" > "$(IntermediateOutputPath)mingwlib.def" dlltool --dllname $(PyDllName).dll --def "$(IntermediateOutputPath)mingwlib.def" --output-lib "$(BuildPath)lib$(PyDllName).a" -m $(_GenDefPlatform) which looks OK to me (_GenDefPlatform is i386 on 32-bit and i386:x86-64 on 64-bit). Paul

The builds I am responsible for include it because someone reported an issue and was persistent and helpful enough that I fixed it for them. That said, until MinGW agrees on a stable branch/version/fork, there seems to be a good chance that the shipped lib won't work for some people. If this is what's happened here, I see it as a good enough reason to stop shipping the lib and to add instructions on generating it instead (the gendef/dlltool dance). Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Paul Moore<mailto:p.f.moore@gmail.com> Sent: 5/26/2015 2:50 To: MRAB<mailto:python@mrabarnett.plus.com> Cc: Python-Dev<mailto:python-dev@python.org> Subject: Re: [Python-Dev] Unable to build regex module against Python 3.5 32-bit On 26 May 2015 at 07:49, Paul Moore <p.f.moore@gmail.com> wrote:
Do you get the same failure when using distutils to build the extension?
Hmm, I just checked and it seems that only Python 3.5 ships libpythonXY.a by default - 3.4 and earlier (at least on my machine) don't have it. Presumably you generated it yourself for the previous versions? So I wonder if this is a difference between how you built your version previously and how the shipped version is now built. The code to build the shipped version is in Tools\msi\dev\dev.wixproj. It runs gendef - "$(BuildPath)$(PyDllName).dll" > "$(IntermediateOutputPath)mingwlib.def" dlltool --dllname $(PyDllName).dll --def "$(IntermediateOutputPath)mingwlib.def" --output-lib "$(BuildPath)lib$(PyDllName).a" -m $(_GenDefPlatform) which looks OK to me (_GenDefPlatform is i386 on 32-bit and i386:x86-64 on 64-bit). Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...

On 26 May 2015 at 13:55, Steve Dower <Steve.Dower@microsoft.com> wrote:
Agreed. If shipping it helps, then great. If it's going to cause bug reports, let's go back to the status quo of not having it. The instructions for generating it were in the old distutils docs, now removed in the cleanup / redirection to packaging.python.org. I'm inclined to just leave it undocumented - the people who need it know how to do it or can find it, whereas documenting the process implies a level of support that we're not yet really able to provide. Let's wait to see what the OP comes back with before making a final decision. I see a few questions: 1. Does using distutils work (as opposed to the quoted manual compile steps)? 2. Does using whatever process he used in the past to generate libpythonXY.a result in a working version? https://docs.python.org/2.7/install/index.html#gnu-c-cygwin-mingw suggests using pexports rather than gendef. I don't know if that could produce a different result, but I can't see how... It also implicitly assumes you're using dlltool from the toolchain you'll be building with rather than using -m. Again, I can't see why that would affect the results. Paul.

to be a good chance that the shipped lib won't work for some
On 2015-05-26 14:24, Paul Moore wrote: there seems people. If this
It's been so long since the last Python release that I didn't remember that I had to make the lib file. I made libpython35.a like I did for the others and it's all working now. All the tests pass. :-)

On 26 May 2015 at 18:23, MRAB <python@mrabarnett.plus.com> wrote:
I made libpython35.a like I did for the others and it's all working now. All the tests pass. :-)
Cool. How did you make libpython35.a? Sounds like there is some difference in the process being used for the shipped version. Paul

On 2015-05-26 18:27, Paul Moore wrote:
For making the .def files, I used: C:\MinGW64\x86_64-w64-mingw32\bin\gendef.exe MinGW didn't contain gendef.exe! For making the .a files, I used: C:\MinGW64\bin\dlltool.exe for the 64-bit builds and: C:\MinGW\bin\dlltool.exe for the 32-bit builds. They all worked. When I tried: C:\MinGW64\bin\dlltool.exe or: C:\MinGW64\x86_64-w64-mingw32\bin\dlltool.exe for the 32-bit builds, they wouldn't link.

On 27 May 2015 at 03:02, MRAB <python@mrabarnett.plus.com> wrote:
Was that with "-m i386"? If so, then I suspect that's the issue. Steve, did you use 64-bit mingw to build the .a files? Assuming so, I guess we either stop shipping libpythonXY.a, or the instructions for building the Windows release need to clearly state that you want a 32-bit mingw on PATH for the 32-bit builds, and a 64-bit mingw on PATH for the 64-bit builds (which sounds messy and error-prone :-() I'd be inclined to call this a mingw bug. However, I don't have the first clue where to report it. Paul

On 26 May 2015 23:25, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 26 May 2015 at 13:55, Steve Dower <Steve.Dower@microsoft.com> wrote:
The builds I am responsible for include it because someone reported an
issue this
The old distutils docs aren't gone, the top level links just moved to the distutils package docs: https://docs.python.org/3/library/distutils.html I kept them (with the same deep link URLs) because I know there's stuff in there that isn't currently documented anywhere else. I moved them to a more obscure location because there's also stuff in there that's thoroughly outdated, and it's a non-trivial task to figure out which is which and move the still useful stuff to a more appropriate home :) Cheers, Nick.

On 27 May 2015 at 09:10, Nick Coghlan <ncoghlan@gmail.com> wrote:
Thanks. Your plan worked perfectly, because I never knew they were there :-) https://docs.python.org/3/install/index.html#older-versions-of-python-and-mi... implies that the libpythonXY.a files are only needed in older versions of Python/mingw. I don't know how true that is, although I do know that mingw should be able to link directly to a DLL without needing a lib file. It would be interesting to know if MRAB's build process can use the DLL, rather than requiring a lib file (or for that matter if distutils works without the lib file!) Paul

On 2015-05-27 09:25, Paul Moore wrote:
Steve Dower's post has prompted me to look again at building the regex module for Python 3.5, 32-bit and 64-bit, using just Mingw64 and linking against python32.dll. It works! Earlier versions of Python, however, including Python 2.7, still seem to want libpython??.a.

On 2015-06-05 01:37, MRAB wrote:
For reference, here's how I can build the regex module on Windows 8.1, 64-bit, using only MinGW64. For Python 3.5, I can link against "python35.dll", but for earlier versions, including Python 2.7, I need "libpython.a". I have built regex module for all of the 16 supported versions of Python (2.5-2.7, 3.1-3.5, 64-bit and 32-bit) and they have all passed the tests. rem For Python 3.5, 64-bit. rem Can link against the Python DLL. rem Compile "C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35-64\include" -c "D:\mrab-regex\source\_regex_unicode.c" -o "D:\mrab-regex\release\3.5-64\_regex_unicode.o" "C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35-64\include" -c "D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.5-64\_regex.o" rem Link "C:\MinGW64\bin\gcc.exe" -m64 -shared -s "D:\mrab-regex\release\3.5-64\_regex_unicode.o" "D:\mrab-regex\release\3.5-64\_regex.o" -L"C:\Python35" -lpython35 -o "D:\mrab-regex\release\3.5-64\_regex.pyd" rem For Python 3.5, 32-bit. rem Can link against the Python DLL. rem Compile "C:\MinGW64\bin\gcc.exe" -mdll -m32 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35-32\include" -c "D:\mrab-regex\source\_regex_unicode.c" -o "D:\mrab-regex\release\3.5-32\_regex_unicode.o" "C:\MinGW64\bin\gcc.exe" -mdll -m32 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35-32\include" -c "D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.5-32\_regex.o" rem Link "C:\MinGW64\bin\gcc.exe" -m32 -shared -s "D:\mrab-regex\release\3.5-32\_regex_unicode.o" "D:\mrab-regex\release\3.5-32\_regex.o" -L"C:\Python35-32" -lpython35 -o "D:\mrab-regex\release\3.5-32\_regex.pyd" rem For Python 3.4, 64-bit. rem Need to link against the Python .a file. rem Make libpython34.a "C:\MinGW64\x86_64-w64-mingw32\bin\gendef.exe" - "C:\Windows\System32\python34.dll" >"C:\Python34-64\libs\libpython34.def" "C:\MinGW64\bin\dlltool.exe" --dllname python34.dll --def "C:\Python34-64\libs\libpython34.def" --output-lib "C:\Python34-64\libs\libpython34.a" rem Compile "C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python34-64\include" -c "D:\mrab-regex\source\_regex_unicode.c" -o "D:\mrab-regex\release\3.4-64\_regex_unicode.o" rem Link "C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python34-64\include" -c "D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.4-64\_regex.o" "C:\MinGW64\bin\gcc.exe" -m64 -shared -s "D:\mrab-regex\release\3.4-64\_regex_unicode.o" "D:\mrab-regex\release\3.4-64\_regex.o" -L"C:\Python34-64\libs" -lpython34 -o "D:\mrab-regex\release\3.4-64\_regex.pyd" rem For Python 3.4, 32-bit. rem Need to link against the Python .a file. rem Make libpython34.a "C:\MinGW64\x86_64-w64-mingw32\bin\gendef.exe" - "C:\Windows\SysWOW64\python34.dll" >"C:\Python34-32\libs\libpython34.def" "C:\MinGW64\x86_64-w64-mingw32\bin\dlltool.exe" --as-flags=--32 -m i386 --dllname python34.dll --def "C:\Python34-32\libs\libpython34.def" --output-lib "C:\Python34-32\libs\libpython34.a" rem Compile "C:\MinGW64\bin\gcc.exe" -mdll -m32 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python34-32\include" -c "D:\mrab-regex\source\_regex_unicode.c" -o "D:\mrab-regex\release\3.4-32\_regex_unicode.o" "C:\MinGW64\bin\gcc.exe" -mdll -m32 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python34-32\include" -c "D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.4-32\_regex.o" rem Link "C:\MinGW64\bin\gcc.exe" -m32 -shared -s "D:\mrab-regex\release\3.4-32\_regex_unicode.o" "D:\mrab-regex\release\3.4-32\_regex.o" -L"C:\Python34-32\libs" -lpython34 -o "D:\mrab-regex\release\3.4-32\_regex.pyd" rem For earlier versions of Python, follow the pattern of Python 3.4.

On 5 June 2015 at 02:28, MRAB <python@mrabarnett.plus.com> wrote:
[...] Note that even if we were to ship appropriate libpython.a libraries, this would always be a totally unsupported way of building Python extensions using mingw. If you want to build extensions, whether using mingw or otherwise, you should use distutils/setuptools. To use mingw, there's the --compiler=mingw switch that allows you to choose your compiler. There are a number of open issues around distutils with mingw which may mean that currently this approach doesn't work, but getting the mingw-using community to help with fixing those issues is the *only* way we'll ever get to a point where mingw is a supported toolchain for Python. I appreciate that "just getting it to work" is a powerful motivator here, but the plethora of different ways that people use for building extensions with mingw is *precisely* why mingw has failed to remain a supported toolchain. I'd be interested to know how well using setup.py --compiler=mingw works for your project, and why you chose to go with the manual process. (Just as an aside, setup.py works prerfectly for me when building the regex package with MSVC). Paul

On 05/06/2015 01:37, MRAB wrote:
This http://bugs.python.org/issue24385 should interest you. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence

Try building the module with -m32. The error message basically means: "../libpython35.a is 32-bit, but what you're building is 64-bit." Gotta love ld! On May 25, 2015 3:06:01 PM CDT, MRAB <python@mrabarnett.plus.com> wrote:
-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

On 25 May 2015 at 21:06, MRAB <python@mrabarnett.plus.com> wrote:
MinGW is (and always has been) only marginally supported, unfortunately. I'd rather it didn't break totally for 3.5, but I am anticipating some difficulties (there have been a lot of compiler-related changes with 3.5). Could you raise a bug, including details of precisely how you tried to build the module (presumably https://pypi.python.org/pypi/regex/2015.05.10) and assign it to me? I'll try to take a look and reproduce the issue. With luck, it may be as simple as the wrong version of libpython35.a being picked up somewhere. Just to check the obvious - you *are* using 32-bit Python 3.5b1 and a 32-bit Mingw to build the 32-bit version, and 64-bit Python 3.5b1 and a 64-bit Mingw to build the 64-bit one? (I.e., two installations of Python and two of Mingw) Paul

On 2015-05-25 22:59, Paul Moore wrote:
I'm not sure what happened, but I'm now getting this for Python 3.5 (32-bit): C:\Python35(32-bit)\libs/libpython35.a(dsxbs01290.o):(.idata$7+0x0): undefined reference to `_head_C__build_cpython_PCBuild_win32_libpython35_a' C:\Python35(32-bit)\libs/libpython35.a(dsxbs00283.o):(.idata$7+0x0): undefined reference to `_head_C__build_cpython_PCBuild_win32_libpython35_a' C:\Python35(32-bit)\libs/libpython35.a(dsxbs00291.o):(.idata$7+0x0): undefined reference to `_head_C__build_cpython_PCBuild_win32_libpython35_a' C:\Python35(32-bit)\libs/libpython35.a(dsxbs00273.o):(.idata$7+0x0): undefined reference to `_head_C__build_cpython_PCBuild_win32_libpython35_a' C:\Python35(32-bit)\libs/libpython35.a(dsxbs00255.o):(.idata$7+0x0): undefined reference to `_head_C__build_cpython_PCBuild_win32_libpython35_a' C:\Python35(32-bit)\libs/libpython35.a(dsxbs01280.o):(.idata$7+0x0): more undefined references to `_head_C__build_cpython_PCBuild_win32_libpython35_a' follow collect2: ld returned 1 exit status All other builds, from Python 2.5 to Python 3.4, both 32-bit and 64-bit, and also Python 3.5 (64-bit), work. The 32-bit Python says it's 32-bit and the 64-bit Python says it's 64-bit. ---8<--- C: rem Compile for Python 3.5 (64-bit) [works] cd C:\MinGW64\bin "C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35\include" -c "D:\projects\mrab-regex\regex_3\regex\_regex_unicode.c" -o "D:\projects\mrab-regex\regex_3\Release(3.5)\_regex_unicode.o" "C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35\include" -c "D:\projects\mrab-regex\regex_3\regex\_regex.c" -o "D:\projects\mrab-regex\regex_3\Release(3.5)\_regex.o" "C:\MinGW64\bin\gcc.exe" -m64 -shared -s "D:\projects\mrab-regex\regex_3\Release(3.5)\_regex_unicode.o" "D:\projects\mrab-regex\regex_3\Release(3.5)\_regex.o" -L"C:\Python35\libs" -lpython35 -o "D:\projects\mrab-regex\regex_3\Release(3.5)\_regex.pyd" rem Compile for Python 3.5 (32-bit) [fails] cd C:\MinGW\bin "C:\MinGW\bin\gcc.exe" -mdll -m32 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35(32-bit)\include" -c "D:\projects\mrab-regex\regex_3\regex\_regex_unicode.c" -o "D:\projects\mrab-regex\regex_3\Release(3.5)(32-bit)\_regex_unicode.o" "C:\MinGW\bin\gcc.exe" -mdll -m32 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35(32-bit)\include" -c "D:\projects\mrab-regex\regex_3\regex\_regex.c" -o "D:\projects\mrab-regex\regex_3\Release(3.5)(32-bit)\_regex.o" "C:\MinGW\bin\gcc.exe" -m32 -shared -s "D:\projects\mrab-regex\regex_3\Release(3.5)(32-bit)\_regex_unicode.o" "D:\projects\mrab-regex\regex_3\Release(3.5)(32-bit)\_regex.o" -L"C:\Python35(32-bit)\libs" -lpython35 -o "D:\projects\mrab-regex\regex_3\Release(3.5)(32-bit)\_regex.pyd" ---8<---

On 26 May 2015 at 00:34, MRAB <python@mrabarnett.plus.com> wrote:
Do you get the same failure when using distutils to build the extension? Paul PS This discussion should probably be moved to bugs.python.org, as I mentioned...

On 26 May 2015 at 07:49, Paul Moore <p.f.moore@gmail.com> wrote:
Do you get the same failure when using distutils to build the extension?
Hmm, I just checked and it seems that only Python 3.5 ships libpythonXY.a by default - 3.4 and earlier (at least on my machine) don't have it. Presumably you generated it yourself for the previous versions? So I wonder if this is a difference between how you built your version previously and how the shipped version is now built. The code to build the shipped version is in Tools\msi\dev\dev.wixproj. It runs gendef - "$(BuildPath)$(PyDllName).dll" > "$(IntermediateOutputPath)mingwlib.def" dlltool --dllname $(PyDllName).dll --def "$(IntermediateOutputPath)mingwlib.def" --output-lib "$(BuildPath)lib$(PyDllName).a" -m $(_GenDefPlatform) which looks OK to me (_GenDefPlatform is i386 on 32-bit and i386:x86-64 on 64-bit). Paul

The builds I am responsible for include it because someone reported an issue and was persistent and helpful enough that I fixed it for them. That said, until MinGW agrees on a stable branch/version/fork, there seems to be a good chance that the shipped lib won't work for some people. If this is what's happened here, I see it as a good enough reason to stop shipping the lib and to add instructions on generating it instead (the gendef/dlltool dance). Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Paul Moore<mailto:p.f.moore@gmail.com> Sent: 5/26/2015 2:50 To: MRAB<mailto:python@mrabarnett.plus.com> Cc: Python-Dev<mailto:python-dev@python.org> Subject: Re: [Python-Dev] Unable to build regex module against Python 3.5 32-bit On 26 May 2015 at 07:49, Paul Moore <p.f.moore@gmail.com> wrote:
Do you get the same failure when using distutils to build the extension?
Hmm, I just checked and it seems that only Python 3.5 ships libpythonXY.a by default - 3.4 and earlier (at least on my machine) don't have it. Presumably you generated it yourself for the previous versions? So I wonder if this is a difference between how you built your version previously and how the shipped version is now built. The code to build the shipped version is in Tools\msi\dev\dev.wixproj. It runs gendef - "$(BuildPath)$(PyDllName).dll" > "$(IntermediateOutputPath)mingwlib.def" dlltool --dllname $(PyDllName).dll --def "$(IntermediateOutputPath)mingwlib.def" --output-lib "$(BuildPath)lib$(PyDllName).a" -m $(_GenDefPlatform) which looks OK to me (_GenDefPlatform is i386 on 32-bit and i386:x86-64 on 64-bit). Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...

On 26 May 2015 at 13:55, Steve Dower <Steve.Dower@microsoft.com> wrote:
Agreed. If shipping it helps, then great. If it's going to cause bug reports, let's go back to the status quo of not having it. The instructions for generating it were in the old distutils docs, now removed in the cleanup / redirection to packaging.python.org. I'm inclined to just leave it undocumented - the people who need it know how to do it or can find it, whereas documenting the process implies a level of support that we're not yet really able to provide. Let's wait to see what the OP comes back with before making a final decision. I see a few questions: 1. Does using distutils work (as opposed to the quoted manual compile steps)? 2. Does using whatever process he used in the past to generate libpythonXY.a result in a working version? https://docs.python.org/2.7/install/index.html#gnu-c-cygwin-mingw suggests using pexports rather than gendef. I don't know if that could produce a different result, but I can't see how... It also implicitly assumes you're using dlltool from the toolchain you'll be building with rather than using -m. Again, I can't see why that would affect the results. Paul.

to be a good chance that the shipped lib won't work for some
On 2015-05-26 14:24, Paul Moore wrote: there seems people. If this
It's been so long since the last Python release that I didn't remember that I had to make the lib file. I made libpython35.a like I did for the others and it's all working now. All the tests pass. :-)

On 26 May 2015 at 18:23, MRAB <python@mrabarnett.plus.com> wrote:
I made libpython35.a like I did for the others and it's all working now. All the tests pass. :-)
Cool. How did you make libpython35.a? Sounds like there is some difference in the process being used for the shipped version. Paul

On 2015-05-26 18:27, Paul Moore wrote:
For making the .def files, I used: C:\MinGW64\x86_64-w64-mingw32\bin\gendef.exe MinGW didn't contain gendef.exe! For making the .a files, I used: C:\MinGW64\bin\dlltool.exe for the 64-bit builds and: C:\MinGW\bin\dlltool.exe for the 32-bit builds. They all worked. When I tried: C:\MinGW64\bin\dlltool.exe or: C:\MinGW64\x86_64-w64-mingw32\bin\dlltool.exe for the 32-bit builds, they wouldn't link.

On 27 May 2015 at 03:02, MRAB <python@mrabarnett.plus.com> wrote:
Was that with "-m i386"? If so, then I suspect that's the issue. Steve, did you use 64-bit mingw to build the .a files? Assuming so, I guess we either stop shipping libpythonXY.a, or the instructions for building the Windows release need to clearly state that you want a 32-bit mingw on PATH for the 32-bit builds, and a 64-bit mingw on PATH for the 64-bit builds (which sounds messy and error-prone :-() I'd be inclined to call this a mingw bug. However, I don't have the first clue where to report it. Paul

On 26 May 2015 23:25, "Paul Moore" <p.f.moore@gmail.com> wrote:
On 26 May 2015 at 13:55, Steve Dower <Steve.Dower@microsoft.com> wrote:
The builds I am responsible for include it because someone reported an
issue this
The old distutils docs aren't gone, the top level links just moved to the distutils package docs: https://docs.python.org/3/library/distutils.html I kept them (with the same deep link URLs) because I know there's stuff in there that isn't currently documented anywhere else. I moved them to a more obscure location because there's also stuff in there that's thoroughly outdated, and it's a non-trivial task to figure out which is which and move the still useful stuff to a more appropriate home :) Cheers, Nick.

On 27 May 2015 at 09:10, Nick Coghlan <ncoghlan@gmail.com> wrote:
Thanks. Your plan worked perfectly, because I never knew they were there :-) https://docs.python.org/3/install/index.html#older-versions-of-python-and-mi... implies that the libpythonXY.a files are only needed in older versions of Python/mingw. I don't know how true that is, although I do know that mingw should be able to link directly to a DLL without needing a lib file. It would be interesting to know if MRAB's build process can use the DLL, rather than requiring a lib file (or for that matter if distutils works without the lib file!) Paul

On 2015-05-27 09:25, Paul Moore wrote:
Steve Dower's post has prompted me to look again at building the regex module for Python 3.5, 32-bit and 64-bit, using just Mingw64 and linking against python32.dll. It works! Earlier versions of Python, however, including Python 2.7, still seem to want libpython??.a.

On 2015-06-05 01:37, MRAB wrote:
For reference, here's how I can build the regex module on Windows 8.1, 64-bit, using only MinGW64. For Python 3.5, I can link against "python35.dll", but for earlier versions, including Python 2.7, I need "libpython.a". I have built regex module for all of the 16 supported versions of Python (2.5-2.7, 3.1-3.5, 64-bit and 32-bit) and they have all passed the tests. rem For Python 3.5, 64-bit. rem Can link against the Python DLL. rem Compile "C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35-64\include" -c "D:\mrab-regex\source\_regex_unicode.c" -o "D:\mrab-regex\release\3.5-64\_regex_unicode.o" "C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35-64\include" -c "D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.5-64\_regex.o" rem Link "C:\MinGW64\bin\gcc.exe" -m64 -shared -s "D:\mrab-regex\release\3.5-64\_regex_unicode.o" "D:\mrab-regex\release\3.5-64\_regex.o" -L"C:\Python35" -lpython35 -o "D:\mrab-regex\release\3.5-64\_regex.pyd" rem For Python 3.5, 32-bit. rem Can link against the Python DLL. rem Compile "C:\MinGW64\bin\gcc.exe" -mdll -m32 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35-32\include" -c "D:\mrab-regex\source\_regex_unicode.c" -o "D:\mrab-regex\release\3.5-32\_regex_unicode.o" "C:\MinGW64\bin\gcc.exe" -mdll -m32 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python35-32\include" -c "D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.5-32\_regex.o" rem Link "C:\MinGW64\bin\gcc.exe" -m32 -shared -s "D:\mrab-regex\release\3.5-32\_regex_unicode.o" "D:\mrab-regex\release\3.5-32\_regex.o" -L"C:\Python35-32" -lpython35 -o "D:\mrab-regex\release\3.5-32\_regex.pyd" rem For Python 3.4, 64-bit. rem Need to link against the Python .a file. rem Make libpython34.a "C:\MinGW64\x86_64-w64-mingw32\bin\gendef.exe" - "C:\Windows\System32\python34.dll" >"C:\Python34-64\libs\libpython34.def" "C:\MinGW64\bin\dlltool.exe" --dllname python34.dll --def "C:\Python34-64\libs\libpython34.def" --output-lib "C:\Python34-64\libs\libpython34.a" rem Compile "C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python34-64\include" -c "D:\mrab-regex\source\_regex_unicode.c" -o "D:\mrab-regex\release\3.4-64\_regex_unicode.o" rem Link "C:\MinGW64\bin\gcc.exe" -mdll -m64 -DMS_WIN64 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python34-64\include" -c "D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.4-64\_regex.o" "C:\MinGW64\bin\gcc.exe" -m64 -shared -s "D:\mrab-regex\release\3.4-64\_regex_unicode.o" "D:\mrab-regex\release\3.4-64\_regex.o" -L"C:\Python34-64\libs" -lpython34 -o "D:\mrab-regex\release\3.4-64\_regex.pyd" rem For Python 3.4, 32-bit. rem Need to link against the Python .a file. rem Make libpython34.a "C:\MinGW64\x86_64-w64-mingw32\bin\gendef.exe" - "C:\Windows\SysWOW64\python34.dll" >"C:\Python34-32\libs\libpython34.def" "C:\MinGW64\x86_64-w64-mingw32\bin\dlltool.exe" --as-flags=--32 -m i386 --dllname python34.dll --def "C:\Python34-32\libs\libpython34.def" --output-lib "C:\Python34-32\libs\libpython34.a" rem Compile "C:\MinGW64\bin\gcc.exe" -mdll -m32 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python34-32\include" -c "D:\mrab-regex\source\_regex_unicode.c" -o "D:\mrab-regex\release\3.4-32\_regex_unicode.o" "C:\MinGW64\bin\gcc.exe" -mdll -m32 -O2 -Wall -Wsign-compare -Wconversion -I"C:\Python34-32\include" -c "D:\mrab-regex\source\_regex.c" -o "D:\mrab-regex\release\3.4-32\_regex.o" rem Link "C:\MinGW64\bin\gcc.exe" -m32 -shared -s "D:\mrab-regex\release\3.4-32\_regex_unicode.o" "D:\mrab-regex\release\3.4-32\_regex.o" -L"C:\Python34-32\libs" -lpython34 -o "D:\mrab-regex\release\3.4-32\_regex.pyd" rem For earlier versions of Python, follow the pattern of Python 3.4.

On 5 June 2015 at 02:28, MRAB <python@mrabarnett.plus.com> wrote:
[...] Note that even if we were to ship appropriate libpython.a libraries, this would always be a totally unsupported way of building Python extensions using mingw. If you want to build extensions, whether using mingw or otherwise, you should use distutils/setuptools. To use mingw, there's the --compiler=mingw switch that allows you to choose your compiler. There are a number of open issues around distutils with mingw which may mean that currently this approach doesn't work, but getting the mingw-using community to help with fixing those issues is the *only* way we'll ever get to a point where mingw is a supported toolchain for Python. I appreciate that "just getting it to work" is a powerful motivator here, but the plethora of different ways that people use for building extensions with mingw is *precisely* why mingw has failed to remain a supported toolchain. I'd be interested to know how well using setup.py --compiler=mingw works for your project, and why you chose to go with the manual process. (Just as an aside, setup.py works prerfectly for me when building the regex package with MSVC). Paul

On 05/06/2015 01:37, MRAB wrote:
This http://bugs.python.org/issue24385 should interest you. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence
participants (6)
-
Mark Lawrence
-
MRAB
-
Nick Coghlan
-
Paul Moore
-
Ryan Gonzalez
-
Steve Dower