Re: [Python-Dev] Getting python-bz2 into 2.3
Earlier this month, Tim P. wrote:
Semi-unfortunately, the author of that [bzip2] has
no idea if it actually works on 95/98/ME/NT/XP
and in the docs for "3.8 Making a Windows DLL"
I haven't tried any of this stuff myself, but it all looks plausible.
That means it will require some real work to build and test this stuff on 6 flavors of Windows. Not a showstopper, but does raise the bar for getting into the PLabs Windows distro.
Sorry if I'm way off base here, but does the underlying bzip2 package have to be in a DLL, or can't that be built as a static library, which gets linked into the .pyd, which *is* a DLL? In either case, it doesn't seem like it would be very difficult to create whatever flavor library is needed. The code for bzip2 seems to be very portably written. The python-bz2 code, on the other hand, needed a little bit of tweaking to make it compile with Microsoft's compiler. I'll be happy to help in whatever way would be useful in dealing with the "raised bar," as the prospect of having Python support on all platforms for bz2 compression (and tarfiles) is very appealing. -- Bob Kline mailto:bkline@rksystems.com http://www.rksystems.com
[Bob Kline]
Earlier this month, Tim P. wrote:
Semi-unfortunately, the author of that [bzip2] has
no idea if it actually works on 95/98/ME/NT/XP
and in the docs for "3.8 Making a Windows DLL"
I haven't tried any of this stuff myself, but it all looks plausible.
That means it will require some real work to build and test this stuff on 6 flavors of Windows. Not a showstopper, but does raise the bar for getting into the PLabs Windows distro.
Sorry if I'm way off base here, but does the underlying bzip2 package have to be in a DLL, or can't that be built as a static library, which gets linked into the .pyd, which *is* a DLL? In either case, it doesn't seem like it would be very difficult to create whatever flavor library is needed. The code for bzip2 seems to be very portably written.
The python-bz2 code, on the other hand, needed a little bit of tweaking to make it compile with Microsoft's compiler.
I'll be happy to help in whatever way would be useful in dealing with the "raised bar," as the prospect of having Python support on all platforms for bz2 compression (and tarfiles) is very appealing.
This work has already been completed; current CVS Python has full bz2 support for Windows.
On Fri, 29 Nov 2002, Tim Peters wrote:
I'll be happy to help in whatever way would be useful in dealing with the "raised bar," as the prospect of having Python support on all platforms for bz2 compression (and tarfiles) is very appealing.
This work has already been completed; current CVS Python has full bz2 support for Windows.
Excellent! Can't ask for better turnaround than that. Thanks! -- Bob Kline mailto:bkline@rksystems.com http://www.rksystems.com
Earlier this month, Tim P. wrote:
Semi-unfortunately, the author of that [bzip2] has
no idea if it actually works on 95/98/ME/NT/XP
and in the docs for "3.8 Making a Windows DLL"
I haven't tried any of this stuff myself, but it all looks plausible.
That means it will require some real work to build and test this stuff on 6 flavors of Windows. Not a showstopper, but does raise the bar for getting into the PLabs Windows distro.
Tim & I have different beliefs here. Tim believes that if something works on one flavor of Windows, that says nonthing about the other flavors. I believe that when it works on one flavor, you can assume that it works on all, unless proof to the contrary is shown. I'm sure we both have experience to back this up. :-)
Sorry if I'm way off base here, but does the underlying bzip2 package have to be in a DLL, or can't that be built as a static library, which gets linked into the .pyd, which *is* a DLL? In either case, it doesn't seem like it would be very difficult to create whatever flavor library is needed. The code for bzip2 seems to be very portably written.
The python-bz2 code, on the other hand, needed a little bit of tweaking to make it compile with Microsoft's compiler.
I'll be happy to help in whatever way would be useful in dealing with the "raised bar," as the prospect of having Python support on all platforms for bz2 compression (and tarfiles) is very appealing.
If you could get Python from CVS, build it with MSVC 6.0 for Windows (elaborate instructions are in PCBuild/readme.txt!!!), and see if the bz2 module works on all flavors of Windows to which you have access, that would be tremendously helpful IMO. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Fri, 29 Nov 2002, Guido van Rossum wrote:
I'll be happy to help in whatever way would be useful in dealing with the "raised bar," as the prospect of having Python support on all platforms for bz2 compression (and tarfiles) is very appealing.
If you could get Python from CVS, build it with MSVC 6.0 for Windows (elaborate instructions are in PCBuild/readme.txt!!!), and see if the bz2 module works on all flavors of Windows to which you have access, that would be tremendously helpful IMO.
If MSVC 7.0 isn't at least as good, I'll find a machine which still has 6.0 on it (or reinstall it on one of our development systems), but I pulled down the sources from CVS and built it with MSC++ 7 and the all 28 tests in the bz2 test suite pass on both Windows 2000 and Windows XP. Should I dig up a MSC6 system, or is this just as good? -- Bob Kline mailto:bkline@rksystems.com http://www.rksystems.com
On Fri, 29 Nov 2002, Bob Kline wrote:
On Fri, 29 Nov 2002, Guido van Rossum wrote:
I'll be happy to help in whatever way would be useful in dealing with the "raised bar," as the prospect of having Python support on all platforms for bz2 compression (and tarfiles) is very appealing.
If you could get Python from CVS, build it with MSVC 6.0 for Windows (elaborate instructions are in PCBuild/readme.txt!!!), and see if the bz2 module works on all flavors of Windows to which you have access, that would be tremendously helpful IMO.
If MSVC 7.0 isn't at least as good, I'll find a machine which still has 6.0 on it (or reinstall it on one of our development systems), but I pulled down the sources from CVS and built it with MSC++ 7 and the all 28 tests in the bz2 test suite pass on both Windows 2000 and Windows XP. Should I dig up a MSC6 system, or is this just as good?
I found a Windows 2000 Server that still had Visual Studio 6.0 on it. Pulled down the latest CVS sources, built Python (Release) with the bz2 module, and ran the test suite, which again passed. So far, therefore: W2K Pro + MSC7: passed all tests WXP Pro + MSC7: passed all tests W2K Server + MSC6: passed all tests -- Bob Kline mailto:bkline@rksystems.com http://www.rksystems.com
I'll be happy to help in whatever way would be useful in dealing with the "raised bar," as the prospect of having Python support on all platforms for bz2 compression (and tarfiles) is very appealing.
If you could get Python from CVS, build it with MSVC 6.0 for Windows (elaborate instructions are in PCBuild/readme.txt!!!), and see if the bz2 module works on all flavors of Windows to which you have access, that would be tremendously helpful IMO.
If MSVC 7.0 isn't at least as good, I'll find a machine which still has 6.0 on it (or reinstall it on one of our development systems), but I pulled down the sources from CVS and built it with MSC++ 7 and the all 28 tests in the bz2 test suite pass on both Windows 2000 and Windows XP. Should I dig up a MSC6 system, or is this just as good?
More testing is always welcome; if you could test-drive the instructions in PCbuild/readme.txt for Tcl/Tk and Sleepycat Berkeley DB on multiple platforms that would be super. I have understood (from Tim, again) that we can't upgrade to MSVC 7.0, because the runtime is incompatible in some places, and we don't want to break existing add-on distributions that are compiled with 6.0 -- not every open source developer can afford to upgrade. --Guido van Rossum (home page: http://www.python.org/~guido/)
On Sat, 30 Nov 2002, Guido van Rossum wrote:
More testing is always welcome; if you could test-drive the instructions in PCbuild/readme.txt for Tcl/Tk and Sleepycat Berkeley DB on multiple platforms that would be super.
The instructions for building Tcl and Tk worked as written with no problems (though I would have no idea where to begin with the XXX questions, such as "Should we compile with OPTS=threads?"). The project file for the _tkinter module also worked without any hitches, but I don't see a test suite (no surprise for a GUI-based module) and I have no clue about how to test it or even get some of my simple _tkinter program to run. I'm so far out of my depth here with respect to Tcl/Tk that I won't waste list bandwidth to pursue this any further. As for the Berkeley DB package, that too built without any difficulties. The instructions conclude: ============================== snip ============================== XXX We're actually linking against Release_static\libdb40s.lib. XXX This yields the following warnings: """ Compiling... _bsddb.c Linking... Creating library ./_bsddb.lib and object ./_bsddb.exp LINK : warning LNK4049: locally defined symbol "_malloc" imported LINK : warning LNK4049: locally defined symbol "_free" imported LINK : warning LNK4049: locally defined symbol "_fclose" imported LINK : warning LNK4049: locally defined symbol "_fopen" imported _bsddb.pyd - 0 error(s), 4 warning(s) """ XXX This isn't encouraging, but I don't know what to do about it. ============================== snip ============================== I did a little poking around, and found that the problem is that the static library against which the _bsddb package is built is compiled with the -MT option, which causes the multi-threaded runtime libraries to be statically linked into libdb40s.lib, hence the warnings. The warnings should probably be taken seriously, because I think they mean that we would end up with (for example) two sets of static heap control information. The instructions in PCBuild/readme.txt were actually a little confusing, because right before the part I just quoted above tells us to "[c]hoose configuration "db_buildall - Win32 Release", and build db_buildall.exe." That version uses the -MD compilation option, which uses the DLL version of the runtime libraries (which is what we want), but it creates a DLL for libdb40 instead of a library which can be statically linked into _bsddb.pyd. So it looks as there are two ways to eliminate the warnings above: 1. Use the DLL version of libdb40 and link to its import library. 2. Create a third configuration for the underlying Berkeley DB library, using the -MD compilation option, but building a statically linkable library instead of a DLL. I would have assumed that the second option was needed, because I assumed that Python wouldn't want to drag around lots of DLLs for the various packages that get installed, but then I noticed that the _tkinter package is doing just that, so I guess I'm not sure. If the second option is used, it should presumably be done with the cooperation of the folks behind the Berkeley DB library, so it wouldn't be necessary to recreate the third configuration by hand each time a new release of the library came out. Finally, I was reluctant to dig any further into this package because I read at the top of the instructions: "XXX The Sleepycat release we use will probably change before XXX 2.3a1." and I didn't know whether that referred to a version upgrade or a complete replacement of the underlying DB engine with some other library, which would make any further work throwaway. -- Bob Kline mailto:bkline@rksystems.com http://www.rksystems.com
[Bob Kline, tries building a more-or-less complete Windows Python]
The instructions for building Tcl and Tk worked as written with no problems (though I would have no idea where to begin with the XXX questions, such as "Should we compile with OPTS=threads?").
That's OK, neither do I <wink>. We didn't before (8.3.2), and changing anything is risky.
The project file for the _tkinter module also worked without any hitches, but I don't see a test suite (no surprise for a GUI-based module) and I have no clue about how to test it or even get some of my simple _tkinter program to run. I'm so far out of my depth here with respect to Tcl/Tk that I won't waste list bandwidth to pursue this any further.
I don't know how to test it either. I install Python after building it, then try Tkinter._test() from a DOS-box Python shell, and try a bunch of stuff in IDLE. That all looked fine. It was more worrisome to me that some of Tcl and Tk's own tests didn't pass, but I can't dig into that.
As for the Berkeley DB package, that too built without any difficulties. The instructions conclude:
============================== snip ============================== XXX We're actually linking against Release_static\libdb40s.lib. XXX This yields the following warnings: """ Compiling... _bsddb.c Linking... Creating library ./_bsddb.lib and object ./_bsddb.exp LINK : warning LNK4049: locally defined symbol "_malloc" imported LINK : warning LNK4049: locally defined symbol "_free" imported LINK : warning LNK4049: locally defined symbol "_fclose" imported LINK : warning LNK4049: locally defined symbol "_fopen" imported _bsddb.pyd - 0 error(s), 4 warning(s) """ XXX This isn't encouraging, but I don't know what to do about it. ============================== snip ==============================
I did a little poking around, and found that the problem is that the static library against which the _bsddb package is built is compiled with the -MT option, which causes the multi-threaded runtime libraries to be statically linked into libdb40s.lib, hence the warnings. The warnings should probably be taken seriously, because I think they mean that we would end up with (for example) two sets of static heap control information.
Right, but it's not clear that it matters. For example, it's unlikely there's a case where memory is allocated by something in libdb40s but freed outside it, or vice versa. Then two heaps are just two distinct heaps. I'm inclined to ignore it for now, since we're probably going to move to a current Sleepycat release anyway (but I don't know more about that -- Barry is straightening that out). It's quite unclear to me what good libdb40s.lib *is* if anyone using, e.g., malloc, can't link it in safely!
The instructions in PCBuild/readme.txt were actually a little confusing, because right before the part I just quoted above tells us to "[c]hoose configuration "db_buildall - Win32 Release", and build db_buildall.exe."
It just echoes the Sleepycat instructions here.
That version uses the -MD compilation option,
In part. It also triggers the building of subprojects that end up creating libdb40s.lib too.
which uses the DLL version of the runtime libraries (which is what we want), but it creates a DLL for libdb40 instead of a library which can be statically linked into _bsddb.pyd.
It creates both the DLL and the static library.
So it looks as there are two ways to eliminate the warnings above:
1. Use the DLL version of libdb40 and link to its import library.
This works but is clumsier: you can't run the Python tests then from the build directory, without first physically copying the Sleepycat DLL into PCbuild, and you *also* need to build-and-copy a distinct debug-mode Sleepycat DLL else you can't run the debug-build Python tests. In addition, the installer needs to be fiddled then to copy more files. Like the old 1.85 bsddb Windows port from Sam Rushing, and like the current bz2 and zlib modules, life is easier all around if 3rd-party code can be sucked directly into the Python wrapper .pyd file.
2. Create a third configuration for the underlying Berkeley DB library, using the -MD compilation option, but building a statically linkable library instead of a DLL.
I'm not sure how to do that, but don't want to muck with Sleepycat's build process regardless.
I would have assumed that the second option was needed, because I assumed that Python wouldn't want to drag around lots of DLLs for the various packages that get installed, but then I noticed that the _tkinter package is doing just that, so I guess I'm not sure.
It doesn't want to, but Tcl/Tk are big and do their own kinds of linking tricks.
If the second option is used, it should presumably be done with the cooperation of the folks behind the Berkeley DB library, so it wouldn't be necessary to recreate the third configuration by hand each time a new release of the library came out.
Finally, I was reluctant to dig any further into this package because I read at the top of the instructions:
"XXX The Sleepycat release we use will probably change before XXX 2.3a1."
and I didn't know whether that referred to a version upgrade or a complete replacement of the underlying DB engine with some other library, which would make any further work throwaway.
It refers to the current Sleepycat release; if you followed the build instructions closely <wink>, they started by pointing you to a page of obsolete releases. For some reason Barry understands, we can't yet use the current release (I think the bsddb C API changed, and our wrapper needs to change accordingly). Since I'm having a hard time (as above) imagining what good libdb40s.lib is in its current state, I'm simply hoping that Sleepycat fixed the linking glitches in their current release. Thanks for the effort, Bob! It helps.
[Guido]
Tim & I have different beliefs here. Tim believes that if something works on one flavor of Windows, that says nonthing about the other flavors. I believe that when it works on one flavor, you can assume that it works on all, unless proof to the contrary is shown. I'm sure we both have experience to back this up. :-)
It depends on whether "something" relies on behavior that's more due to the OS than to C. It's really the same as whether something that works on one flavor of Unix works on all other flavors of Unix. The latest Windows example was just last week, where someone's "clever" use of the new tempfile.NamedTemporaryFile in the Zope test suite worked fine on Win9X but died with an IOError on Win2K, and couldn't be fixed short of rewriting that part of the code from scratch. The bz2 code, and especially its test suite, had Unix-specific stuff at first, but I didn't see anything plausibly Windows-flavor-specific in it (or Unix-flavor-specific either).
... If you could get Python from CVS, build it with MSVC 6.0 for Windows (elaborate instructions are in PCBuild/readme.txt!!!), and see if the bz2 module works on all flavors of Windows to which you have access, that would be tremendously helpful IMO.
Yes it would. You can't know for sure until it's tried. Don't forget the French and German variants of Windows too <wink>.
participants (4)
-
Bob Kline -
Guido van Rossum -
Tim Peters -
Tim Peters