windows 2.5 build: use OpenSSL for hashlib [bug 1535502]
Whoever knows how the windows build process works and controls the python 2.5 windows release builds could you please make sure the hashlib module gets built + linked with OpenSSL rather than falling back to its much slower builtin implementations. http://sourceforge.net/tracker/index.php?func=detail&aid=1535502&group_id=5470&atid=105470 thanks, greg
Gregory P. Smith schrieb:
Whoever knows how the windows build process works and controls the python 2.5 windows release builds could you please make sure the hashlib module gets built + linked with OpenSSL rather than falling back to its much slower builtin implementations.
If the project files are changed in that direction, then my build will pick that up automatically. I can't promise to change the files myself. I'm somewhat worried about yet another size increase in pythonxy.dll (actually, I'm personally not worried, but I anticipate complaints about such a change). What should happen to the existing sha* modules? I believe that the performance of the OpenSSL routines depends on the way OpenSSL was built, e.g. whether the assembler implementations are used or not. Somebody would have to check, but I doubt they are. So in short: I'm very doubtful that such a change can still be made, and if it is, that it will be "right". I'm accepting patches regardless. Regards, Martin
On Mon, Aug 07, 2006 at 03:16:22PM +0200, "Martin v. L?wis" wrote:
Gregory P. Smith schrieb:
Whoever knows how the windows build process works and controls the python 2.5 windows release builds could you please make sure the hashlib module gets built + linked with OpenSSL rather than falling back to its much slower builtin implementations.
If the project files are changed in that direction, then my build will pick that up automatically. I can't promise to change the files myself.
I'm somewhat worried about yet another size increase in pythonxy.dll (actually, I'm personally not worried, but I anticipate complaints about such a change).
What should happen to the existing sha* modules?
Actually, this change will either: (a) leave pythonxy.dll the same size. OR (b) *reduce* the size of pythonxy.dll. hashlib's OpenSSL implementation on windows comes in the form of a 300k _hashlib.pyd library. Build that and pythonxy.dll won't change. If you want to reduce the pythonxy.dll size you can remove _md5, _sha, _sha256 and _sha512 from the builtins list so long as _hashlib.pyd is built. There is OpenSSL library code duplication between the _hashlib (300k) and _ssl (650k) modules the way things are linked today (static). If you wanted absolute smallest distro size and code reuse we'd need to change things to use an OpenSSL.dll. I'm not proposing that (though it is a good idea).
I believe that the performance of the OpenSSL routines depends on the way OpenSSL was built, e.g. whether the assembler implementations are used or not. Somebody would have to check, but I doubt they are.
That'd be unfortunate as that negatively impacts the socket _ssl module as well. OpenSSL should always be built with the assembler implementations.
I'm nervous about this change being made at this stage of the release process. It seems to me to have a chance of causing breakages - admittedly a small chance, but one that's higher than I'd like. I'd also like to make sure that the PCBuild8 directory is updated at the same time - with the recent disappearance of the previous free MS compiler version, I think this will become more important over the 18 months or so of Python 2.5's life. Anthony
On Tue, Aug 08, 2006 at 03:25:46AM +1000, Anthony Baxter wrote:
I'm nervous about this change being made at this stage of the release process. It seems to me to have a chance of causing breakages - admittedly a small chance, but one that's higher than I'd like.
Sigh. Half the reason I did the hashlib work was to get much faster optimized versions of the hash algorithms into python. I'll be disappointed if that doesn't happen. hashlib passes its test suite with our without openssl. If I make the windows project file updates to simply build and include _hashlib.pyd in the windows installer what harm is that going to cause? IMHO the windows python 2.5 build as it is is missing a feature by not including this.
I'd also like to make sure that the PCBuild8 directory is updated at the same time - with the recent disappearance of the previous free MS compiler version, I think this will become more important over the 18 months or so of Python 2.5's life.
agreed. frustrated.. -greg
Gregory P. Smith schrieb:
Sigh. Half the reason I did the hashlib work was to get much faster optimized versions of the hash algorithms into python. I'll be disappointed if that doesn't happen.
Sad as it sounds, it appears you just did half of the work, then (omitting the Windows build process).
hashlib passes its test suite with our without openssl. If I make the windows project file updates to simply build and include _hashlib.pyd in the windows installer what harm is that going to cause?
None, if you do it correctly (where correctly includes AMD64 and IA64 builds), add text to PCbuild/readme.txt, and edit msi.py properly. That is, assuming hashlib then still works correctly on Windows (which is hard to tell).
IMHO the windows python 2.5 build as it is is missing a feature by not including this.
Wrong incantation :-) We are in feature freeze now, so adding a feature is a big no-no. You should have argued this is a bug <0.5 wink>. Regards, Martin
On Tue, Aug 08, 2006 at 02:23:02AM +0200, "Martin v. L?wis" wrote:
Gregory P. Smith schrieb:
Sigh. Half the reason I did the hashlib work was to get much faster optimized versions of the hash algorithms into python. I'll be disappointed if that doesn't happen.
Sad as it sounds, it appears you just did half of the work, then (omitting the Windows build process).
I had no access to a windows build environment at the time (as many python developers do not). Apparently I neglected to bribe someone who did to do it after I checked the module in. ;) So is it worth my time doing this in a hurry for 2.5 or do other people really just not care if python for windows uses a slow OpenSSL? Widely deployed popular applications use python for both large scale hashing and ssl communications. If no, can this go in 2.5.1? Its not an API change. -g
Gregory P. Smith schrieb:
So is it worth my time doing this in a hurry for 2.5 or do other people really just not care if python for windows uses a slow OpenSSL?
As I said: I would accept patches. If you arrange for a separate _hashlib.pyd file, most of my concerns would go away. So please do produce a patch, so we have something to review.
Widely deployed popular applications use python for both large scale hashing and ssl communications.
Yet, nobody has worried about performance in all these years to notice that the assembler code isn't being used. So it can't be that bad. For SSL specifically, the usage of hashing is minimal, as the actual communication uses symmetric encryption. Regards, Martin
On Tue, Aug 08, 2006 at 08:26:08AM +0200, "Martin v. L?wis" wrote:
Gregory P. Smith schrieb:
Widely deployed popular applications use python for both large scale hashing and ssl communications.
Yet, nobody has worried about performance in all these years to notice that the assembler code isn't being used. So it can't be that bad. For SSL specifically, the usage of hashing is minimal, as the actual communication uses symmetric encryption.
OpenSSL uses x86 asm to speed up other things besides hashes. For SSL sockets each new connection requires an RSA or DSA public key operation for the symmetric key exchange. Huge speedup there: 2ghz pentium-m with C ssl: sign verify sign/s verify/s rsa 1024 bits 0.007576s 0.000368s 132.0 2715.4 sign verify sign/s verify/s dsa 1024 bits 0.003655s 0.004409s 273.6 226.8 2ghz pentium-m with x86 asm ssl: sign verify sign/s verify/s rsa 1024 bits 0.003410s 0.000171s 293.3 5843.5 sign verify sign/s verify/s dsa 1024 bits 0.001632s 0.001987s 612.9 503.4 [data comes from running "out32\openssl speed" in openssl 0.9.8b]
I have supplied a patch that does everything needed to both make the windows build process build OpenSSL with x86 assembly optimizations on Win32 and to build the _hashlib.pyd module. It works for me. The only thing the patch doesn't do is add _hashlib.pyd to the .msi windows installer because I want to go to bed and haven't looked into how that works. http://sourceforge.net/tracker/index.php?func=detail&aid=1535502&group_id=5470&atid=105470 I know Anthony suggested "no" earlier (which is why I haven't committed it) but I really think that should be reconsidered. This doesn't change python at all, only fixes a build process. Ship with optimized code by default! help delay the heat death of the universe from cpu fans. :) either way, enjoy.
So is it worth my time doing this in a hurry for 2.5 or do other people really just not care if python for windows uses a slow OpenSSL?
Widely deployed popular applications use python for both large scale hashing and ssl communications.
If no, can this go in 2.5.1? Its not an API change.
-g
Gregory P. Smith schrieb:
hashlib's OpenSSL implementation on windows comes in the form of a 300k _hashlib.pyd library.
What do you mean by "comes"? I can't find any _hashlib.vcproj file inside the PCbuild directory.
I believe that the performance of the OpenSSL routines depends on the way OpenSSL was built, e.g. whether the assembler implementations are used or not. Somebody would have to check, but I doubt they are.
That'd be unfortunate as that negatively impacts the socket _ssl module as well. OpenSSL should always be built with the assembler implementations.
But Visual Studio doesn't ship with an assembler! So how could I build it? Regards, Martin
On Tue, Aug 08, 2006 at 01:46:13AM +0200, "Martin v. L?wis" wrote:
Gregory P. Smith schrieb:
hashlib's OpenSSL implementation on windows comes in the form of a 300k _hashlib.pyd library.
What do you mean by "comes"? I can't find any _hashlib.vcproj file inside the PCbuild directory.
I'll see about creating one later today when I'm reunited with my laptop.
I believe that the performance of the OpenSSL routines depends on the way OpenSSL was built, e.g. whether the assembler implementations are used or not. Somebody would have to check, but I doubt they are.
That'd be unfortunate as that negatively impacts the socket _ssl module as well. OpenSSL should always be built with the assembler implementations.
But Visual Studio doesn't ship with an assembler! So how could I build it?
yes it does. Visual Studio comes with MASM (ml) and OpenSSL ships with build scripts to use it. See openssl's INSTALL.W32 file. Also, a free assembler (NASM) is available that OpenSSL is also capable of building with if for some reason you don't have masm installed. Looking into how the windows python build process works (honestly something i've not looked at since 2.0) it appears that PCbuild/build_ssl.py handles the compilation of OpenSSL.. I haven't tested this yet (I'll try it tonight) but I believe this patch is all thats needed to enable the openssl assembly build: --- build_ssl.py (revision 51136) +++ build_ssl.py (working copy) @@ -98,6 +98,8 @@ if not cmd: break if cmd.strip()[:5].lower() == "nmake": continue + if re.match("set OPTS=no-asm", cmd): + continue temp_bat.write(cmd) in_bat.close() temp_bat.close() Alternatively just modifying build_ssl.py to run "ms\do_masm.bat" before it runs "nmake -f ms\32.mak" should work. -g
participants (3)
-
"Martin v. Löwis" -
Anthony Baxter -
Gregory P. Smith