On Feb 10, 2005, at 9:15 PM, Donovan Baarda wrote:

On Tue, 2005-02-08 at 11:52 -0800, Gregory P. Smith wrote:

...

...

The md5.h/md5c.c files allow "copy and use", but no modification of the files. There are some alternative implementations, i.e. in glibc, openssl, so a replacement should be sage. Any other requirements when considering a replacement?

One thing to consider is "degree of difficulty" :-)

...

...

Matthias

I believe the "plan" for md5 and sha1 and such is to use the much faster openssl versions "in the future" (based on a long thread debating future interfaces to such things on python-dev last summer). That'll sidestep any tedious license issue and give a better implementation at the same time. i don't believe anyone has taken the time to make such a patch yet.

I wasn't around for that discussion. There are two viable replacements for the RSA implementation currently used;

libmd <http://www.penguin.cz/~mhi/libmd/> openssl <http://www.openssl.org/>. -- In the Linux world, openssl is starting to become ubiquitous, so not including it and statically or even dynamically linking against it is feasible. However, using Python in other lands will probably require something to be included.

Long term, I think openssl is the way to go. Short term, libmd is a painless replacement that gets around the licencing issues.

OpenSSL is also ubiquitous on Mac OS X (as a shared lib): Mac OS X 10.2.8 has OpenSSL 0.9.6i Feb 19 2003 Mac OS X 10.3.8 has OpenSSL 0.9.7b 10 Apr 2003 One possible alternative would be to bring in something like PyOpenSSL <http://pyopenssl.sourceforge.net/> and just rewrite the md5 (and sha?) extensions as Python modules that use that API. -bob

On Feb 10, 2005, at 9:50 PM, Donovan Baarda wrote:

On Thu, 2005-02-10 at 21:30 -0500, Bob Ippolito wrote:

...

On Feb 10, 2005, at 9:15 PM, Donovan Baarda wrote:

...

On Tue, 2005-02-08 at 11:52 -0800, Gregory P. Smith wrote: [...] One possible alternative would be to bring in something like PyOpenSSL <http://pyopenssl.sourceforge.net/> and just rewrite the md5 (and sha?) extensions as Python modules that use that API.

Only problem with this, is pyopenssl doesn't yet include any mdX or sha modules.

My bad, how about M2Crypto <http://sandbox.rulemaker.net/ngps/m2/> then? This one supports message digests and is more license compatible with Python to boot. -bob

On Fri, 2005-02-11 at 17:15 +1100, Donovan Baarda wrote: [...]

I think it would be cleaner and simpler to modify the existing md5module.c to use the openssl md5 layer API (this is just a search/replace to change the function names). The bigger problem is deciding what/how/whether to include the openssl md5 implementation sources so that win32 can use them.

Thinking about it, probably the best way is to include the libmd md5c.c modified to use the openssl API, and then use configure to check for and use openssl if it is available. That way win32 could use the provided md5c.c, and other platforms could use the faster openssl. -- Donovan Baarda <abo@minkirri.apana.org.au> http://minkirri.apana.org.au/~abo/

G'day again, From: "Gregory P. Smith" <greg@electricrain.com>

...

I think it would be cleaner and simpler to modify the existing md5module.c to use the openssl md5 layer API (this is just a search/replace to change the function names). The bigger problem is deciding what/how/whether to include the openssl md5 implementation sources so that win32 can use them.

yes, that is all i was suggesting.

win32 python is already linked against openssl for the socket module ssl support, having the md5 and sha1 modules depend on openssl should not cause a problem.

IANAL... I have too much common sense, so I won't argue licences :-) So is openssl already included in the Python sources, or is it just a dependency? I had a quick look and couldn't find it so it must be a dependency. Given that Python is already dependant on openssl, it makes sense to change md5sum to use it. I have a feeling that openssl internally uses md5, so this way we wont link against two different md5sum implementations. ---------------------------------------------------------------- Donovan Baarda http://minkirri.apana.org.au/~abo/ ----------------------------------------------------------------

G'day, From: "Bob Ippolito" <bob@redivi.com>

On Feb 11, 2005, at 6:11 PM, Donovan Baarda wrote: [...]

...

Given that Python is already dependant on openssl, it makes sense to change md5sum to use it. I have a feeling that openssl internally uses md5, so this way we wont link against two different md5sum implementations.

It is an optional dependency that is used when present (read: not just win32). The sources are not included with Python.

Are there any potential problems with making the md5sum module availability "optional" in the same way as this?

OpenSSL does internally have an implementation of md5 (and sha1, among other things).

Yeah, I know, that's why it could be used for the md5sum module :-) What I meant was a Python application using ssl sockets and the md5sum module will effectively have two different md5sum implementations in memory. Using the openssl md5sum for the md5sum module will make it "leaner", as well as faster. ---------------------------------------------------------------- Donovan Baarda http://minkirri.apana.org.au/~abo/ ----------------------------------------------------------------

On Sat, 2005-02-12 at 08:37, A.M. Kuchling wrote:

The md5 module has been a standard module for a long time; making it optional in the next version of Python isn't possible. We'd have to require OpenSSL to compile Python.

I totally agree. -Barry

On Sat, Feb 12, 2005 at 08:37:21AM -0500, A.M. Kuchling wrote:

On Sat, Feb 12, 2005 at 01:54:27PM +1100, Donovan Baarda wrote:

...

Are there any potential problems with making the md5sum module availability "optional" in the same way as this?

The md5 module has been a standard module for a long time; making it optional in the next version of Python isn't possible. We'd have to require OpenSSL to compile Python.

I'm happy to replace the MD5 and/or SHA implementations with other code, provided other code with a suitable license can be found.

agreed. it can not be made optional. What I'd prefer (and will do if i find the time) is to have the md5 and sha1 module use OpenSSLs implementations when available. Falling back to their built in ones when openssl isn't present. That way its always there but uses the much faster optimized openssl algorithms when they exist. -g

fyi - i've updated the python sha1/md5 openssl patch. it now replaces the entire sha and md5 modules with a generic hashes module that gives access to all of the hash algorithms supported by OpenSSL (including appropriate legacy interface wrappers and falling back to the old code when compiled without openssl). https://sourceforge.net/tracker/index.php?func=detail&aid=1121611&group_id=5470&atid=305470 I don't quite like the module name 'hashes' that i chose for the generic interface (too close to the builtin hash() function). Other suggestions on a module name? 'digest' comes to mind. -greg

On Wed, 2005-02-16 at 22:53 -0800, Gregory P. Smith wrote:

fyi - i've updated the python sha1/md5 openssl patch. it now replaces the entire sha and md5 modules with a generic hashes module that gives access to all of the hash algorithms supported by OpenSSL (including appropriate legacy interface wrappers and falling back to the old code when compiled without openssl).

https://sourceforge.net/tracker/index.php?func=detail&aid=1121611&group_id=5470&atid=305470

I don't quite like the module name 'hashes' that i chose for the generic interface (too close to the builtin hash() function). Other suggestions on a module name? 'digest' comes to mind.

I just had a quick look, and have these comments (psedo patch review?). Apologies for the noise on the list... DESCRIPTION =========== This patch keeps the current md5c.c, md5module.c files and adds the following; _hashopenssl.c, hashes.py, md5.py, sha.py. The old md5 and sha extension modules get replaced by hashes.py, md5.py, and sha.py python modules that leverage off _hash (openssl) or _md5 and _sha (no openssl) extension modules. The new _hash extension module "wraps" the high level openssl EVP interface, which uses a string parameter to indicate what type of message digest algorithm to use. The advantage of this is it makes all openssl supported digests available, and if openssl adds more, we get them for free. A disadvantage of this is it is an abstraction level above the actual md5 and sha implementations, and this may add overheads. These overheads are probably negligible compared to the actual implementation speedups. The new _md5 and _sha extension modules are simply re-named versions of the old md5 and sha modules. The hashes.py module acts as an import wrapper for _hash, and falls back to using _md5 and _sha modules if _hash is not available. It provides an EVP style API (string hash name parameter), that supports only md5 and sha hashes if openssl is not available. The new md5.py and sha.py modules simply use hash.py. COMMENTS ======== The introduction of a "hashes" module with a new API that supports many different digests (provided openssl is available) is extending Python, not just "fixing the licenses" of md5 and sha modules. If all we wanted to do was fix the md5 module, a simpler solution would be to change the md5c.c API to match openssl's implementation, and make md5module.c use it, conditionally compiling against md5c.c or linking against openssl in setup.py. A similar approach could be used for sha, but would require stripping the sha implementation out of shamodule.c I am mildly of concerned about the namespace/filespace clutter introduced by this implementation... it feels unnecessary, as does the tangled dependencies between them. With openssl, hashes.py duplicates the functionality of _hash. Without openssl, md5.py and sha.py duplicate _md5 and _sha, via a roundabout route through hash.py. The python wrappers seem overly complicated, with things like def new(name, string=None): if string: return _hash.new(name) else: return _hash.new.(name,string) being common where the following would suffice; def new(name,string=""): return _hash.new(name,string) I think this is because _hash.new() uses an optional string parameter, but I have a feeling a C update with a zero length string is faster than this Python if. If it was a concern, the C implementation could check the value of the string length before calling update. Given the convenience methods for different hashes in hashes.py (which incidentally look like they are only available when _hash is not available... something else that needs fixing), the md5.py module could be simply coded as; from hashes import md5 new = md5 Despite all these nit-picks, it looks pretty good. It is orders of magnitude better than any of the other non-existent solutions, including the one I didn't code :-) -- Donovan Baarda <abo@minkirri.apana.org.au> http://minkirri.apana.org.au/~abo/

From: "Martin v. Löwis" <martin@v.loewis.de>

Donovan Baarda wrote:

...

This patch keeps the current md5c.c, md5module.c files and adds the following; _hashopenssl.c, hashes.py, md5.py, sha.py. [...] If all we wanted to do was fix the md5 module

If we want to fix the licensing issues with the md5 module, this patch does not help at all, as it keeps the current md5 module (along with its licensing issues). So any patch to solve the problem will need to delete the code with the questionable license.

It maybe half fixes it in that if Python is happy with the RSA one, they can continue to include it, and if Debian is unhappy with it, they can remove it and build against openssl. It doesn't fully fix the license problem. It is still worth considering because it doesn't make it worse, and it does allow Python to use much faster implementations and support other digest algorithms when openssl is available.

Then, the approach in the patch breaks the promise that the md5 module is always there. It would require that OpenSSL is always there - a promise that we cannot make (IMO).

It would be better if found an alternative md5c.c. I found one that was the libmd implementation that someone mildly tweaked and then slapped an LGPL on. I have a feeling that would make the lawyers tremble more than the "public domain" libmd one, unless they are happy that someone else is prepared to wear the grief for slapping a LGPL onto something public domain. Probably the best at the moment is the sourceforge one, which is listed as having a "zlib/libpng licence". ---------------------------------------------------------------- Donovan Baarda http://minkirri.apana.org.au/~abo/ ----------------------------------------------------------------

Gregory P. Smith wrote:

fyi - i've updated the python sha1/md5 openssl patch. it now replaces the entire sha and md5 modules with a generic hashes module that gives access to all of the hash algorithms supported by OpenSSL (including appropriate legacy interface wrappers and falling back to the old code when compiled without openssl).

https://sourceforge.net/tracker/index.php?func=detail&aid=1121611&group_id=5470&atid=305470

I don't quite like the module name 'hashes' that i chose for the generic interface (too close to the builtin hash() function). Other suggestions on a module name? 'digest' comes to mind.

'hashtools' and 'hashlib' would both have precedents in the standard library (itertools and urllib, for example). It occurs to me that such a module would provide a way to fix the bug with incorrectly hashable instances of new-style classes: Py> class C: ... def __eq__(self, other): return True ... Py> hash(C()) Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: unhashable instance Py> class C(object): ... def __eq__(self, other): return True ... Py> hash(C()) 10357232 Guido wanted to fix this by eliminating object.__hash__, but that caused problems for Jython. If I remember that discussion correctly, the problem was that, in Jython, the default hash is _not_ simply hash(id(obj)) the way it is in CPython, so Python code needs a way to get access to the default implementation. A hashtools.default_hash that worked like the current object.__hash__ would seem to provide such a spelling, and allow object.__hash__ to be removed (fixing the above bug). Cheers, Nick. -- Nick Coghlan | ncoghlan@email.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.skystorm.net

On Sat, 2005-02-12 at 17:35 -0800, Gregory P. Smith wrote:

I've created an OpenSSL version of the sha module. trivial to modify to be a md5 module. Its a first version with cleanup to be done and such. being managed in the SF patch manager:

https://sourceforge.net/tracker/?func=detail&aid=1121611&group_id=5470&atid=305470

enjoy. i'll do more cleanup and work on it soon.

Hmmm. I see the patch entry, but it seems to be missing the actual patch. Did you code this from scratch, or did you base it on the current md5module.c? Is it using the openssl sha interface, or the higher level EVP interface? The reason I ask is it would be pretty trivial to modify md5module.c to use the openssl API for any digest, and would be less risk than fresh-coding one. -- Donovan Baarda <abo@minkirri.apana.org.au> http://minkirri.apana.org.au/~abo/

Donovan Baarda writes:

On Tue, 2005-02-08 at 11:52 -0800, Gregory P. Smith wrote:

...

...

The md5.h/md5c.c files allow "copy and use", but no modification of the files. There are some alternative implementations, i.e. in glibc, openssl, so a replacement should be sage. Any other requirements when considering a replacement?

One thing to consider is "degree of difficulty" :-)

...

...

Matthias

I believe the "plan" for md5 and sha1 and such is to use the much faster openssl versions "in the future" (based on a long thread debating future interfaces to such things on python-dev last summer). That'll sidestep any tedious license issue and give a better implementation at the same time. i don't believe anyone has taken the time to make such a patch yet.

I wasn't around for that discussion. There are two viable replacements for the RSA implementation currently used;

libmd <http://www.penguin.cz/~mhi/libmd/> openssl <http://www.openssl.org/>.

The libmd implementation is by Colin Plumb and has the licence; "This code is in the public domain; do with it what you wish." The API is identical to the RSA implementation and BSD world's libmd and hence is a drop in replacement. This implementation is faster than the RSA implementation.

[...]

Currently md5c.c is included in the python sources. The libmd implementation has a drop in replacement for md5c.c. The openssl implementation is a complicated tangle of Makefile expanded template code that would be harder to include in the Python sources.

I would prefer that one as a short term solution. Patch at #1118602.

Jeremy Hylton writes:

Maybe some ambitious PSF activitst could contact Roskind and Steve Kirsch and see if they know who at Disney to talk to... Or maybe the Disney guys who were at PyCon last year could help.

please could somebody give me a contact address? Matthias

On Tue, 8 Feb 2005 15:52:29 -0500, Jeremy Hylton <jhylton@gmail.com> wrote:

Maybe some ambitious PSF activitst could contact Roskind and Steve Kirsch and see if they know who at Disney to talk to... Or maybe the Disney guys who were at PyCon last year could help.

I contacted Jim. His response follows: --- I'm a strong supporter of Opensource software, but I'm probably not going to be able to help you very much. I could be much more helpful with understanding the code or its use ;-). To summarize what I'll say: I don't own the rights to this stuff. ... but I don't believe there are any patents that I was ever involved with that might encumber this work. I would note that my profiler code is really very rarely used in commercial products, and it is much more typically used by developers (I guess a developer toolkit, if sold, would use it). I'm pretty delighted that the code has found so much use by developers over the years. As I noted in the intro to the documentation, I had only been coding in Python for 3 weeks when I wrote it. On the positive side, it exposed many weaknesses in many developer's code (including our own at InfoSeek), as well as in core Python code (subtle bugs in the interpreter) that surely helped everyone. Even though I was a newbie, It was VERY carefully crafted,, and I'd expect that it would take a fair amount of effort to reproduce it (and that is is probably why it has not been changed much... or at least no one told me when they changed/fixed it ;-) ). With regard to why I probably can't help much..... First off, InfoSeek (holder of the copyright) was bought by Disney, and I don't know what if anything has eventually become of the tradename. There is a chance that Disney owns the rights... and I have no idea who to ask there :-/. Second, I took a look at the Copyright, and it sure seems pretty permissive. I'm amazed if folks want something more permissive. This is what I found on the web for it: Copyright © 1994, by InfoSeek Corporation, all rights reserved. Written by James Roskind.10.1 Permission to use, copy, modify, and distribute this Python software and its associated documentation for any purpose (subject to the restriction in the following sentence) without fee is hereby granted, provided that the above copyright notice appears in all copies, and that both that copyright notice and this permission notice appear in supporting documentation, and that the name of InfoSeek not be used in advertising or publicity pertaining to distribution of the software without specific, written prior permission. This permission is explicitly restricted to the copying and modification of the software to remain in Python, compiled Python, or other languages (such as C) wherein the modified or derived code is exclusively imported into a Python module. INFOSEEK CORPORATION DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL INFOSEEK CORPORATION BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. As I recall, I probably personally created the terms of the above license. I used a similar license on my C/C++ grammar, and Infoseek just added a bunch of wording to be sure that they were not at risk, and that their name would not be used in vain (or in advertising material). I think they were also interested in limiting its use to Python.... but I don't think that is a concern that would bother you. I read the link you directed me to, and its primary focus seemed ot be on patents for related or included technology. I don't believe that infoseek applied for or got any patents in this area (and certainly if they did so without my name, it would probably invalidate the patent), and I'm sure I didn't get any patents in this area at Netscape/AOL. In fact I don't think I got any patents back in 1994 or 1995. My only prior patent dated back to about 1983 (a hardware patent) that has since expired. I have some patents since (roughly) 1995, and even though I don't think any of them relate to profiling (though some did relate to languages, or more specifically, security in languages), I wouldn't want to mess with assigning rights to any of those patents, as they belong to AOL/Netscape. Here again, to my knowledge, none of my patents relate in any way to this area (profiling). Sadly, if they did, I would not have the right to assign them. I'm sure you're just doing your job, and following through by dotting all the I's and crossing all T's. My suggestion is to (as you said) work around the issue. You could always re-write the code from scratch, as the approaches are not rocket science and are pretty thoroughly explained. I wouldn't suggest it unless you are desperate. If I were you, I'd wait for a license problem to emerge (which I don't believe will ever happen). --- FWIW, I agree. Personnally, I think that if Debian has a problem with the above, it's their problem to deal with, not Python's. --david