CTO wrote:

...

However, it reminded me of an idea from a couple of years ago: extend the hashlib module to produce two additional kinds of hashes: a digital signature for some sequence of bytes, and an encrypted/decrypted version of a sequence of bytes. Basically, the would bring more of the OpenSSL EVP API out to Python (hashlib already uses OpenSSL EVP for various hash formations).

http://www.openssl.org/docs/crypto/evp.html

Besides the fact that hashes and encryption are pretty much totally different

I know it seems that way at first glance, but in fact they are strongly related. There's a reason all three (and nothing else) are exported through OpenSSL's EVP API. Bill

On Sun, Sep 20, 2009 at 11:52 AM, CTO wrote:

On Sep 20, 1:07 pm, Bill Janssen wrote:

...

CTO wrote:

...

...

However, it reminded me of an idea from a couple of years ago: extend the hashlib module to produce two additional kinds of hashes: a digital signature for some sequence of bytes, and an encrypted/decrypted version of a sequence of bytes.  Basically, the would bring more of the OpenSSL EVP API out to Python (hashlib already uses OpenSSL EVP for various hash formations).

...

...

http://www.openssl.org/docs/crypto/evp.html

...

Besides the fact that hashes and encryption are pretty much totally different

I know it seems that way at first glance, but in fact they are strongly related.  There's a reason all three (and nothing else) are exported through OpenSSL's EVP API.

Bill

Don't get me wrong, I like the basic idea you're advancing, and in use hashes and crypto are frequently seen together, but they solve different problems in *very* different ways. The fact that you- who obviously have some interest in crypto and presumably a good deal of knowledge about it- seem to be under the impression that RSA and SHA have some kinship, well, it doesn't reassure me about the ability of non-experts to figure out what's what. IMO, adding public key crypto routines to hashlib seems almost guaranteed to increase that confusion. Others will doubtless disagree.

Geremy Condra

I don't like the attempt to overload the hash function API. Encryption and decryption should not be done using a digest() method. That makes no sense. They are stream APIs with a constant mapping of bytes in to bytes out rather than a hash function that always outputs a constant number of bytes. I wouldn't put signing functions in hashlib itself but any common EVP wrapping code under could be shared. Before doing that I really suggest someone fleshes out the API and limits its scope to avoid feature creep. http://pycrypto.org/ is the API that most Python code wanting crypto services use today.

CTO wrote:

...

I know it seems that way at first glance, but in fact they are strongly related.  There's a reason all three (and nothing else) are exported through OpenSSL's EVP API.

Bill

Don't get me wrong, I like the basic idea you're advancing, and in use hashes and crypto are frequently seen together,

Yes, that's the relationship I was thinking of. But from a broader philosophical view, a ciphertext can be thought of as a hash of a plaintext, too. A reversible hash.

IMO, adding public key crypto routines to hashlib seems almost guaranteed to increase that confusion.

Well, that could be. Perhaps the packaging "insight" I had wasn't inspired :-). I was thinking that from the crypto-ignorant point of view, they seem quite similar. A SHA256 hash can be seen as a digital "signature" (or I've heard it called a "fingerprint") of a sequence of bytes, just as with a public-key signature. Sure, what's going on is different, but from a utility point of view, it's much the same. This is why people post md5 checksums of downloadable packages -- it's a signature. Bill

CTO wrote:

On Sep 21, 11:43 am, Bill Janssen wrote:

...

CTO wrote:

...

...

I know it seems that way at first glance, but in fact they are strongly related.  There's a reason all three (and nothing else) are exported through OpenSSL's EVP API.

...

...

Bill

...

Don't get me wrong, I like the basic idea you're advancing, and in use hashes and crypto are frequently seen together,

Yes, that's the relationship I was thinking of.  But from a broader philosophical view, a ciphertext can be thought of as a hash of a plaintext, too.  A reversible hash.

You really shouldn't conflate these things. The point of a hash is to ensure message integrity, while the point of encryption is to preserve secrecy. As an example, ElGamal is a common cryptosystem that nevertheless preserves the multiplicative homomorphism..., ie,

I know lots of non-crypto users -- the people the "batteries included" aspect of Python are for -- that don't understand the fine points of this, and don't want to. They just want to encrypt some text with a "good" cipher scheme, and they depend on the library implementor to know how to do that. They want a function "encrypt(plaintext, key)", and don't really want to know more than that. And, by the by, hashes are often used for purposes other than message integrity, outside the sphere of crypto.

...

...

IMO, adding public key crypto routines to hashlib seems almost guaranteed to increase that confusion.

Well, that could be.  Perhaps the packaging "insight" I had wasn't inspired :-).  I was thinking that from the crypto-ignorant point of view, they seem quite similar.  A SHA256 hash can be seen as a digital "signature" (or I've heard it called a "fingerprint") of a sequence of bytes, just as with a public-key signature.  Sure, what's going on is different, but from a utility point of view, it's much the same.  This is why people post md5 checksums of downloadable packages -- it's a signature.

Also a very bad idea. Hashes ensure data integrity, not that it came from the person that you think it came from. As an example, if I took

Sure. And lots of people use digital signatures in that way, too. Again, I wasn't proposing to replace m2cryto or pycrypto or anything else; I was suggesting that providing easy-to-use APIs to a couple of commonly-requested crypto features, for use by non-cryptographers, wouldn't be a bad idea. Bill

Robert Kern wrote:

On 2009-09-21 16:37 PM, Bill Janssen wrote:

...

Again, I wasn't proposing to replace m2cryto or pycrypto or anything else; I was suggesting that providing easy-to-use APIs to a couple of commonly-requested crypto features, for use by non-cryptographers, wouldn't be a bad idea.

Going back to CTO's original reply, I would say that he agrees with you. Where he (and I, for that matter) diverge is that we don't think they should go into hashlib. The name is inappropriate and misleading.

OK, so let's not do that, then. Greg commented that the underlying C implementation of access to EVP can be shared, which eliminates the only real functional justification for adding it to hashlib, which is to avoid duplicating code. Suppose we added, then, a simple-minded API to the EVP functions: EVP_Seal... and EVP_Open... provide public key encryption EVP_Sign... and EVP_Verify... provide digital signatures EVP_Encrypt and EVP_Decrypt provide symmetric key encryption (The EVP_Digest... API is already brought out by hashlib.) Let's call this new module "evp". (Or perhaps there should be a "crypto" package, with "hashes", "encryption", and "signature" submodules.) Let's look at symmetric encryption. You'd want to be able to create a new encryptor: import evp e = evp.encryptor(key, cipher="AES256", padding=True) "cipher" defaults to AES256, the constructor raises an exception if that isn't available (or the specified cipher isn't available), or for a bad key (wrong length). e.update(plaintext) # repeat as needed ciphertext = e.result() Very similar for decryption. What can be done to make something like this foolproof? Bill

CTO wrote:

...

Let's look at symmetric encryption.  You'd want to be able to create a new encryptor:

  import evp   e = evp.encryptor(key, cipher="AES256", padding=True)

"cipher" defaults to AES256, the constructor raises an exception if that isn't available (or the specified cipher isn't available), or for a bad key (wrong length).

I'm personally against the idea of default ciphers, etc. Since difference ciphers, keylengths, and padding choices have immediate consequences for what kinds of security you're going to have, I would rather be explicit than implicit here.

Sure, your privilege. Unfortunately, most users won't be able to make those choices sanely, and will rely on some sort of external advice about it. So I think it makes sense to try to build some such advice into the code, by adding a reasonably strong encryption standard as a default, and by adding some code to do sanity/compatibility checks on the user-selected keys, if possible.

...

  e.update(plaintext)           # repeat as needed   ciphertext = e.result()

Very similar for decryption.

Most ciphers are not stream ciphers, so it doesn't make a lot of sense in the case of, say, RSA or AES, but again- bikeshedding.

Still, good point. Multiple calls to update() should raise an exception if the chosen cipher is not a stream cipher. Or, allow multiple calls, and buffer the input until result() is called. Bill

CTO wrote:

If you don't know what the application is, you don't know what's secure and what isn't. We have no way of knowing, and so should resist the temptation to guess.

"secure" is not the same as "strongly encrypted". I'm looking at providing a simple way to do encryption here, not security. Let's just focus on that, first. I think defaulting to Blowfish or AES256 would be a reasonable tack to take there. I suggested AES256 because it seems to me more likely to be widely available.

AFAIK, AES and RSA are the most commonly used algorithms in EVP. Maybe it would make more sense to take the more traditional keygen-encrypt-decrypt approach?

Sure, maybe so. What would a proposed interface look like, then? Bill

Nick Coghlan wrote:

Bill Janssen wrote:

...

Again, I wasn't proposing to replace m2cryto or pycrypto or anything else; I was suggesting that providing easy-to-use APIs to a couple of commonly-requested crypto features, for use by non-cryptographers, wouldn't be a bad idea.

Actually, it could be a really bad idea that leads to people thinking they have secured something when they have in fact done nothing of the sort.

I don't think so. People make mistakes continually. Part of life. Making it easier to do things leads to that sort of misunderstanding. That being said, are there ways to make things more foolproof?

Having to go find a crypto library at least means a developer has put in a minimal amount of thought.

Too much, IMO. Bill

Bill Janssen writes:

Again, I wasn't proposing to replace m2cryto or pycrypto or anything else; I was suggesting that providing easy-to-use APIs to a couple of commonly-requested crypto features, for use by non-cryptographers, wouldn't be a bad idea.

I think it would be good indeed. Since we already wrapping OpenSSL, let's give access to more of its features instead of having people find additional binary packages (of varying quality) for their platform. As for some of the points which have been raised here: - Putting non-hash functions in "hashlib" would look strange. - Please don't call the package "evp", it's cryptic (;-)) and tied to a specific implementation. "crypto" would be fine and obvious. - I don't think there should be a default argument. People shouldn't try to do any crypto at all if they aren't able to choose an algorithm. Documenting (perhaps recommending) a couple of them (AES, RSA) would be more helpful than supporting a silent default. - The API needn't (and shouldn't) be the same as for hashing objects. A digest() method doesn't make sense. Ideally some of the {en,de}crypting objects could be file-like objects, but it's not really necessary IMO (and it can always be done in pure Python on top of a lower-level C extension, assuming the extension does provide a streaming interface). Regards Antoine.

On Sep 25, 7:44 am, Antoine Pitrou wrote:

Bill Janssen writes:

...

Again, I wasn't proposing to replace m2cryto or pycrypto or anything else; I was suggesting that providing easy-to-use APIs to a couple of commonly-requested crypto features, for use by non-cryptographers, wouldn't be a bad idea.

I think it would be good indeed. Since we already wrapping OpenSSL, let's give access to more of its features instead of having people find additional binary packages (of varying quality) for their platform.

I've started putting the code together. It's very rough, but it does vaguely what it's supposed to do. It only supports AES192 right now, but it shouldn't be any harder to expand on that than it was to write the existing code, and I'd appreciate the contributions of anyone interested in seeing this in the standard library- it needs the help. You can find my code over at http://gitorious.org/cryptography-py/cryptography-py.

As for some of the points which have been raised here: ...snip... - Please don't call the package "evp", it's cryptic (;-)) and tied to a specific implementation. "crypto" would be fine and obvious.

I know of at least one implementation that uses Crypto as its name, and forgot to send them an email earlier this week. I'll do that today, and if they don't seem bent on changing the capitalization, I don't see any problem there. A side note: the code I've posted is a little schizophrenic since we haven't figured a name or toplevel structure out. That can change whenever a consensus emerges.

- I don't think there should be a default argument. People shouldn't try to do any crypto at all if they aren't able to choose an algorithm. Documenting (perhaps recommending) a couple of them (AES, RSA) would be more helpful than supporting a silent default.

Right now it just divides one cryptosystem into each module. For example, to use AES192, you can simply say: import aes ciphertext = aes.encrypt("this is a message", "this is my password") plaintext = aes.decrypt(ciphertext, "this is my password") assert plaintext == "this is a message" and it handles salt generation, IV generation, bitlength selection, and key strengthening for you. Eventually, if crypto becomes the name of the package, it will probably turn into "from crypto import aes".

- The API needn't (and shouldn't) be the same as for hashing objects. A digest() method doesn't make sense. Ideally some of the {en,de}crypting objects could be file-like objects, but it's not really necessary IMO (and it can always be done in pure Python on top of a lower-level C extension, assuming the extension does provide a streaming interface).

I can provide the streaming interface if it's desired. It would in some cases be more efficient, but for most I think it's needless complexity. Again, others may disagree, and I'm sure there's fertile ground for discussion there. Geremy Condra

2009/9/25 Robert Kern :

On 2009-09-25 10:40 AM, CTO wrote:

...

On Sep 25, 7:44 am, Antoine Pitrou  wrote:

...

- Please don't call the package "evp", it's cryptic (;-)) and tied to a specific implementation. "crypto" would be fine and obvious.

I know of at least one implementation that uses Crypto as its name, and forgot to send them an email earlier this week. I'll do that today, and if they don't seem bent on changing the capitalization, I don't see any problem there.

I wouldn't rely on capitalization to differentiate between packages. I think that Python will be able to do so in most circumstances, even on Windows, but it makes it hard to talk about the different packages and ask questions about them on the mailing lists.

Agreed. How about "encryption"? Paul.

CTO wrote:

EVP covers hashing, signatures, and encryption/decryption. If we're going to go for a longer name, maybe "cryptography" would be more appropriate?

Something to keep in mind while working on this is your threat model for the library. If you aren't going to do anything to guard against side-channel attacks (which are rather hard to avoid in a cross platform algorithm on a general purpose PC) or against attacks which grab unencrypted messages and keys from released-but-not-overwritten computer memory or (worse) the swap file, then this should be mentioned in the documentation. That way application developers that are looking for that extra level of security will know they need to look elsewhere. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

geremy condra wrote:

On Fri, Sep 25, 2009 at 9:49 PM, Nick Coghlan mailto:ncoghlan@gmail.com> wrote:

CTO wrote: > EVP covers hashing, signatures, and encryption/decryption. If we're > going > to go for a longer name, maybe "cryptography" would be more > appropriate?

Something to keep in mind while working on this is your threat model for the library. If you aren't going to do anything to guard against side-channel attacks (which are rather hard to avoid in a cross platform algorithm on a general purpose PC) or against attacks which grab unencrypted messages and keys from released-but-not-overwritten computer memory or (worse) the swap file, then this should be mentioned in the documentation.

That way application developers that are looking for that extra level of security will know they need to look elsewhere.

Regards, Nick.

I can make a note of it, although I'm unsure what concrete steps I could take to prevent such attacks from succeeding. Any ideas?

OpenSSL may actually guard against of the first part already. I'm unsure about the second part though. And I don't know enough about the problems to know how to fix them either - I just know when I'm theoretically leaving these attack vectors open and make sure to defend them by other means (such as physically securing the affected networks). But it's this kind of stuff that people are talking about when they point out that practical crypto is harder than just using good algorithms. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------

On Sat, Sep 26, 2009 at 12:01 AM, geremy condra wrote:

On Fri, Sep 25, 2009 at 10:35 PM, Nick Coghlan wrote:

...

geremy condra wrote:

...

On Fri, Sep 25, 2009 at 9:49 PM, Nick Coghlan mailto:ncoghlan@gmail.com> wrote:

CTO wrote: > EVP covers hashing, signatures, and encryption/decryption. If

we're

...

> going > to go for a longer name, maybe "cryptography" would be more > appropriate?

Something to keep in mind while working on this is your threat model

for

...

the library. If you aren't going to do anything to guard against side-channel attacks (which are rather hard to avoid in a cross

platform

...

algorithm on a general purpose PC) or against attacks which grab unencrypted messages and keys from released-but-not-overwritten

computer

...

memory or (worse) the swap file, then this should be mentioned in

the

...

documentation.

That way application developers that are looking for that extra

level of

...

security will know they need to look elsewhere.

Regards, Nick.

I can make a note of it, although I'm unsure what concrete steps I could take to prevent such attacks from succeeding. Any ideas?

OpenSSL may actually guard against of the first part already. I'm unsure about the second part though. And I don't know enough about the problems to know how to fix them either - I just know when I'm theoretically leaving these attack vectors open and make sure to defend them by other means (such as physically securing the affected networks).

But it's this kind of stuff that people are talking about when they point out that practical crypto is harder than just using good algorithms.

Cheers, Nick.

It seems to me that most timing attacks should already be out, so I'm not *too* worried about that- but I have literally no idea how to stop the secrets from being dropped into swap in this context. I think what I'm going to do is just make a note of it in the docs for the module and make sure that there's enough contact info in there to ensure that anybody reviewing the code can let me know if we're doing something stupid.

And on that note: since most of my knowledge of crypto is theoretical in nature, I'm going to be leaning a lot on those of you with more practical experience as we go forward with this. Therefore, for everybody reading this list- please, *review the code*! If you think there's a problem, assume you're right and let me know- we really, really, really do not want half-baked crypto in the standard lib. Assuming that's even where this is headed.

Geremy Condra

Added the notice to the top of aes.c. Geremy Condra