From bram at GAWTH.COM Sun Mar 17 23:56:15 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Sun, 17 Mar 2002 14:56:15 -0800 Subject: [PYTHON-CRYPTO] things missing from PEP 272 Message-ID: The standard NIST cipher modes are defined in this document http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf The modes it gives are CBC, ECB, CFB, OFB, and CTR, the last two of which are missing from PEP 272. Some further justification for CTR is given here - http://www.cs.berkeley.edu/~daw/papers/ctr-aes00.ps I'd really like CTR to ship with Python because, well, I'm using it, and it's the only thing keeping my app from being pure Python :-) The API for OFB is obvious, but the one for CTR is considerably less so. My suggestion is to have the IV be the first block to be encrypted, and have the big/little endian issue be addressed by having CTR mean big endian and CLE mean little endian. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From phr-pycrypt at nightsong.com Mon Mar 18 07:00:22 2002 From: phr-pycrypt at nightsong.com (Paul Rubin) Date: Mon, 18 Mar 2002 06:00:22 -0000 Subject: [PYTHON-CRYPTO] things missing from PEP 272 Message-ID: <20020318060022.30362.qmail@brouhaha.com> The standard NIST cipher modes are defined in this document http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf The modes it gives are CBC, ECB, CFB, OFB, and CTR, the last two of which are missing from PEP 272. Some further justification for CTR is given here - http://www.cs.berkeley.edu/~daw/papers/ctr-aes00.ps I'd really like CTR to ship with Python because, well, I'm using it, and it's the only thing keeping my app from being pure Python :-) I agree that CTR mode should be included. I think OFB and CFB modes can be dropped. I think PGP mode should be be dropped unless someone specifically wants to write a PGP-compatible application. Certainly, implementations of the API should not be REQUIRED to support all these modes. The API for OFB is obvious, but the one for CTR is considerably less so. My suggestion is to have the IV be the first block to be encrypted, and have the big/little endian issue be addressed by having CTR mean big endian and CLE mean little endian. The IV for counter mode should be an int or long giving the counter value. CTR/CLE for endianness is ok, or just choose one endianness and stick to it. Are there any applications/standards with specific endianness requirements? Actually, I just read the PEP and it looks to me like it needs several further changes: 1. It says nothing about padding methods. CBC mode should probably pad per RFC 1423 a/k/a PKCS #5. I don't know about CFB mode. Requiring that the plaintext size be a multiple of the block size means there has to be another API layer to deal with padding. That seems clumsy to me, but if it's what's intended, the PEP should also specify that layer. I'd rather permit arbitrary lengths for plaintext and specify a padding algorithm. It's best to specify how to do what users actually want, instead of leaving them on their own to code up some unspecified intermediate layer. I.e. if you're going to specify a function taking a string plaintext arg and returning a string ciphertext, it should handle arbitrary lengths. 2. The treatment of stream ciphers is bogus. The idea of ECB mode for a stream cipher makes no sense at all. Stream ciphers should use CTR or CFB mode. They should be treated separately from block ciphers rather than kludged in by specifying a block size of 1. 3. The specifications of block_size and key_size are vague: is "size" measured in bits, or in bytes? The key and data args to the encryption function are byte strings, so block_size should probably be bytes (too bad for 1-bit CFB mode or 1-bit stream ciphers). Usually though we think of "56-bit DES" or "128-bit AES", so it's slightly surprising (but ok) to have key_size be bytes. 4. The key_size variable must be either an integer (a specific number of bits), or None (meaning any length at all is valid). This doesn't handle AES very well--AES accepts 128, 192, or 256 bits, but not other values. RC4 accepts any key length that's a multiple of 8 bits, but won't accept (say) 23 bits. Maybe this variable should just be eliminated, and let the module throw an error if you try to set a key length it doesn't like. 5. I think it should be ok to pass an initial IV of None or 0, equivalent to "\0"*(module.block_size) where block_size is in bytes. Usually you use IV=0 to compute a CBC MAC, for example. 6. The PEP should specify what exceptions the modules can raise and when. 7. There should be a 'mac' operation added, similar to encrypt and decrypt. MAC runs the cipher and updates the IV and returns the final IV instead of encrypted or decrypted text. There should also be a 'verify' operation which tests whether a string has a given MAC and returns 1 or 0. 8. The encrypt, decrypt, mac, and verify operations should all be allowed to raise NotImplementedError if the cipher doesn't support that mode. It's perfectly ok for a cipher to support encryption but not decryption, or a cipher to not support PGP chaining mode, etc. From bram at GAWTH.COM Mon Mar 18 09:15:15 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Mon, 18 Mar 2002 00:15:15 -0800 Subject: [PYTHON-CRYPTO] things missing from PEP 272 In-Reply-To: <20020318060022.30362.qmail@brouhaha.com> Message-ID: Paul Rubin wrote: > The IV for counter mode should be an int or long giving the counter > value. CTR/CLE for endianness is ok, or just choose one endianness > and stick to it. Are there any applications/standards with specific > endianness requirements? Yes. All of them. If one machine encryptes big-endian, and another decrypts little-endian, you get two very confused machines :-). Seriously, endainness is a major gotcha for implementation which is often forgotten about in algorithm design. I go so far in BitTorrent as to start my counter at 1 instead of 0 to make endianness problems visible immediately. Unfortunately the NIST document on modes shrugs off the whole endianness issue. I wish I'd submitted comments during the review period emphasizing how important this issue is. One *can* use an integer to represent the counter, but it makes the API different from other modes. They both work, but I slightly prefer it as a string. > 3. The specifications of block_size and key_size are vague: is > "size" measured in bits, or in bytes? Every crypto library I've used has measured everything in bytes. > 4. The key_size variable must be either an integer (a specific number > of bits), or None (meaning any length at all is valid). This doesn't > handle AES very well--AES accepts 128, 192, or 256 bits, but not other > values. AES also has defined variants for block sizes of 192 and 256 bits, but those aren't NIST standards. > 5. I think it should be ok to pass an initial IV of None or 0, > equivalent to "\0"*(module.block_size) where block_size is in bytes. > Usually you use IV=0 to compute a CBC MAC, for example. Also, the IV should be passed as an argument to encrypt() and decrypt(), not to new(). Additionally, there should be support for encrypting a stream incrementally, possibly by having make_encrypter() and make_decrypter() methods, which return functions which you pass pieces of the string to be processed in order, and they return the processed string as it's possible to compute it. Here's some test code in my app which shows how incremental encryption should work - def checkpoints(s, l): cm = make_encrypter('abcd' * 4) buf = '' for i in xrange(len(l) - 1): buf += cm(s[l[i]:l[i+1]]) return buf def test(): p = ''.join([chr((3 * i) % 256) for i in xrange(500)]) a = checkpoints(p, [0, 3, 11, 200, 350, 400, 500]) b = checkpoints(p, [0, 0, 21, 27, 28, 29, 301, 500]) assert a == b -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From philh at COMUNO.FREESERVE.CO.UK Mon Mar 18 13:04:43 2002 From: philh at COMUNO.FREESERVE.CO.UK (phil hunt) Date: Mon, 18 Mar 2002 12:04:43 +0000 Subject: [PYTHON-CRYPTO] things missing from PEP 272 In-Reply-To: <20020318060022.30362.qmail@brouhaha.com> References: <20020318060022.30362.qmail@brouhaha.com> Message-ID: <02031812044301.01811@comuno> On Monday 18 March 2002 6:00 am, Paul Rubin wrote: > The standard NIST cipher modes are defined in this document > > http://csrc.nist.gov/publications/nistpubs/800-38a/sp800-38a.pdf > > The modes it gives are CBC, ECB, CFB, OFB, and CTR, the last two of > which are missing from PEP 272. Some further justification for CTR is > given here - > > http://www.cs.berkeley.edu/~daw/papers/ctr-aes00.ps > > I'd really like CTR to ship with Python because, well, I'm using it, > and it's the only thing keeping my app from being pure Python :-) > > I agree that CTR mode should be included. I think OFB and CFB modes > can be dropped. I think PGP mode should be be dropped unless someone > specifically wants to write a PGP-compatible application. I think it's a reasonable assumption that someone might want to do that. PGP is a fairly well-known application. -- <"><"><"> Philip Hunt <"><"><"> "I would guess that he really believes whatever is politically advantageous for him to believe." -- Alison Brooks, referring to Michael Portillo, on soc.history.what-if From bram at GAWTH.COM Mon Mar 18 18:48:14 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Mon, 18 Mar 2002 09:48:14 -0800 Subject: [PYTHON-CRYPTO] things missing from PEP 272 In-Reply-To: Message-ID: Bram Cohen wrote: > Also, the IV should be passed as an argument to encrypt() and decrypt(), > not to new(). I should probably explain - reusing IV's is almost always a bad idea and one of the common gotchas in protocol design. No need to encourage it. Also, no need to force the overhead of another key setup just to change IV (not a big deal for AES, but key setup for blowfish (for example) is glacial.) -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From phr-pycrypt at nightsong.com Mon Mar 18 18:50:39 2002 From: phr-pycrypt at nightsong.com (Paul Rubin) Date: Mon, 18 Mar 2002 17:50:39 -0000 Subject: [PYTHON-CRYPTO] things missing from PEP 272 Message-ID: <20020318175039.3594.qmail@brouhaha.com> > Also, the IV should be passed as an argument to encrypt() and > decrypt(), not to new(). I should probably explain - reusing IV's is almost always a bad idea and one of the common gotchas in protocol design. No need to encourage it. Also, no need to force the overhead of another key setup just to change IV (not a big deal for AES, but key setup for blowfish (for example) is glacial.) The IV is an instance variable in the crypto object. You can set it to a new value without calling new(). From bram at GAWTH.COM Mon Mar 18 18:59:37 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Mon, 18 Mar 2002 09:59:37 -0800 Subject: [PYTHON-CRYPTO] things missing from PEP 272 In-Reply-To: <20020318175039.3594.qmail@brouhaha.com> Message-ID: Paul Rubin wrote: > The IV is an instance variable in the crypto object. You can set it > to a new value without calling new(). That's a fairly klunky API - rather than doing an encryption be a single call, it's an undocumented variable assignment followed by a call, much less readable and much more likely to be implemented wrong. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From zooko at zooko.com Mon Mar 18 22:31:08 2002 From: zooko at zooko.com (Zooko) Date: Mon, 18 Mar 2002 13:31:08 -0800 Subject: [PYTHON-CRYPTO] things missing from PEP 272 In-Reply-To: Message from Bram Cohen of "Sun, 17 Mar 2002 14:56:15 PST." References: Message-ID: Bram Cohen wrote: > > The API for OFB is obvious, but the one for CTR is considerably less so. > My suggestion is to have the IV be the first block to be encrypted, Err... Wouldn't that mean that if you sent a message consisting of BLOCK0 and BLOCK1, and then you sent a message consisting of BLOCK0 and BLOCK2, that both messages would be completely readable to a passive eavesdroppper? :-) Regards, Zooko --- zooko.com Security and Distributed Systems Engineering --- From zooko at zooko.com Mon Mar 18 22:33:14 2002 From: zooko at zooko.com (Zooko) Date: Mon, 18 Mar 2002 13:33:14 -0800 Subject: [PYTHON-CRYPTO] things missing from PEP 272 In-Reply-To: Message from Paul Rubin of "Mon, 18 Mar 2002 06:00:22 GMT." <20020318060022.30362.qmail@brouhaha.com> References: <20020318060022.30362.qmail@brouhaha.com> Message-ID: Paul Rubin wrote: > > The IV for counter mode should be an int or long giving the counter > value. I think it is a mistake to make the IV a counter, in general, because then you have to guarantee that you never lose the persistent counter state while retaining the key after having sent a message encrypted with that counter state. (I.e., because your key is persistent, your counter state is updated and used in volatile memory, and your system crashed after sending an encrypted message but before the counter state update reached non-volatile media.) The only solutions that I can think of are either to use keys that are guaranteed to be no more persistent than the IV, or to delay sending encrypted messages until you get a positive "I'm now durable" signal (yeah right) from the counter state's persistent store, or better, generate a new random IV on startup. (You can then increment it for each successive message or generate a new random one for each successive message.) Anyway, that's why I think the IV should be a string not an integer. (BTW, you have the same problem about the persistence of the state used to generate your randomness, but at least that pushes it out to more widely used code (i.e. /dev/urandom) which you are already vulnerable to and which more people are working on, and it is also ameliorated by "adding in fresh entropy" derived from events after startup.) Regards, Zooko --- zooko.com Security and Distributed Systems Engineering --- From bram at GAWTH.COM Tue Mar 19 02:23:19 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Mon, 18 Mar 2002 17:23:19 -0800 Subject: [PYTHON-CRYPTO] things missing from PEP 272 In-Reply-To: Message-ID: Zooko wrote: > Bram Cohen wrote: > > > > The API for OFB is obvious, but the one for CTR is considerably less so. > > My suggestion is to have the IV be the first block to be encrypted, > > Err... Wouldn't that mean that if you sent a message consisting of BLOCK0 and > BLOCK1, and then you sent a message consisting of BLOCK0 and BLOCK2, that both > messages would be completely readable to a passive eavesdroppper? :-) Well, yes - you aren't supposed to reuse keys for CTR mode. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From phr-pycrypt at nightsong.com Tue Mar 19 10:00:26 2002 From: phr-pycrypt at nightsong.com (Paul Rubin) Date: Tue, 19 Mar 2002 09:00:26 -0000 Subject: [PYTHON-CRYPTO] things missing from PEP 272 Message-ID: <20020319090026.18934.qmail@brouhaha.com> The only solutions that I can think of are either to use keys that are guaranteed to be no more persistent than the IV, or to delay sending encrypted messages until you get a positive "I'm now durable" signal (yeah right) from the counter state's persistent store, or better, generate a new random IV on startup. (You can then increment it for each successive message or generate a new random one for each successive message.) Anyway, that's why I think the IV should be a string not an integer. Normally you use CTR mode by choosing a random key and initializing the counter value to 0. For the next message, you use a different random key and start counting at 0 again. CTR mode is really a stream cipher and you should never re-use a key. Holding the key fixed and choosing the IV randomly per message would lead to likely collisions after only ~2**32 messages (for a 64 bit block cipher) regardless of the key length--fewer messages than that, if they contained multiple blocks. Perhaps the confusion here is calling the counter value an IV when it's not a random IV in the way that CBC IV's usually are. Maybe it should be called something else. I still think it should be an integer. How much flexibility exists in revising/rewriting the PEP 272 anyway? From akuchlin at mems-exchange.org Tue Mar 19 15:30:46 2002 From: akuchlin at mems-exchange.org (Andrew Kuchling) Date: Tue, 19 Mar 2002 09:30:46 -0500 Subject: [PYTHON-CRYPTO] things missing from PEP 272 In-Reply-To: <20020319090026.18934.qmail@brouhaha.com> References: <20020319090026.18934.qmail@brouhaha.com> Message-ID: <20020319143046.GA10348@django> >How much flexibility exists in revising/rewriting the PEP 272 anyway? It was never finalized, so it's free to change; any volunteers to update it? (I lack the time to rewrite it, or even to read this thread, so someone else will have to do it.) If you lack CVS access, I'll be happy to check in a revised version of the PEP. --amk From bram at GAWTH.COM Mon Mar 18 20:06:01 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Mon, 18 Mar 2002 11:06:01 -0800 Subject: [PYTHON-CRYPTO] new release of BitTorrent out Message-ID: I just pushed out a new release of BitTorrent, you can get it here - http://bitconjurer.org/BitTorrent/download.html BitTorrent is a crypto-using application written in Python :-) -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From bram at GAWTH.COM Sat Mar 23 05:00:38 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Fri, 22 Mar 2002 20:00:38 -0800 Subject: [PYTHON-CRYPTO] things missing from PEP 272 In-Reply-To: <20020319143046.GA10348@django> Message-ID: Andrew Kuchling wrote: > >How much flexibility exists in revising/rewriting the PEP 272 anyway? > > It was never finalized, so it's free to change; any volunteers to > update it? (I lack the time to rewrite it, or even to read this > thread, so someone else will have to do it.) I'd like to take a stab at this. I'd have done it a few days ago, but I was, you know, getting slashdotted. Here are the changes I'd like to make - Add CTR mode Add incremental encryption versions of all modes where it's applicable Add support for multiple padding modes for CBC and others to which it might be applicable Get rid of cross-mode API unification. This produces some awful kludges and I've never heard of anyone actually swapping out one mode for another They're just too different. Well, that's the current plan. Does anyone have any comments? I'd like to get any feedback people may have before doing a rewrite. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From bram at GAWTH.COM Sat Mar 23 05:16:36 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Fri, 22 Mar 2002 20:16:36 -0800 Subject: [PYTHON-CRYPTO] A Python crypto project on slashdot In-Reply-To: Message-ID: Bram Cohen wrote: > I'd have done it a few days ago, but I was, you know, getting > slashdotted. The project I got slashdotted for is actually topical here - http://slashdot.org/developers/02/03/20/0143248.shtml?tid=156 It's written in Python, except for a small bit of encryption code in C which is there because there's none in the standard libs (that's actually why I'm on this list, to try to get it into the standard libs). Notably, none of the CPUs involved so much as flinched, even though they were pushing through as much as 10 megabits per second at times. (It's a tool for distributing load, but hey, it can't hurt to contribute a decent amount for the demo anyway :-) ) -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From phr-pycrypt at nightsong.com Sat Mar 23 05:20:39 2002 From: phr-pycrypt at nightsong.com (Paul Rubin) Date: Sat, 23 Mar 2002 04:20:39 -0000 Subject: [PYTHON-CRYPTO] PEP 272 Message-ID: <20020323042039.31822.qmail@brouhaha.com> [Bram]: >I'd like to take a stab at this. I'd have done it a few days ago, >but I was, you know, getting slashdotted. I begin to wonder if this PEP is useful at all. It basically specifies a fairly low level interface to a symmetric. I just don't see the need to swap block ciphers in and out of applications. Rather than a PEP specifying an API, I think it's better to just write a crypto module that supplies the needed features, and submit it to the Python standard library to replace the silly rotor module. If there's an API PEP, it should specify a more complete architecture including public key interfaces and key management, along the lines of JCA. But re your specific changes: >Add CTR mode OK. >Add incremental encryption versions of all modes where it's applicable This doesn't seem really appropriate for a low level interface. Rather, the low level interface should ensure that incremental interface can be provided by a class that calls generic cipher objects, and then include an implementation of such a class in our module. I think the incremental encryption interface should look similar to the current interface for hash functions, e.g.: e = encrypter.new(cipher) for plaintext in getplaintext(): ciphertext = e.update(plaintext) output_stream.write(ciphertext) output_stream.write(e.final()) >Add support for multiple padding modes for CBC and others to which it >might be applicable Don't add multiple padding modes for CBC without a good reason. Just pick a mode and specify it. >Get rid of cross-mode API unification. This produces some awful >kludges and I've never heard of anyone actually swapping out one >mode for another They're just too different. I'm not sure what this refers to. From bram at GAWTH.COM Sat Mar 23 08:15:19 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Fri, 22 Mar 2002 23:15:19 -0800 Subject: [PYTHON-CRYPTO] PEP 272 In-Reply-To: <20020323042039.31822.qmail@brouhaha.com> Message-ID: Paul Rubin wrote: > I begin to wonder if this PEP is useful at all. It basically > specifies a fairly low level interface to a symmetric. I just don't > see the need to swap block ciphers in and out of applications. I agree that it makes much more sense to just create an aes module, the discussion of what it's API should be still applies though. > >Add incremental encryption versions of all modes where it's applicable > > This doesn't seem really appropriate for a low level interface. > Rather, the low level interface should ensure that incremental > interface can be provided by a class that calls generic cipher > objects, and then include an implementation of such a class in our > module. That would be a good design, if done right, I'm not sure how the lowel-level stuff would work though, and it should definitely be nicely wrapped in normal modes for people to call from their programs. > I think the incremental encryption interface should look similar to > the current interface for hash functions, e.g.: > > e = encrypter.new(cipher) > for plaintext in getplaintext(): > ciphertext = e.update(plaintext) > output_stream.write(ciphertext) > output_stream.write(e.final()) That's what I was thinking as well. > >Add support for multiple padding modes for CBC and others to which it > >might be applicable > > Don't add multiple padding modes for CBC without a good reason. > Just pick a mode and specify it. One could just support exact size roundoffs and make people do their own padding with another call or two to ECB. > >Get rid of cross-mode API unification. This produces some awful > >kludges and I've never heard of anyone actually swapping out one > >mode for another They're just too different. > > I'm not sure what this refers to. Oh, a few things, like how counter mode should really take an integer to specify it's start value instead of a string, and how CBC might not accept arbitrary byte boundaries - minor things, but getting all the modes to look *exactly* the same takes a lot of uncomfortable shoehorning. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From Paul at CRYPTORIGHTS.ORG Mon Mar 25 06:42:10 2002 From: Paul at CRYPTORIGHTS.ORG (Paul) Date: Sun, 24 Mar 2002 21:42:10 -0800 Subject: [PYTHON-CRYPTO] things missing from PEP 272 In-Reply-To: References: Message-ID: At 8:00 PM -0800 3/22/02, Bram Cohen wrote: >Well, that's the current plan. Does anyone have any comments? I'd like to >get any feedback people may have before doing a rewrite. A few comments on PEP 272: 1) it seems a little odd to have the different modes built into the api. It seems more reasonable to simply have a bunch of named algorithms and modes. 2) IV should not be required. Perhaps optional if there were some good reason to control the IV. In general it's best to hide this from applications. 3) Why only support multiples of block sizes. This is a pain. Both padding and IV (addition and removal) should be automatic. Padding for each algorithm could use a default padding mechanism, with an optional parameter allowing the selection of alternative padding mechanisms. 4) The interface/methods should support the encryption of large data and data streams. This means that you need keep track of state in the class so that a stream of data can be encrypted in small pieces. The default behavior should allow encryption with a single method (e.g aes.encrypt("foo") ). Methods to encrypt a stream require either two or three methods (encryptFirst, encryptMiddle, encryptLast) or a parameter that indicates than the stream is not complete (e.g. a "more" flag). Either approach works, I'm partial to the latter. 5) The stream encryption, automatic IV, automatic padding could allow be handled by defining a Cipher class that provides keeps track of the state and blocking information. Block algorithms would then inherit the base encrypt and decrypt methods. The less useful basic block oriented operations would then be encrypt_block and decrypt_block methods that would be used by the Cipher class. 6) Not all algorithms need IV or padding ... so we might want every algorithm to provide the minimum and maximum data expansion (IV + padding). 7) Each algorithm should have a unique asci name (uniqueness could be a problem to manage) 8) OIDs should be considered ... they belong either as part of the algorithm class or will need to be collected separately. Seems easier to design them in from the start so that every algorithm knows it's OID(s). 9) Counter mode or other combination encryption/integrity algorithms should be build on Cipher, but have some additional methods and parameters. Recent work on 802.11i with AES-OCB or AES-CCM should serve as a good example. I suggest we give these algorithms some type of new name since they are not the same as a "Cipher". New methods (or assertions) are needed to check the integrity of the decryption process. 10) On AES-OCB and AES-CCM ... these integrity checking algorithms are also being used in a way that allows parts of the unencrypted header of PDUs to be combined into the processing. This means that that encrypting with integrity checking of headers has two parameters: ct=aes-ocb.encrypt_with_integrity("plain header","body to encrypt") (suggestions for shorter name for above method would be greatly appreciated) Paul > >Well, that's the current plan. Does anyone have any comments? I'd like to >get any feedback people may have before doing a rewrite. > >-Bram Cohen > >"Markets can remain irrational longer than you can remain solvent" > -- John Maynard Keynes -- From phr-pycrypt at nightsong.com Mon Mar 25 11:33:14 2002 From: phr-pycrypt at nightsong.com (Paul Rubin) Date: Mon, 25 Mar 2002 10:33:14 -0000 Subject: [PYTHON-CRYPTO] things missing from PEP 272 Message-ID: <20020325103314.4838.qmail@brouhaha.com> A few comments on PEP 272: 1) it seems a little odd to have the different modes built into the api. It seems more reasonable to simply have a bunch of named algorithms and modes. I think the idea was to make the API algorithm-independent. 2) IV should not be required. Perhaps optional if there were some good reason to control the IV. In general it's best to hide this from applications. Generally you want to use a random IV, but the PEP (and the cipher implementations) don't have any way to provide randomness, so it's left to the application. A more general PEP could specify a CPRNG and then the encryption methods could use it. 3) Why only support multiples of block sizes. This is a pain. Both padding and IV (addition and removal) should be automatic. Padding for each algorithm could use a default padding mechanism, with an optional parameter allowing the selection of alternative padding mechanisms. Agreed. 4) The interface/methods should support the encryption of large data and data streams. This means that you need keep track of state in the class so that a stream of data can be encrypted in small pieces. The default behavior should allow encryption with a single method (e.g aes.encrypt("foo") ). Methods to encrypt a stream require either two or three methods (encryptFirst, encryptMiddle, encryptLast) or a parameter that indicates than the stream is not complete (e.g. a "more" flag). Either approach works, I'm partial to the latter. A wrapper class can take care of this. 5) The stream encryption, automatic IV, automatic padding could allow be handled by defining a Cipher class that provides keeps track of the state and blocking information. Block algorithms would then inherit the base encrypt and decrypt methods. The less useful basic block oriented operations would then be encrypt_block and decrypt_block methods that would be used by the Cipher class. Again, random numbers are the obstacle to automatic IV's. The rest can be handled by a simple wrapper class. 6) Not all algorithms need IV or padding ... so we might want every algorithm to provide the minimum and maximum data expansion (IV + padding). Do you have something specific in mind? How would you do this for AES? 7) Each algorithm should have a unique asci name (uniqueness could be a problem to manage) 8) OIDs should be considered ... they belong either as part of the algorithm class or will need to be collected separately. Seems easier to design them in from the start so that every algorithm knows it's OID(s). I think these start turning the PEP into a more general architectural design rather than just an API for block ciphers. Maybe that's a good thing, but if that's what we want to do, we should state the goal and face it squarely, rather than edging toward it cautiously. 9) Counter mode or other combination encryption/integrity algorithms should be build on Cipher, but have some additional methods and parameters. Recent work on 802.11i with AES-OCB or AES-CCM should serve as a good example. I suggest we give these algorithms some type of new name since they are not the same as a "Cipher". New methods (or assertions) are needed to check the integrity of the decryption process. 10) On AES-OCB and AES-CCM ... these integrity checking algorithms are also being used in a way that allows parts of the unencrypted header of PDUs to be combined into the processing. This means that that encrypting with integrity checking of headers has two parameters: ct=aes-ocb.encrypt_with_integrity("plain header","body to encrypt") (suggestions for shorter name for above method would be greatly appreciated) Too many security bugs keep getting discovered in these new modes. I think it's best to omit them unless required for some specific application (e.g. 802.11 interoperability). I'm in favor of abandoning this PEP in its current form. There's no need for these abstract interfaces to block ciphers. The only reason to have more than one algorithm for anything is to interoperate with other programs, so those programs should be identified by name before worrying about adding algorithms besides AES. I spoke with Bram a couple nights ago and proposed that instead of an API PEP, we just write an AES module, document it, and submit it to the Python library. Bram wants to support all the standard NIST modes of operation for AES. I don't especially see the need for that, but at least they're specified in a single standard, and it may be easier to get them all into the library now than to do some now and more later. Anyway, Bram says he'll draft a spec for this, so that will give us something to start from. We should also have a module that provides secure pseudorandom numbers. On Unix/Linux (including MacOS 10), this is trivial--just read the /dev/urandom device. On Windows, use the CryptGenRandom function from the Windows Crypto API (CAPI). I don't know about other systems. IMO, proposals about interchangeable ciphers, OIDs, etc. should be in a more comprehensive document that I see as a separate project from getting an AES module out the door. We should look at the Java JCA and the Microsoft CAPI to get ideas of how to do this. From phr-pycrypt at nightsong.com Sat Mar 30 01:09:18 2002 From: phr-pycrypt at nightsong.com (Paul Rubin) Date: Sat, 30 Mar 2002 00:09:18 -0000 Subject: [PYTHON-CRYPTO] aes library Message-ID: <20020330000918.15965.qmail@brouhaha.com> 1. I think CLE mode should be folded into CTR mode. Just pass an optional arg to the CTR mode constructor specifying the endianness you want. 2. I'd like to see more justification for supporting all those modes than "they're in the standard". 3. If CFB mode is supported, arbitary CFB feedback sizes should be supported (as long as the feedback size divides the block size). 4. Making the Python app handle padding is a big slowdown when the messages are short. The C library should handle padding per PKCS #5. If there's a standard for ciphertext stealing and other applications use it, then support that too. Otherwise skip it. From bram at GAWTH.COM Sat Mar 30 02:19:17 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Fri, 29 Mar 2002 17:19:17 -0800 Subject: [PYTHON-CRYPTO] aes library In-Reply-To: <20020330000918.15965.qmail@brouhaha.com> Message-ID: Paul Rubin wrote: > 1. I think CLE mode should be folded into CTR mode. Just pass an > optional arg to the CTR mode constructor specifying the endianness you > want. Good idea. It'll be a parameter in the next draft. > 2. I'd like to see more justification for supporting all those modes > than "they're in the standard". There are applications which use them, and I'd like for it to be possible for those applications to be written in pure Python. In particular, BitTorrent uses counter, and I'm scratching my own itch :-). Seriously, even without specific examples to point to, since those are the standard modes, it's reasonable to expect that each of them will be used by at least a few applications. > 3. If CFB mode is supported, arbitary CFB feedback sizes should be > supported (as long as the feedback size divides the block size). Okay, I read over the docs and think I understand CFB mode better now. I'll have an updated API for it in the next draft. > 4. Making the Python app handle padding is a big slowdown when the > messages are short. The C library should handle padding per PKCS #5. > If there's a standard for ciphertext stealing and other applications > use it, then support that too. Otherwise skip it. That only matters for *really* short files, like less than 1k. Future versions can have added support for padding modes in C, but past arguments have convinced me that noone can agree on what the standard common padding modes are, so I'd rather not open that can of worms. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From phr-pycrypt at nightsong.com Sat Mar 30 06:01:23 2002 From: phr-pycrypt at nightsong.com (Paul Rubin) Date: Sat, 30 Mar 2002 05:01:23 -0000 Subject: [PYTHON-CRYPTO] aes library Message-ID: <20020330050123.20455.qmail@brouhaha.com> The purpose of this module is *not* to be a complete library for all the standard things one might want to do with aes. Rather, it's a bare-bones library which does just enough to make it possible to implement all the standard aes stuff efficiently in 'pure' Python, with the only C calls being to this library. If it's a bare-bones library it needn't implement all the operation modes. If it's going to implement them all, it should also implement a reasonable padding mode for CBC. "That only matters for *really* short files, like less than 1k" shows a belief that the stuff being encrypted will be files, but I believe it equally likely that it will be short strings like credit card numbers or other encryption keys. past arguments have convinced me that noone can agree on what the standard common padding modes are, so I'd rather not open that can of worms. This surprises me. I thought RFC 1423 was an internet standard and section 1.1 specifies a perfectly good padding algorithm a/k/a PKCS #5. I don't see any worms. Do you know of another standard? The bigendian parameter determines whether encoding of the counter into a block to be encrypted is done big or little endian. It defaults to 1 (true). The default should be chosen so that the NIST CTR mode test vectors work consistent with ECB mode under the default. That's probably big endian but we should make sure. I think CTR mode should use the same encrypt/decrypt operation names as the other modes, even if encrypt and decrypt in CTR mode do the same thing, instead of process/processor. It's just less junk to remember. Endianness should be specified in the object constructor, not on each crypto operation. Why would anyone want to mix endianness in one crypto object? From bram at GAWTH.COM Sat Mar 30 10:25:56 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Sat, 30 Mar 2002 01:25:56 -0800 Subject: [PYTHON-CRYPTO] aes library In-Reply-To: <20020330050123.20455.qmail@brouhaha.com> Message-ID: Paul Rubin wrote: > The purpose of this module is *not* to be a complete library for > all the standard things one might want to do with aes. Rather, > it's a bare-bones library which does just enough to make it > possible to implement all the standard aes stuff efficiently in > 'pure' Python, with the only C calls being to this library. > > If it's a bare-bones library it needn't implement all the operation > modes. I've been waffling on the utility of CFB. I just did a 'vote' of mode popularity by asking google - "CBC mode" 11100 "counter mode" 6250 "ECB mode" 4390 "CFB mode" 2760 "OFB Mode" 1980 The clear winners in terms of popularity are CBC and counter, and ECB needs to be included for completeness. In search results, there are lots of mentions of the problems with CFB and suggestions for all sorts of funky variants of it. Also, while the API I suggested for using CFB is clearly the only reasonable one, it hardly gets any mention in the press, as people seem to greatly favor the moronic practice of doing a reduced block size for *every* encryption, as opposed to the reasonable but still funky practice of adjusting the block size dynamically to suit how the data is coming in. So I think I'll dump CFB. OFB is so simple though that I'm in favor of keeping it - it has the property that multiple messages can be encrypted using the same key but doesn't have the padding problems of CBC. > past arguments have convinced me that noone can agree on what the > standard common padding modes are, so I'd rather not open that can of > worms. > > This surprises me. I thought RFC 1423 was an internet standard and > section 1.1 specifies a perfectly good padding algorithm a/k/a PKCS > #5. I don't see any worms. Do you know of another standard? That's a reasonable padding algorithm, but it's far from the only one in use. Like I said, I'm not designing this API to provide a single handy encryption function for non-cryptographers to use (although one could certainly be written on top of it.) I'm designing it to make interoperation with other protocols possible in pure Python. > The bigendian parameter determines whether encoding of the counter > into a block to be encrypted is done big or little endian. It > defaults to 1 (true). > > The default should be chosen so that the NIST CTR mode test vectors > work consistent with ECB mode under the default. That's probably big > endian but we should make sure. The docs seem to imply big endian but don't talk about it in a coherent manner, I guess one has to run the actual test vectors to find out. > I think CTR mode should use the same encrypt/decrypt operation names > as the other modes, even if encrypt and decrypt in CTR mode do the > same thing, instead of process/processor. How about crypt/crypter? > Endianness should be specified in the object constructor, not on each > crypto operation. Done. I'll post an updated version of the API later. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From bram at GAWTH.COM Fri Mar 29 20:53:15 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Fri, 29 Mar 2002 11:53:15 -0800 Subject: [PYTHON-CRYPTO] aes module API Message-ID: This is a proposal for the API for a module called 'aes', which implements the standard modes of rijndael. The purpose of this module is *not* to be a complete library for all the standard things one might want to do with aes. Rather, it's a bare-bones library which does just enough to make it possible to implement all the standard aes stuff efficiently in 'pure' Python, with the only C calls being to this library. For example, ciphertext stealing in CBC mode is to be implemented using aes.CBC with a few calls at the end to aes.ECB to do the stealing part. This results in the performance benefits of using C without the API bloat of doing *everything* in C. The methods of AES are CTR, CLE, OFB, CBC, ECB, and CFB. Each of them takes a 16, 24, or 32 byte key and returns an encryption object. ECB encryption objects have two methods, encrypt and decrypt, both of which take a string whose length is a multiple of 16 bytes and return an encrypted or decrypted string of the same length. Strings of length 0 are allowed. CTR encryption objects have a method called process, which takes a counter (an integer) and a string to be encrypted. CTR is the same step for encryption and decryption, hence the single process function instead of separate encrypt and decrypt functions. process() returns a string of the same length as the one it received. The counter passed in to process() is modulo 2 ** 128, so numbers such as -1 are allowed. When the counter gets passed 2 ** 128 it should roll back to zero. The other method of CTR objects is processor(), which takes a counter and returns a function. The function takes strings and returns each of them encrypted incrementally. This is the same as using the process() method only it isn't necessary to have the entire string in memory at once. The following test should always pass - def test_concat(key, s): x = CTR(key) p1 = x.process(0, s) f = x.processor(0) assert x.process(0, s) == f(s[:3]) + f(s[3:]) CLE has the same API as CTR, only CTR is big endian and CLE is little endian. OFB has the same API as CTR except that instead of a counter it takes an iv of length 16. CBC and CFB have a similar API to OFB, except that instead of process and processor functions, they have analogous encrypt and encrypter() and also an analagous decrypt and decrypter(). In these modes incremental processing sometimes has to buffer a bit before it knows how to encrypt the next byte, so they return as long a string as they can from each call to incremental processing functions, never trailing the amount of data passed in by 16 bytes or more. Here's a quick cheat sheet to the API. What does everybody think? CTR(key) process(counter, str) processor(counter) __call__(str) CLE(key) process(counter, str) processor(counter) __call__(str) OFB(key) process(iv, str) processor(iv) __call__(str) CBC(key) encrypt(iv, str) decrypt(iv, str) encrypter(iv) __call__(str) decrypter(iv) __call__(str) ECB(key) encrypt(str) decrypt(str) CFB(key) encrypt(iv, str) decrypt(iv, str) encrypter(iv) __call__(str) decrypter(iv) __call__(str) -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From bram at GAWTH.COM Sat Mar 30 00:58:33 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Fri, 29 Mar 2002 15:58:33 -0800 Subject: [PYTHON-CRYPTO] aes module API In-Reply-To: Message-ID: I forgot to mention that all cases when a function gets passed an incorrect value (basically whenever one of the functions which takes a string gets one of an unsupported size) it should raise a ValueError. Also, does anybody know what the deal is for CFB with block sizes other than the block size of the cipher? I'm not sure what it means, so I didn't include support for it in the API. -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From Paul at CRYPTORIGHTS.ORG Sat Mar 30 21:36:24 2002 From: Paul at CRYPTORIGHTS.ORG (Paul) Date: Sat, 30 Mar 2002 12:36:24 -0800 Subject: [PYTHON-CRYPTO] aes library In-Reply-To: <20020330050123.20455.qmail@brouhaha.com> References: <20020330050123.20455.qmail@brouhaha.com> Message-ID: At 5:01 AM +0000 3/30/02, Paul Rubin wrote: > >I think CTR mode should use the same encrypt/decrypt operation names >as the other modes, even if encrypt and decrypt in CTR mode do the >same thing, instead of process/processor. It's just less junk to >remember. Endianness should be specified in the object constructor, >not on each crypto operation. Why would anyone want to mix endianness >in one crypto object? Yes. Process and processor are confusing. Integrity mode algorithms like CTR should include an assertion for integrity failures. Paul -- From Paul at CRYPTORIGHTS.ORG Sat Mar 30 21:40:56 2002 From: Paul at CRYPTORIGHTS.ORG (Paul) Date: Sat, 30 Mar 2002 12:40:56 -0800 Subject: [PYTHON-CRYPTO] aes library In-Reply-To: References: Message-ID: At 1:25 AM -0800 3/30/02, Bram Cohen wrote: > >That's a reasonable padding algorithm, but it's far from the only one in >use. Like I said, I'm not designing this API to provide a single handy >encryption function for non-cryptographers to use (although one could >certainly be written on top of it.) I'm designing it to make >interoperation with other protocols possible in pure Python. Even cryptographer would appreciate simple useable APIs. Few coders are cryptographers, so your current goals low. I suspect just a few enhancements would make AES more usable for the masses. This seems like a good thing :-) Paul -- From bram at GAWTH.COM Sat Mar 30 02:49:42 2002 From: bram at GAWTH.COM (Bram Cohen) Date: Fri, 29 Mar 2002 17:49:42 -0800 Subject: [PYTHON-CRYPTO] aes library, take 2 Message-ID: This is a proposal for the API for a module called 'aes', which implements the standard modes of rijndael. The purpose of this module is *not* to be a complete library for all the standard things one might want to do with aes. Rather, it's a bare-bones library which does just enough to make it possible to implement all the standard aes stuff efficiently in 'pure' Python, with the only C calls being to this library. For example, ciphertext stealing in CBC mode is to be implemented using aes.CBC with a few calls at the end to aes.ECB to do the stealing part. This results in the performance benefits of using C without the API bloat of doing *everything* in C. Padding modes may be added to the aes module in a backwards-compatible way in the future, but that's a can of worms to be opened at a later time. The methods of AES are ECB, CTR, OFB, CBC, and CFB. Each of them takes a 16, 24, or 32 byte key and returns an encryption object. ECB encryption objects have two methods, encrypt and decrypt, both of which take a string whose length is a multiple of 16 bytes and return an encrypted or decrypted string of the same length. Strings of length 0 are allowed. CTR encryption objects have a method called process, which takes a counter (an integer), a string to be encrypted, and an optional bigendian parameter. CTR is the same step for encryption and decryption, hence the single process function instead of separate encrypt and decrypt functions. process() returns a string of the same length as the one it received. The counter passed in to process() is modulo 2 ** 128, so numbers such as -1 are allowed. When the counter gets passed 2 ** 128 it should roll back to zero. The bigendian parameter determines whether encoding of the counter into a block to be encrypted is done big or little endian. It defaults to 1 (true). The other method of CTR objects is processor(), which takes a counter and optional bigendian parameter which defaults to 1 and returns a function. The function takes strings and returns each of them encrypted incrementally. This is the same as using the process() method only it isn't necessary to have the entire string in memory at once. The following test should always pass - def test_concat(key, s): x = CTR(key) f = x.processor(0) assert x.process(0, s) == f(s[:3]) + f(s[3:]) OFB has the same API as CTR except that instead of a counter it takes an iv of length 16, and it has no bigendian parameter. CBC has a similar API to OFB, except that instead of process and processor functions, it has analogous encrypt and encrypter() and also an analagous decrypt and decrypter(). In these modes incremental processing sometimes has to buffer a bit before it knows how to encrypt the next byte, so they return as long a string as they can from each call to incremental processing functions, never trailing the amount of data passed in by 16 bytes or more. CFB objects have encrypter() and decrypter() functions which take an IV, and both of them return a function which accepts strings and returns strings of the same length. Note that unlike CTR and OFB, CFB decryption *must* be subdivided into the exact same block lengths as it's encryption was, or it's decryption won't work. ECB(key) encrypt(str) decrypt(str) CTR(key) process(counter, str, bigendian = 1) processor(counter, bigendian = 1) __call__(str) OFB(key) process(iv, str) processor(iv) __call__(str) CBC(key) encrypt(iv, str) decrypt(iv, str) encrypter(iv) __call__(str) decrypter(iv) __call__(str) CFB(key) encrypter(iv) __call__(str) decrypter(iv) __call__(str) -Bram Cohen "Markets can remain irrational longer than you can remain solvent" -- John Maynard Keynes From Paul at CRYPTORIGHTS.ORG Sat Mar 30 21:30:44 2002 From: Paul at CRYPTORIGHTS.ORG (Paul) Date: Sat, 30 Mar 2002 12:30:44 -0800 Subject: [PYTHON-CRYPTO] aes library, take 2 In-Reply-To: References: Message-ID: At 5:49 PM -0800 3/29/02, Bram Cohen wrote: >This is a proposal for the API for a module called 'aes', which implements >the standard modes of rijndael. > >The purpose of this module is *not* to be a complete library for all the >standard things one might want to do with aes. Rather, it's a bare-bones >library which does just enough to make it possible to implement all the >standard aes stuff efficiently in 'pure' Python, with the only C calls >being to this library. > >For example, ciphertext stealing in CBC mode is to be implemented using >aes.CBC with a few calls at the end to aes.ECB to do the stealing part. >This results in the performance benefits of using C without the API bloat >of doing *everything* in C. Padding modes may be added to the aes module >in a backwards-compatible way in the future, but that's a can of worms to >be opened at a later time. > >The methods of AES are ECB, CTR, OFB, CBC, and CFB. Each of them takes a >16, 24, or 32 byte key and returns an encryption object. Why not just make each mode a class ... e.g AES-ECB, AES-CTR, etc. Modes are types of algorithms that can be build on AES and should not be methods or parameters of AES. > >ECB encryption objects have two methods, encrypt and decrypt, both of >which take a string whose length is a multiple of 16 bytes and return an >encrypted or decrypted string of the same length. Strings of length 0 are >allowed. > >CTR encryption objects have a method called process, which takes a counter >(an integer), a string to be encrypted, and an optional bigendian >parameter. CTR is the same step for encryption and decryption, hence the >single process function instead of separate encrypt and decrypt functions. >process() returns a string of the same length as the one it received. > >The counter passed in to process() is modulo 2 ** 128, so numbers such as >-1 are allowed. When the counter gets passed 2 ** 128 it should roll back >to zero. > >The bigendian parameter determines whether encoding of the counter into a >block to be encrypted is done big or little endian. It defaults to 1 >(true). > >The other method of CTR objects is processor(), which takes a counter and >optional bigendian parameter which defaults to 1 and returns a function. >The function takes strings and returns each of them encrypted >incrementally. This is the same as using the process() method only it >isn't necessary to have the entire string in memory at once. The following >test should always pass - > >def test_concat(key, s): > x = CTR(key) > f = x.processor(0) > assert x.process(0, s) == f(s[:3]) + f(s[3:]) > >OFB has the same API as CTR except that instead of a counter it takes an >iv of length 16, and it has no bigendian parameter. > >CBC has a similar API to OFB, except that instead of process and processor >functions, it has analogous encrypt and encrypter() and also an analagous >decrypt and decrypter(). In these modes incremental processing sometimes >has to buffer a bit before it knows how to encrypt the next byte, so they >return as long a string as they can from each call to incremental >processing functions, never trailing the amount of data passed in by 16 >bytes or more. > >CFB objects have encrypter() and decrypter() functions which take an IV, >and both of them return a function which accepts strings and returns >strings of the same length. Note that unlike CTR and OFB, CFB decryption >*must* be subdivided into the exact same block lengths as it's encryption >was, or it's decryption won't work. > >ECB(key) > encrypt(str) > decrypt(str) > >CTR(key) > process(counter, str, bigendian = 1) > processor(counter, bigendian = 1) > __call__(str) > >OFB(key) > process(iv, str) > processor(iv) > __call__(str) > >CBC(key) > encrypt(iv, str) > decrypt(iv, str) > encrypter(iv) > __call__(str) > decrypter(iv) > __call__(str) > >CFB(key) > encrypter(iv) > __call__(str) > decrypter(iv) > __call__(str) > > >-Bram Cohen > >"Markets can remain irrational longer than you can remain solvent" > -- John Maynard Keynes --