Re: [Mailman-Developers] GSoc - Requirement from Mentor to complete the project
Abhilash Raj writes:
Let me divide the project in a few pieces so that each can be discussed upon separately.
This is a good idea, but you should take them up one at a time, unless you have a good approach and are expecting "sounds good, get started" as the reply to that point.
- Firstly a utility to encrypt or decrypt the message. Well i found [python-gnupg][1] for this purpose and would try to write a wrapper around it for the use by mailman. But I found another option for it [GnupgInterface][2]. GnupgInterface was used in the [mailman-pgp-smime][3] patch for mailman and also has options to sign and encrypt in one call of a function( unlike python-gnupg ). If anyone has ever used any of these two would you please suggest which one is better?
This isn't first. Don't be so eager to write code when you have not stated the requirements with any precision.
*First* you need to describe the life of a message from a thought in the sender's mind until it hits the receiver's eyes. (It could actually be somewhat shorter than that, but these endpoints ensure you'll get everything we need somewhere in between.) Which steps are required for every message? Which are optional, depending on the list policy and/or user choices? Which are implemented in Mailman? Which in MTAs/MDAs? Which in MUAs?
You also need to decide what threats this process is suppose to protect the users from.
The combination of these two will determine what Mailman needs to be able to do with incoming and outgoing posts. Then you need to see what Mailman already does perfectly, what needs to be modified, and what needs a new implementation.
This description of requirements doesn't need to be as authoritative as Scripture, but we need something fairly detailed to start with.
Once that's done, we can talk about implementing crypto operations. But I don't think it matters much which module you start with. (Why not? The answer is a general concept of software engineering.)
- The point of encryption and decryption in the various queues. I was of the opinion that the message is decrypted as soon as it enters the IN queue and while its about to leave the queue it is encrypted with a symmetric key algorithm using the list's secret key. And then it is subsequently decrypted in the next queue and finally in the OUTGOING queue it is signed and encrypted with each user's pub-key. Any suggestions about this?
I think the concerns about decrypted material hitting disk are valid.
Therefore I would recommend that you avoid decrypting until necessary. (It may not even be necessary. When? What exactly do I mean by "not decrypting"? This requires a good understand of the OpenPGP format, as well as Mailman queue processing. Don't hurry, feel free to ask questions -- as long as they don't amount to "I don't know, tell me"!)
Consider this flow of message from one user to all list-subscribers:
A user sends a message( signed as well as encrypted )which as soon as received by mailman is first queued in "IN" queue.
When the incoming runner wakes up the message is checked if it is for owners, posting or encrypted.An encrypted chain is created for encrypted messages.
Now according to the use case there can be two encrypted chain:
Where the posters are fully trusted and thus this rules only checks the signature of a message. If a message signed by a member is received it is sent to pipeline queue in a encrypted form for further processing.
Where the posters are not fully trusted. In this case there needs to be various checks on decrypted message like current default-posting-chain.
The messaged is moved in between queues in the encrypted form encrypted by a symmetric key algorithm which uses list's secret key to encrypt the message.
When the pipeline handler wakes up it decides which pipeline the message is to passed through. In this case an encrypted_posting_pipeline is created to process an encrypted message.
This encrypted_posting_pipeline is used to keep the message in plain-text for a minimal time. The message is decrypted, headers and footers are added and the message is signed by list's pub-key. It is then again (symmetrically) encrypted and stored in the outgoing queue.
The outgoing runner personalizes each message by encrypting it with each user's pub key and forwards it to MTA using VERP(Variable Envelop Return Path).
Now consider the threat models it addresses.
Firstly this verifies the integrity of the message. This is done by verification of signature and matching that sender's address indeed matches the address in signature so that you can sure that message was indeed sent by the sender.
Secondly this maintains the confidentiality of the message. Even if the message is intercepted it can only be read by the one whom it is addressed to.
There are the threats this does not address:
If the keys/system/MUA of a user sending or receiving the message is compromised or contains a virus, no assurance can be provided about the integrity or confidentiality of the message.
On Tue, May 7, 2013 at 8:27 AM, Stephen J. Turnbull <stephen@xemacs.org> wrote:
Abhilash Raj writes:
Let me divide the project in a few pieces so that each can be discussed upon separately.
This is a good idea, but you should take them up one at a time, unless you have a good approach and are expecting "sounds good, get started" as the reply to that point.
- Firstly a utility to encrypt or decrypt the message. Well i found [python-gnupg][1] for this purpose and would try to write a wrapper around it for the use by mailman. But I found another option for it [GnupgInterface][2]. GnupgInterface was used in the [mailman-pgp-smime][3] patch for mailman and also has options to sign and encrypt in one call of a function( unlike python-gnupg ). If anyone has ever used any of these two would you please suggest which one is better?
This isn't first. Don't be so eager to write code when you have not stated the requirements with any precision.
*First* you need to describe the life of a message from a thought in the sender's mind until it hits the receiver's eyes. (It could actually be somewhat shorter than that, but these endpoints ensure you'll get everything we need somewhere in between.) Which steps are required for every message? Which are optional, depending on the list policy and/or user choices? Which are implemented in Mailman? Which in MTAs/MDAs? Which in MUAs?
You also need to decide what threats this process is suppose to protect the users from.
The combination of these two will determine what Mailman needs to be able to do with incoming and outgoing posts. Then you need to see what Mailman already does perfectly, what needs to be modified, and what needs a new implementation.
This description of requirements doesn't need to be as authoritative as Scripture, but we need something fairly detailed to start with.
Once that's done, we can talk about implementing crypto operations. But I don't think it matters much which module you start with. (Why not? The answer is a general concept of software engineering.)
- The point of encryption and decryption in the various queues. I was of the opinion that the message is decrypted as soon as it enters the IN queue and while its about to leave the queue it is encrypted with a symmetric key algorithm using the list's secret key. And then it is subsequently decrypted in the next queue and finally in the OUTGOING queue it is signed and encrypted with each user's pub-key. Any suggestions about this?
I think the concerns about decrypted material hitting disk are valid.
Therefore I would recommend that you avoid decrypting until necessary. (It may not even be necessary. When? What exactly do I mean by "not decrypting"? This requires a good understand of the OpenPGP format, as well as Mailman queue processing. Don't hurry, feel free to ask questions -- as long as they don't amount to "I don't know, tell me"!)
-- Abhilash Raj
As i about this, i wonder if the OpenPGP integration project shouldn't be broken into two phases, so that the tricky nuances could be handled more simply.
I'm imagining a first phase ("message authentication") would not expect or handle encrypted messages, but would just use cryptographic signatures to verify the authenticity of origin of the incoming message, enhancing (and/or replacing) some of the current checks mailman uses to decide if a message should be passed on to the list.
Then the second phase ("encrypted messages") would address the possibility of having encrypted message content.
If you can get the "message authentication" phase done properly, you'll have a good argument that your keyring management is effective, without getting sidetracked into issues of exactly where the content of the message might leak (since the messages aren't encrypted at all).
When the first phase is done, you could focus on the "encrypted messages" phase, based on the assumption that you have decent keyring management, and handle the peculiarities and concerns there separately.
I'm not a mailman developer; but I'd be curious to hear what mailman devs think of this breakdown.
More comments and questions below, based on the workflow you've outlined that mixes "message authentication" with "encrypted messages" and tries to solve them all at once:
On 05/09/2013 01:31 AM, Abhilash Raj wrote:
Consider this flow of message from one user to all list-subscribers:
- A user sends a message( signed as well as encrypted )which as soon as received by mailman is first queued in "IN" queue.
who does the sender encrypt the message to? are we talking about a public key associated with the list? how does the sender learn of that public key?
- When the incoming runner wakes up the message is checked if it is for owners, posting or encrypted.An encrypted chain is created for encrypted messages.
what if it is an encrypted message sent to "owners"? how does that get handled?
- Now according to the use case there can be two encrypted chain:
- Where the posters are fully trusted and thus this rules only checks the signature of a message. If a message signed by a member is received it is sent to pipeline queue in a encrypted form for further processing.
when you say "fully trusted" it might be worth saying what they are trusted for. I think here you mean "fully trusted to post messages without moderation".
What happens if the message cannot be decrypted by the mailing list and the signature is inside the encryption? does the message bounce?
what if the decryption succeeds but there is no message signature inside it?
what if the signature appears to be valid, but was made 3 years ago (or in the future)?
what if the signature appears to be valid, and recent, but is the same signature that was used on a previous e-mail to the list?
what if the signature is cryptographically valid, but the list member's key has expired?
what if only part of the message is signed?
- Where the posters are not fully trusted. In this case there needs to be various checks on decrypted message like current default-posting-chain.
One common thing that might happen here is that the message is being held for moderation. if the message is encrypted, how is a moderator supposed to review it and make a decision?
- The messaged is moved in between queues in the encrypted form encrypted by a symmetric key algorithm which uses list's secret key to encrypt the message.
this doesn't make sense to me. when i hear "list's secret key" i tend to think people are talking about asymmetric encryption, but it seems here that you're talking about symmetric encryption. Maybe it would help to enumerate exactly what secrets each entity holds (and maybe what public data is associated with those secrets, in the case of the public/private keypairs used by asymmetric crypto). If you give those secrets names, you can refer to them more concisely and precisely.
- When the pipeline handler wakes up it decides which pipeline the message is to passed through. In this case an encrypted_posting_pipeline is created to process an encrypted message.
which case is "this case" are we talking here about every encrypted message or just a subset of them?
- This encrypted_posting_pipeline is used to keep the message in plain-text for a minimal time. The message is decrypted, headers and footers are added and the message is signed by list's pub-key. It is then again (symmetrically) encrypted and stored in the outgoing queue.
public keys don't sign messages; secret keys can sign messages, and public keys are used to verify those signatures.
what happens to the original sender's signature? is it stripped? or does it remain, nested within the list's signature? How are list members expected to verify these signatures?
i'm again confused by the symmetrical encryption proposal here.
- The outgoing runner personalizes each message by encrypting it with each user's pub key and forwards it to MTA using VERP(Variable Envelop Return Path).
What happens if a user's public key is found to be expired or revoked? How does mailman become aware of revocations, of extensions of the expiration date, or of new keys or certificates for any given user?
i hope these questions are useful,
--dkg
On 13-05-09 11:17 AM, Daniel Kahn Gillmor wrote:
I'm imagining a first phase ("message authentication") would not expect or handle encrypted messages, but would just use cryptographic signatures to verify the authenticity of origin of the incoming message, enhancing (and/or replacing) some of the current checks mailman uses to decide if a message should be passed on to the list.
Then the second phase ("encrypted messages") would address the possibility of having encrypted message content.
I'm not a mailman developer; but I'd be curious to hear what mailman devs think of this breakdown.
Nice timing! This is *exactly* what a few of Mailman devs who are awake were talking about on IRC an hour or two ago. Re-organizing like this would be a great way to make sure that openPGP as a GSoC project can have a useful but smaller scale first goal, leaving full encryption as more of a stretch goal, which might work better for the GSoC timeline.
Terri
On Thu, May 9, 2013 at 10:47 PM, Daniel Kahn Gillmor <dkg@fifthhorseman.net> wrote:
As i about this, i wonder if the OpenPGP integration project shouldn't be broken into two phases, so that the tricky nuances could be handled more simply.
I'm imagining a first phase ("message authentication") would not expect or handle encrypted messages, but would just use cryptographic signatures to verify the authenticity of origin of the incoming message, enhancing (and/or replacing) some of the current checks mailman uses to decide if a message should be passed on to the list.
Then the second phase ("encrypted messages") would address the possibility of having encrypted message content.
If you can get the "message authentication" phase done properly, you'll have a good argument that your keyring management is effective, without getting sidetracked into issues of exactly where the content of the message might leak (since the messages aren't encrypted at all).
When the first phase is done, you could focus on the "encrypted messages" phase, based on the assumption that you have decent keyring management, and handle the peculiarities and concerns there separately.
I'm not a mailman developer; but I'd be curious to hear what mailman devs think of this breakdown.
More comments and questions below, based on the workflow you've outlined that mixes "message authentication" with "encrypted messages" and tries to solve them all at once:
After the Barry's comment on my proposal I decided to cut down the proposal to implement use of OpenPGP signatures for posting privileges instead of both signed and encrypted list. Most of the infrastructure for encrypted list will be created along with this project so that later on it can also be implemented easily.
Though I would like to continue this discussion on encrypted list for future reference. Some of these questions are already answered in my proposal here[1].
On 05/09/2013 01:31 AM, Abhilash Raj wrote:
Consider this flow of message from one user to all list-subscribers:
- A user sends a message( signed as well as encrypted )which as soon as received by mailman is first queued in "IN" queue.
who does the sender encrypt the message to? are we talking about a public key associated with the list? how does the sender learn of that public key?
Sender encrypts the message to list's pub key. List's pub-private keypair is uploaded by the list's owner when he creates the list. Subscribers gets to download the list's pub-key when he subscribes to it.
- When the incoming runner wakes up the message is checked if it is for owners, posting or encrypted.An encrypted chain is created for encrypted messages.
what if it is an encrypted message sent to "owners"? how does that get handled?
So for both regular posting and owners we can add separate chains say encypted_posting, encrypted_owners.
Now according to the use case there can be two encrypted chain:
- Where the posters are fully trusted and thus this rules only checks the signature of a message. If a message signed by a member is received it is sent to pipeline queue in a encrypted form for further processing.
when you say "fully trusted" it might be worth saying what they are trusted for. I think here you mean "fully trusted to post messages without moderation".
Yes.
What happens if the message cannot be decrypted by the mailing list and the signature is inside the encryption? does the message bounce?
Yes, it would bounce.
what if the decryption succeeds but there is no message signature inside it?
In that case a notification is sent to the sender with only the headers of the decrypted message that the signature of the message does not match. If it is not sent by him he is asked to ignore the notification.
what if the signature appears to be valid, but was made 3 years ago (or in the future)?
Is that a problem if he has not revoked his signature yet?
what if the signature appears to be valid, and recent, but is the same signature that was used on a previous e-mail to the list?
Can there be same signatures for different messages? I think the digital signature is created by encrypting the message-hash, so if the contents of the message are changed then signature cannot be valid.
what if the signature is cryptographically valid, but the list member's key has expired?
Well a expired key cannot be used to sign messages. If this situation happens this means it was signed in past some time ago. We can bounce the message and notify the user about this.
what if only part of the message is signed? My guess is that you are referring to inline-pgp. I read this article which talks about the inline-pgp as a bad messaging format. I think we should not support this. So if the message is partially signed it should probably bounce back.
2) Where the posters are not fully trusted. In this case there needs tobe various checks on decrypted message like current default-posting-chain.
One common thing that might happen here is that the message is being held for moderation. if the message is encrypted, how is a moderator supposed to review it and make a decision?
Yes this might be an issue, will have to think for something for this.
- The messaged is moved in between queues in the encrypted form encrypted by a symmetric key algorithm which uses list's secret key to encrypt the message.
this doesn't make sense to me. when i hear "list's secret key" i tend to think people are talking about asymmetric encryption, but it seems here that you're talking about symmetric encryption. Maybe it would help to enumerate exactly what secrets each entity holds (and maybe what public data is associated with those secrets, in the case of the public/private keypairs used by asymmetric crypto). If you give those secrets names, you can refer to them more concisely and precisely.
Sorry to sound so confusing. What I actually meant was that we use public-private key encryption via OpenPGP so send and receive signed and encrypted messages. Now each list has a public-private keypair which the list owner uploads when he creates the list.
During processing of the message inside mailman it is decrypted and sometimes stored on disk as python pickles in various queues which is bad for security reasons. I propose that when it is being stored on disk it can be encrypted using a symmetric key encryption algorithm so that only one who has access to list's secret key can decrypt it. When the message is being sent to the subscribers it is encrypted with the user's pub-key so that only he can decrypt it.
- When the pipeline handler wakes up it decides which pipeline the message is to passed through. In this case an encrypted_posting_pipeline is created to process an encrypted message.
which case is "this case" are we talking here about every encrypted message or just a subset of them?
Currently pipeline runner( not handler, typo last time ) decides which pipeline will process the message. For the encrypted message we add another pipeline encrypted_posting_pipeline to process all encrypted message that are to be posted. For message to the owners we can have another different pipeline.
- This encrypted_posting_pipeline is used to keep the message in plain-text for a minimal time. The message is decrypted, headers and footers are added and the message is signed by list's pub-key. It is then again (symmetrically) encrypted and stored in the outgoing queue.
public keys don't sign messages; secret keys can sign messages, and public keys are used to verify those signatures.
Yes, sorry again a typo!
what happens to the original sender's signature? is it stripped? or does it remain, nested within the list's signature? How are list members expected to verify these signatures?
The original sender's signature is stripped. I guess the list subscribers just have to trust that the list owner. We cannot expect each member of the list to have every other member's pub-key to separately verify himself that the message is indeed from the sender.
i'm again confused by the symmetrical encryption proposal here.
- The outgoing runner personalizes each message by encrypting it with each user's pub key and forwards it to MTA using VERP(Variable Envelop Return Path).
What happens if a user's public key is found to be expired or revoked? How does mailman become aware of revocations, of extensions of the expiration date, or of new keys or certificates for any given user?
If the user has his key uploaded to a public key server and wants( he will be given option of whether ot not to) mailman to check , mailman can periodically check for revocation and expiration of the key. Though i don't know how will that be done.
New keys will be added by user only from postorius after a confirmation link is sent to him on the address the key verifies.
i hope these questions are useful,
Yes, I had not though about revocation and expiration issues. Thanks for it.
--dkg
Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/raj.abhilash1%40gm...
Security Policy: http://wiki.list.org/x/QIA9
[1] : https://gist.github.com/maxking/5455462
-- Abhilash Raj
participants (4)
-
Abhilash Raj -
Daniel Kahn Gillmor -
Stephen J. Turnbull -
Terri Oda