opportunistically encrypted mailman lists with Autocrypt
Hi all,
maybe some of you know me for my works on pypy, tox or pytest but this mail will be about something else ...
In the last year i co-instigated a new opportunistic mail encryption effort called Autocrypt (https://autocrypt.org). With Autocrypt Level 1, mail clients (e.g. enigmail, K-9 mail, Delta.chat, ...) send and parse public keys in "Autocrypt" e-mail headers and offer single-click encryption. Releases are upcoming in spring 2018, we have been doing fun and well-received user-testing sessions already with in-development versions.
In 2018, I'd like to tackle basic integration of Mailman and Autocrypt, to achieve opportunistically encrypted mailing lists. The main idea is to grow a mailman mode/plugin to:
have mailman lists maintain a public pgp identity that is added through Autocrypt headers to outgoing mails.
have mailman keep track of "incoming" autocrypt keys and decrypt incoming mails if they are encrypted.
encrypt to those subscribers where we have a proper Autocrypt key
Code wise, i'd like to tackle this based on the the new and evolving (and quite unpublic so far) "muacrypt" project: https://muacrypt.readthedocs.io
From looking at archives and the GSOC idea page i see there were related efforts and ideas. Are there pointers to draft implementations that are still somewhat up to date (with mm3)?
Also, i am considering to submit a project proposal for the Mozilla Mission Partners program which would include a few more things ... OnionMX routing for postfix, and documentation on how to setup a best-practise MM3 mail instance, and maybe organizing a sprint around the mentioned topics.
collaboration welcome'ly yours,
holger
P.S.: i had discussed mm-encryption with Barry 2-3 years ago and feel now, with Autocrypt getting out the door, it's all becoming more feasible.
Hi again,
To be a bit clearer wrt my questions:
are there any active efforts wrt mailman and encryption? any "post mortems" on former ones?
are some of you interested in mailman+encryption efforts?
I btw expect autocrypt integration to be significantly simpler than previous efforts that dealt with keyservers IISIC.
holger
On Sun, Jan 21, 2018 at 06:47 +0100, holger krekel wrote:
Hi all,
maybe some of you know me for my works on pypy, tox or pytest but this mail will be about something else ...
In the last year i co-instigated a new opportunistic mail encryption effort called Autocrypt (https://autocrypt.org). With Autocrypt Level 1, mail clients (e.g. enigmail, K-9 mail, Delta.chat, ...) send and parse public keys in "Autocrypt" e-mail headers and offer single-click encryption. Releases are upcoming in spring 2018, we have been doing fun and well-received user-testing sessions already with in-development versions.
In 2018, I'd like to tackle basic integration of Mailman and Autocrypt, to achieve opportunistically encrypted mailing lists. The main idea is to grow a mailman mode/plugin to:
have mailman lists maintain a public pgp identity that is added through Autocrypt headers to outgoing mails.
have mailman keep track of "incoming" autocrypt keys and decrypt incoming mails if they are encrypted.
encrypt to those subscribers where we have a proper Autocrypt key
Code wise, i'd like to tackle this based on the the new and evolving (and quite unpublic so far) "muacrypt" project: https://muacrypt.readthedocs.io
From looking at archives and the GSOC idea page i see there were related efforts and ideas. Are there pointers to draft implementations that are still somewhat up to date (with mm3)?
Also, i am considering to submit a project proposal for the Mozilla Mission Partners program which would include a few more things ... OnionMX routing for postfix, and documentation on how to setup a best-practise MM3 mail instance, and maybe organizing a sprint around the mentioned topics.
collaboration welcome'ly yours,
holger
P.S.: i had discussed mm-encryption with Barry 2-3 years ago and feel now, with Autocrypt getting out the door, it's all becoming more feasible.
Mailman-Developers mailing list Mailman-Developers@python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/holger%40merlinux...
Security Policy: http://wiki.list.org/x/QIA9
Hi Holger. Very cool initiative. I just skimmed the specs for now, but I do like how it takes an opportunistic approach to key management, in order to simplify the UX and increase adoption.
On Jan 21, 2018, at 00:47, holger krekel <holger@merlinux.eu> wrote:
maybe some of you know me for my works on pypy, tox or pytest but this mail will be about something else …
Indeed. If autocrypt becomes even a fraction as amazing, joyful to use, and essential to our toolkits as those others that you work on, it will undoubtedly succeed.
In the last year i co-instigated a new opportunistic mail encryption effort called Autocrypt (https://autocrypt.org). With Autocrypt Level 1, mail clients (e.g. enigmail, K-9 mail, Delta.chat, ...) send and parse public keys in "Autocrypt" e-mail headers and offer single-click encryption. Releases are upcoming in spring 2018, we have been doing fun and well-received user-testing sessions already with in-development versions.
Have there been any talks about creating plugins for commercial MUAs like Mail.app, Postbox, and Outlook? Do you have plans for webmail clients like Gmail or Outlook?
In 2018, I'd like to tackle basic integration of Mailman and Autocrypt, to achieve opportunistically encrypted mailing lists. The main idea is to grow a mailman mode/plugin to:
have mailman lists maintain a public pgp identity that is added through Autocrypt headers to outgoing mails.
have mailman keep track of "incoming" autocrypt keys and decrypt incoming mails if they are encrypted.
encrypt to those subscribers where we have a proper Autocrypt key
Code wise, i'd like to tackle this based on the the new and evolving (and quite unpublic so far) "muacrypt" project: https://muacrypt.readthedocs.io
From looking at archives and the GSOC idea page i see there were related efforts and ideas. Are there pointers to draft implementations that are still somewhat up to date (with mm3)?
I’m not really sure what the state of those previous efforts are, i.e. whether they’d be applicable to the current code base without a serious rebase. ISTM that at least some of the changes in general would still be applicable, e.g. model changes to manage keys.
Also, i am considering to submit a project proposal for the Mozilla Mission Partners program which would include a few more things ... OnionMX routing for postfix, and documentation on how to setup a best-practise MM3 mail instance, and maybe organizing a sprint around the mentioned topics.
Cool. I think for best-practices in standing up an instance, you should definitely look at Abhilash’s docker images. A sprint would be fun for sure. Are you thinking Pycon US 2018 or elsewhere?
collaboration welcome'ly yours,
holger
P.S.: i had discussed mm-encryption with Barry 2-3 years ago and feel now, with Autocrypt getting out the door, it's all becoming more feasible.
Indeed; having a specification and library for many of the details would help quite a bit.
I’ll say this; there have been countless discussions on the mailing list about whether list traffic encryption really helps or not, how easy it would be to subvert, and other related topics. I still think it’s an interesting and useful feature that will satisfy enough use cases to make it worth working on. Then we have to ask whether this is of general utility to make it a built-in Mailman 3 feature. That decision can certainly be deferred because you can probably get pretty far with the much better plugin support in Mailman 3. I’m sure you could do a pretty good PoC as a plugin.
Cheers, -Barry
Hi Barry, all,
thanks for your encouragement and feedback! more inline ... also would appreciate some quick feedback on the rough technical design at the end.
On Tue, Jan 23, 2018 at 11:51 -0500, Barry Warsaw wrote:
In the last year i co-instigated a new opportunistic mail encryption effort called Autocrypt (https://autocrypt.org). With Autocrypt Level 1, mail clients (e.g. enigmail, K-9 mail, Delta.chat, ...) send and parse public keys in "Autocrypt" e-mail headers and offer single-click encryption. Releases are upcoming in spring 2018, we have been doing fun and well-received user-testing sessions already with in-development versions.
Have there been any talks about creating plugins for commercial MUAs like Mail.app, Postbox, and Outlook? Do you have plans for webmail clients like Gmail or Outlook?
We are working with Mailvelope and Roundcube developers, also have contacts to several others. It's going to be a long game ... Getting 3-4 MUAs to seemlessly work and have happy users will hopefully convince others to join efforts. Autocrypt itself has no funding and does not employ anyone at this point.
Also, i am considering to submit a project proposal for the Mozilla Mission Partners program which would include a few more things ... OnionMX routing for postfix, and documentation on how to setup a best-practise MM3 mail instance, and maybe organizing a sprint around the mentioned topics.
Cool. I think for best-practices in standing up an instance, you should definitely look at Abhilash’s docker images. A sprint would be fun for sure. Are you thinking Pycon US 2018 or elsewhere?
I am thinking of one in Freiburg (germany, black forest) where i live :)
I’ll say this; there have been countless discussions on the mailing list about whether list traffic encryption really helps or not, how easy it would be to subvert, and other related topics. I still think it’s an interesting and useful feature that will satisfy enough use cases to make it worth working on. Then we have to ask whether this is of general utility to make it a built-in Mailman 3 feature. That decision can certainly be deferred because you can probably get pretty far with the much better plugin support in Mailman 3. I’m sure you could do a pretty good PoC as a plugin.
Indeed, i guess a plugin should get us >90% there. Here are my current technical considerations in a quick list:
Autocrypt L1 specifies how to generate an Autocrypt key, transfer and parse public keys and settings through headers of regular e-mails.
Mailman is to create and maintain a per-list Autocrypt account (managed through "muacrypt") which keeps track of subscriber's encryption settings by parsing incoming mail messages. muacrypt in turn uses "gpg" or "gpg2" and would maintain keyring/account data on a per-list basis. muacrypt also implements mime-encryption and decryption and needs to intercept incoming/outgoing mails. Can a plugin modify outgoing messages on a per-recipient basis?
No special interface is needed on the mailing list web page maybe except from enabling/disabling the plugin/support.
With the current Autocrypt specification, a mailman list will need to have the DMARC-mitigation "replace-from" action enabled to a) allow the list's public key to be added to outgoing mails in an Autocrypt compliant way b) send properly signed and encrypted mails to those subscribers which are Autocrypt-capable and have a "mutual" prefer-encrypt settings (see Autocrypt spec for details "prefer-encrypt"). If we get this to work nicely i think we can see about advancing the Autocrypt spec to support other ML modes (i.e. not replacing From).
For a mixed set of subscribers (some with and some without Autocrypt support) mailman will send out encrypted and unencrypted mails respectively. The status of the group might be added to a group's footer so that subscribers know about the group's security status (and get incentivized to upgrade their e-mail program).
Does that sound sensible? And achievable through a plugin, leaving the core (largely) untouched?
holger
Hi all!
On 01/25/2018 12:21 PM, holger krekel wrote:
Indeed, i guess a plugin should get us >90% there. Here are my current technical considerations in a quick list:
Autocrypt L1 specifies how to generate an Autocrypt key, transfer and parse public keys and settings through headers of regular e-mails.
Mailman is to create and maintain a per-list Autocrypt account (managed through "muacrypt") which keeps track of subscriber's encryption settings by parsing incoming mail messages. muacrypt in turn uses "gpg" or "gpg2" and would maintain keyring/account data on a per-list basis. muacrypt also implements mime-encryption and decryption and needs to intercept incoming/outgoing mails. Can a plugin modify outgoing messages on a per-recipient basis?
Yes a plugin can do that, although it has to go through some hoops as when I was making the delivery part of the plugin support I didn't want to change Mailman's delivery mechanisms too much, which means a plugin doing this will have some code duplication/boilerplate. As there are only 2 plugins currently(mailman-pgp and mailman-rest-events), I think the plugin API could still evolve.
If you would like to use a Python OpenPGP implementation you could look at PGPy and how I used it in mailman-pgp.
- No special interface is needed on the mailing list web page maybe except from enabling/disabling the plugin/support.
Plugin configuration is done through the Mailman configuration and those are read-only through the REST interface. However a plugin might supply it's own REST endpoints for example for per-list setup/configuration.
With the current Autocrypt specification, a mailman list will need to have the DMARC-mitigation "replace-from" action enabled to a) allow the list's public key to be added to outgoing mails in an Autocrypt compliant way b) send properly signed and encrypted mails to those subscribers which are Autocrypt-capable and have a "mutual" prefer-encrypt settings (see Autocrypt spec for details "prefer-encrypt"). If we get this to work nicely i think we can see about advancing the Autocrypt spec to support other ML modes (i.e. not replacing From).
For a mixed set of subscribers (some with and some without Autocrypt support) mailman will send out encrypted and unencrypted mails respectively. The status of the group might be added to a group's footer so that subscribers know about the group's security status (and get incentivized to upgrade their e-mail program).
Does that sound sensible? And achievable through a plugin, leaving the core (largely) untouched?
I definitely think so.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi Jan,
On Wed, Jan 31, 2018 at 15:28 +0100, Jan Jancar wrote:
Hi all!
On 01/25/2018 12:21 PM, holger krekel wrote:
Indeed, i guess a plugin should get us >90% there. Here are my current technical considerations in a quick list:
Autocrypt L1 specifies how to generate an Autocrypt key, transfer and parse public keys and settings through headers of regular e-mails.
Mailman is to create and maintain a per-list Autocrypt account (managed through "muacrypt") which keeps track of subscriber's encryption settings by parsing incoming mail messages. muacrypt in turn uses "gpg" or "gpg2" and would maintain keyring/account data on a per-list basis. muacrypt also implements mime-encryption and decryption and needs to intercept incoming/outgoing mails. Can a plugin modify outgoing messages on a per-recipient basis?
Yes a plugin can do that, although it has to go through some hoops as when I was making the delivery part of the plugin support I didn't want to change Mailman's delivery mechanisms too much, which means a plugin doing this will have some code duplication/boilerplate. As there are only 2 plugins currently(mailman-pgp and mailman-rest-events), I think the plugin API could still evolve.
sounds good!
If you would like to use a Python OpenPGP implementation you could look at PGPy and how I used it in mailman-pgp.
It's under consideration here: https://github.com/hpk42/muacrypt/issues/32 Are your experiences with pgpy indicating it's compatible with enigmail and k9-mail? (see the questions on that issue)
- No special interface is needed on the mailing list web page maybe except from enabling/disabling the plugin/support.
Plugin configuration is done through the Mailman configuration and those are read-only through the REST interface. However a plugin might supply it's own REST endpoints for example for per-list setup/configuration.
I guess read-only REST would allow for a command line interface for debugging or other low-level configuration wrt to autocrypt key status for peers. Can a plugin add per-list configuration options (enable/disable, maybe a choice between 2-3 policies?)
With the current Autocrypt specification, a mailman list will need to have the DMARC-mitigation "replace-from" action enabled to a) allow the list's public key to be added to outgoing mails in an Autocrypt compliant way b) send properly signed and encrypted mails to those subscribers which are Autocrypt-capable and have a "mutual" prefer-encrypt settings (see Autocrypt spec for details "prefer-encrypt"). If we get this to work nicely i think we can see about advancing the Autocrypt spec to support other ML modes (i.e. not replacing From).
For a mixed set of subscribers (some with and some without Autocrypt support) mailman will send out encrypted and unencrypted mails respectively. The status of the group might be added to a group's footer so that subscribers know about the group's security status (and get incentivized to upgrade their e-mail program).
Does that sound sensible? And achievable through a plugin, leaving the core (largely) untouched?
I definitely think so.
thanks a bunch for your feedback and links.
holger
If you would like to use a Python OpenPGP implementation you could look at PGPy and how I used it in mailman-pgp.
It's under consideration here: https://github.com/hpk42/muacrypt/issues/32 Are your experiences with pgpy indicating it's compatible with enigmail and k9-mail? (see the questions on that issue)
PGPy is a quite complete OpenPGP(RFC4880) implementation, its support table shows that:
https://pgpy.readthedocs.io/en/latest/progress.html
The unsupported packets are very rarely used nowadays and are only really produced by very old PGP clients, afaik. PGPy also has a quite extensive test suite that works with gpg internally.
I suggest you look at PGPy issue tracker to see what it lacks currently, the most painful issue I think is the missing support for writing partial length packets. It can read them just well but not output them.
- No special interface is needed on the mailing list web page maybe except from enabling/disabling the plugin/support.
Plugin configuration is done through the Mailman configuration and those are read-only through the REST interface. However a plugin might supply it's own REST endpoints for example for per-list setup/configuration.
I guess read-only REST would allow for a command line interface for debugging or other low-level configuration wrt to autocrypt key status for peers. Can a plugin add per-list configuration options (enable/disable, maybe a choice between 2-3 policies?)
Yes definitely, the configuration will be handled completely inside the pkugin. I'm thinking along the lines of:
Expose a custom REST endpoint for per-list Autocrypt configuration, that can be read-write, however is only protected by one global REST user-password pair. That will be accessed by an Autocrypt Django app, similarly to how I implemented configuration for mailman-pgp in django-pgpmailman. That app can be then run alongside Postorius and HyperKitty which provide Mailman's configuration and archives. So this gives list admins a simple web UI for per-list Autocrypt configuration.
Provide a CLI command component which can manage the per-list Autocrypt configuration of the Mailman instance locally.
That should make per-list configuration of Autocrypt quite nice(I did mailman-pgp per-list config exactly this way). You can also read more on my blog, which contains my GSoC reports and goes into a bit more depth, https://neuromancer.sk/article/18 and other GSoC articles.
With the current Autocrypt specification, a mailman list will need to have the DMARC-mitigation "replace-from" action enabled to a) allow the list's public key to be added to outgoing mails in an Autocrypt compliant way b) send properly signed and encrypted mails to those subscribers which are Autocrypt-capable and have a "mutual" prefer-encrypt settings (see Autocrypt spec for details "prefer-encrypt"). If we get this to work nicely i think we can see about advancing the Autocrypt spec to support other ML modes (i.e. not replacing From).
For a mixed set of subscribers (some with and some without Autocrypt support) mailman will send out encrypted and unencrypted mails respectively. The status of the group might be added to a group's footer so that subscribers know about the group's security status (and get incentivized to upgrade their e-mail program).
Does that sound sensible? And achievable through a plugin, leaving the core (largely) untouched?
I definitely think so.
thanks a bunch for your feedback and links.
holger
Glad that I could help!
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi Jan,
On Wed, Jan 31, 2018 at 19:00 +0100, Jan Jancar wrote:
If you would like to use a Python OpenPGP implementation you could look at [PGPy] and how I used it in mailman-pgp.
It's under consideration here: https://github.com/hpk42/muacrypt/issues/32 Are your experiences with pgpy indicating it's compatible with enigmail and k9-mail? (see the questions on that issue)
PGPy is a quite complete OpenPGP(RFC4880) implementation, its support table shows that:
https://pgpy.readthedocs.io/en/latest/progress.html
The unsupported packets are very rarely used nowadays and are only really produced by very old PGP clients, afaik. PGPy also has a quite extensive test suite that works with gpg internally.
I suggest you look at PGPy issue tracker to see what it lacks currently, the most painful issue I think is the missing support for writing partial length packets. It can read them just well but not output them.
Good to know! I added the links to the muacrypt issue i linked above. dkg and vincent (from k-9 mail) tell me write support of partial length header is not needed for e-mails, so that's good because i'd like muacrypt, my in-progress autocrypt implementation, to work with pgpy sometime (https://github.com/hpk42/muacrypt/issues/32 )
- No special interface is needed on the mailing list web page maybe except from enabling/disabling the plugin/support.
Plugin configuration is done through the Mailman configuration and those are read-only through the REST interface. However a plugin might supply it's own REST endpoints for example for per-list setup/configuration.
I guess read-only REST would allow for a command line interface for debugging or other low-level configuration wrt to autocrypt key status for peers. Can a plugin add per-list configuration options (enable/disable, maybe a choice between 2-3 policies?)
Yes definitely, the configuration will be handled completely inside the pkugin. I'm thinking along the lines of:
Expose a custom REST endpoint for per-list Autocrypt configuration, that can be read-write, however is only protected by one global REST user-password pair. That will be accessed by an Autocrypt Django app, similarly to how I implemented configuration for mailman-pgp in [django-pgpmailman]. That app can be then run alongside Postorius and HyperKitty which provide Mailman's configuration and archives. So this gives list admins a simple web UI for per-list Autocrypt configuration.
Provide a CLI command component which can manage the per-list Autocrypt configuration of the Mailman instance locally.
sounds good. Is it also possible to hook into the standard mm3 configuration, for adding a per-list configuration item that can then be processed by plugin code?
holger
On Jan 31, 2018, at 09:28, Jan Jancar <johny@neuromancer.sk> wrote:
Plugin configuration is done through the Mailman configuration and those are read-only through the REST interface. However a plugin might supply it's own REST endpoints for example for per-list setup/configuration.
Yep; configuration options are read-only through REST because they are specified in the mailman.cfg file. You wouldn’t want to (and probably even can’t) allow REST clients to change things that aren’t kept in the database.
On Jan 31, 2018, at 15:03, holger <holger@merlinux.eu> wrote:
sounds good. Is it also possible to hook into the standard mm3 configuration, for adding a per-list configuration item that can then be processed by plugin code?
Per-list configurations are maintained in the database, configuration files (e.g. mailman.cfg) configure site-wide static settings. That’s because lists are dynamic; they can be created, deleted, and modified. But your plugin should be able to create its own tables and cross reference the mailing list tables (via the SQLAlchemy ORM).
Static configurations can be added to a [plugins.autocrypt] section, or better a separate autocrypt.cfg file, much like the way we separate out the postfix.cfg settings.
Cheers, -Barry
participants (4)
-
Barry Warsaw -
holger -
holger krekel
-
Jan Jancar