Hi Mailman Developers.
I am sending this mail as my proposal of encrypted mailing lists for GNU Mailman got accepted and I will be working on it this summer.
Sorry about not contacting you earlier, I had some issues where my site and mail server were down. If any of you tried to reach me and failed in the last ~week you can try next time on my backup mail jancar.jj@gmail.com which should always work.
You can find the original (accepted) version of my proposal on: https://neuromancer.sk/static/mailman.pdf
# Status report so far into the Community bonding period:
As it was proposed on this list a plugin-like implementation of encrypted mailing lists is really the only way to go forward here, as just pushing in what might end up being a rather niche feature into Mailman Core is not maintainable / wanted.
I started evaluating how much of my current proposal can be implemented without touching Mailman Core at all (as a plugin), what would require general changes and what might require specific changes (that means it needs a better solution).
So far it seems most functionality of encrypted mailing lists can be easily implemented out-of-tree with only minor general changes to Mailman Core with the following exceptions that I'm currently solving:
Making all commands require a confirmation (as subscribe / unsubscribe has). This is necessary to stop replay attacks.
Subscription command needs to contain users public key, either as an argument or attachment or any other way the plugin might get it.
List key fingerprint and per-address/user key fingerprints need to be stored somehow, directly in the mailing list model would make the most sense, but that's very specific so the plugin should store this itself. Although that means data duplication.
Plugins don't seem to have a way to add features to the core REST API, so exposing key administration for Postorious that way is out.
- Some questions that I had in my original proposal:
- Is exposing key management through the REST api and Postorius a good idea at all? Those have very different level of access control, changing a key on a list requires a signed request + signed confirmation token whereas doing it in Postorius might only require a password.
- A way of sharing the lists public key that makes the user trust it the most.
What I would like to definitely finish in the Community bonding period:
- Finish SMTPS/STARTTLS support for Mailman Core (really only needs tests now): https://gitlab.com/J08nY/mailman/tree/mta-smtps-starttls
- Establish real-time communication channels with mentors (text/voice?) and have a meeting to discuss the proposal.
- Add proper objective milestones to the proposal.
- Change the proposal to reflect movement towards a more plugin-like implementation.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi Jan,
Congratulations and welcome to Mailman! Excerpts from Jan Jancar's message of May 10, 2017 5:07 pm:
Hi Mailman Developers.
I am sending this mail as my proposal of encrypted mailing lists for GNU Mailman got accepted and I will be working on it this summer.
Awesome!
Sorry about not contacting you earlier, I had some issues where my site and mail server were down. If any of you tried to reach me and failed in the last ~week you can try next time on my backup mail jancar.jj@gmail.com which should always work.
You can find the original (accepted) version of my proposal on: https://neuromancer.sk/static/mailman.pdf
# Status report so far into the Community bonding period:
- As it was proposed on this list a plugin-like implementation of encrypted mailing lists is really the only way to go forward here, as just pushing in what might end up being a rather niche feature into Mailman Core is not maintainable / wanted.
I feel like core already has the architecture(interfaces everywhere! :) that will make it pretty easy to write plugins. If you feel you need some changes in core to support your plugin better or plugins in general, I should be able to help you with that part.
I started evaluating how much of my current proposal can be implemented without touching Mailman Core at all (as a plugin), what would require general changes and what might require specific changes (that means it needs a better solution).
So far it seems most functionality of encrypted mailing lists can be easily implemented out-of-tree with only minor general changes to Mailman Core with the following exceptions that I'm currently solving:
Making all commands require a confirmation (as subscribe / unsubscribe has). This is necessary to stop replay attacks.
Subscription command needs to contain users public key, either as an argument or attachment or any other way the plugin might get it.
List key fingerprint and per-address/user key fingerprints need to be stored somehow, directly in the mailing list model would make the most sense, but that's very specific so the plugin should store this itself. Although that means data duplication.
Plugins don't seem to have a way to add features to the core REST API, so exposing key administration for Postorious that way is out.
- Some questions that I had in my original proposal:
- Is exposing key management through the REST api and Postorius a good idea at all? Those have very different level of access control, changing a key on a list requires a signed request + signed confirmation token whereas doing it in Postorius might only require a password.
True, but there is a lot of trust already there on the password for postorius. What if someone un-subscribes from the Postorius and then re-subscribes sending along a key not owned by the user?
I don't know if that did make any sense, because as I understand the subscription would be moderated and it would be up to List Admin to not allow keys he doesn't recognize to be subscribed? Is there anything else except the admin stopping some attacker from doing that?
+ A way of sharing the lists public key that makes the user trust itthe most.
I feel the key signed by the List Owner would be the best way to indentify the lists public key. Maybe mandate signing by the Site Owner and List Owner/List Owners?
# What I would like to definitely finish in the Community bonding period:
- Finish SMTPS/STARTTLS support for Mailman Core (really only needs tests now): https://gitlab.com/J08nY/mailman/tree/mta-smtps-starttls
- Establish real-time communication channels with mentors (text/voice?) and have a meeting to discuss the proposal.
I am available as maxking on IRC(#mailman). I am a little busy for next week and then we have Pycon, but I should be able to meet you anytime after Friday 26th. I am not very sure, but I will have some time on 17th too, I will let you know?
- Add proper objective milestones to the proposal.
- Change the proposal to reflect movement towards a more plugin-like implementation.
I hope this summer is fun for you!
thanks, Abhilash
-- thanks, Abhilash Raj
Hey Abhilash!
On 05/14/2017 08:18 AM, Abhilash Raj wrote:
- As it was proposed on this list a plugin-like implementation of encrypted mailing lists is really the only way to go forward here, as just pushing in what might end up being a rather niche feature into Mailman Core is not maintainable / wanted.
I feel like core already has the architecture(interfaces everywhere! :) that will make it pretty easy to write plugins. If you feel you need some changes in core to support your plugin better or plugins in general, I should be able to help you with that part.
Yes, so far it has everything necessary. Some things I noted:
- A plugin cannot create a Pipeline the same way it creates Handlers or Rules, it can only do so in a post_hook. Since the Pipeline classes are enumerated when initializing them. https://gitlab.com/mailman/mailman/blob/master/src/mailman/core/pipelines.py...
- And then the issues I outlined in my previous email, which mainly stem from the encrypted lists plugin having some pretty strong requirements on current Mailman features.
+ Some questions that I had in my original proposal: + Is exposing key management through the REST api and Postorius a good idea at all? Those have very different level of access control,changing a key on a list requires a signed request + signed confirmation token whereas doing it in Postorius might only require a password.
True, but there is a lot of trust already there on the password for postorius. What if someone un-subscribes from the Postorius and then re-subscribes sending along a key not owned by the user?
I don't know if that did make any sense, because as I understand the subscription would be moderated and it would be up to List Admin to not allow keys he doesn't recognize to be subscribed? Is there anything else except the admin stopping some attacker from doing that?
Sure, subscription will be moderated and the List Admin will have to trust both the address he is accepting and the key provided. However, this is something we would like to stop, someone unsubscribing a user from an encrypted mailing list with just a password and not access to his private key.
This is something that definitely needs comments. What actions should a subscriber / unsubscribed user be able to do on an encrypted mailing list with:
- just a password
- his private key and what parts of this should be configurable and in what way / granularity?
I propose the following: per list. [require/allow encrypted, require/allow signed]
- all email commands should require a signed confirmation (the same workflow as subscribe / unsubscribe), to stop replay of commands.
- posts should require being encrypted and signed, and this should be configurable
- Postorius user operations should also somehow require signed confirmation.
+ A way of sharing the lists public key that makes the user trust itthe most.
I feel the key signed by the List Owner would be the best way to indentify the lists public key. Maybe mandate signing by the Site Owner and List Owner/List Owners?
Right, I also think that utilizing PGP web-of-trust the most, is the best we can do here. Also just recently found out, schleuder (an encrypted mailing list server) supports uploading signed list keys to it a will serve them with the signatures from then on.
# What I would like to definitely finish in the Community bonding period:
- Finish SMTPS/STARTTLS support for Mailman Core (really only needs tests now): https://gitlab.com/J08nY/mailman/tree/mta-smtps-starttls
- Establish real-time communication channels with mentors (text/voice?) and have a meeting to discuss the proposal.
I am available as maxking on IRC(#mailman). I am a little busy for next week and then we have Pycon, but I should be able to meet you anytime after Friday 26th. I am not very sure, but I will have some time on 17th too, I will let you know?
Sure, I'm in #mailman and I have some questions better suited for irc, so you'll definitely hear me there.
- Add proper objective milestones to the proposal.
- Change the proposal to reflect movement towards a more plugin-like implementation.
I hope this summer is fun for you!
I think it will!
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all.
Currently still looking into what changes will be necessary in core in order to cleanly implement encrypted mailing lists as a plugin, so this is my second status report. I will start doing these pretty much weekly now, and definitely weekly/ more frequent when the coding period starts.
All of my reports (currently only this one) can be found on: https://neuromancer.sk/ https://neuromancer.sk/recent.atom
(https://neuromancer.sk/article/2)
Also pasting it here for clarity and replies:
## Plugin API enhancements in Core
To cleanly implement encrypted mailing lists as a plugin to Mailman Core I propose several general changes to the plugin api, to allow for cleaner integration of plugins, more flexibility and easier plugin deployment. First I present the current state of pluggability in Mailman core and then present the proposed changes.
### Current state
Relevant mailman-developers thread from GSoC 2015: https://mail.python.org/pipermail/mailman-developers/2015-March/024477.html
Example plugin: http://threebean.org/blog/plugins-for-mailman3/
A plugin creator has many ways of "injecting" his code to run at certain phases of Mailman's operation, since Mailman looks for its classes and components dynamically, it doesn't care whether they are from a plugin or originally from core.
Implementing IHandler, IChain, IRule, IEmailCommand, ICLISubCommand or IStyle and placing modules containing the classes in the appropriate directories where Mailman finds them and instantiates them.
Implementing IRunner and adding the custom runner to mailman.cfg.
Implementing IArchiver and adding the custom archiver to mailman.cfg.
Implementing IMailTransportAgentLifecycle and setting the custom MTA in mailman.cfg.
Setting a custom callable in pre_hook or post_hook (only one callable per hook).
Core config variable ext_dir is unused.
There is not much documentation / examples of plugin creation, however core is very well documented so it's just a matter of figuring out what's pluggable or not.
Proposed changes
Add configuration option similar to config.styles.paths to Handlers, Rules, Chains etc.. Which Mailman will use to look for components, so that plugins can just be accessible on the python path and not necessarily inside the Mailman package. Or use just one path per plugin and a standardized plugin structure:
- plugin package
- handlers
- rules
- chains
- commands
- pipelines
- styles
- plugin package
Let plugins add Pipelines the same way they can add Handlers etc...
This means refactoring BasePipeline, OwnerPipeline, PostingPipeline, VirginPipeline from mailman.core.pipelines.py into a package mailman.pipelines
Use find_components
Let plugins subscribe to receive events.
Let List creator specify List Style when creating it through Postorius.
Allow multiple callables in pre_hook and post_hook run in order specified.
Drop ext_dir.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all.
This is my second update, sketching out the plugin details and issues that came up while working.
https://neuromancer.sk/article/3
# PGPMailman plugin
## Structure
pgpmailman - A Core plugin.
styles - Both styles generate a list keypair based on plugin settings on list creation as well as set other attributes for an encrypted mailing list. Such as the custom encrypted chain.
- EncryptedDefaultStyle
- EncryptedAnnounceStyle
pgp
rules
- EncryptionRule - Decrypts message and enforces per-list encryption requirements.
- SignatureRule - Checks message signature and enforces per-list signature requirements. Strips signature to msgdata.
chains
- EncryptedOwnerChain - Passes message through EncryptionRule before letting it continue default-owner-chain.
- EncryptedPostingChain - Passes message through EncryptionRule and SignatureRule before letting it continue default-posting-chain.
commands
- KeyEmailCommand - Handles user key management through the key command.
- KeyCLICommand
runners
- EncryptedOutgoingRunner - Encrypts and optionally signs for configured lists. This is a runner and not a Pipeline since we need to encrypt all outgoing messages, so digests, virgin messages, posts...
archivers
- EncryptedHyperKittyArchiver - Fetches list archive public keys from pgphyperkitty, uses them to send messages to archive encrypted, for encrypted lists.
pgphyperkitty- Django app, receives messages encrypted with list archive keys, decrypts them and passes them to HyperKitty. Also generates, holds and provides(public part) list archive keypairs.
Issues
Rules shouldn't change the message as they process it, since, well they are "rules" not "handlers". However other Mailman's rules need the message plaintext after decrypting for processing, so they cannot receive the message as it arrived encrypted. One solution would be to use a custom Incoming runner to sort-of unwrap the PGP/MIME message from encryption and signature (storing it in msgdata) and then pass it to the default Incoming runner to run chains. This would also make sense from a PGP/MIME standpoint as the "real" message is really the encrypted content so for that's what other Mailman rules should get in the msg object.
Storing of user and list related metadata is unsolved. This means storing the user key fingerprint for each user / address, as well as storing per-list configuration. The list key fingerprint might not need to be stored as the key can be looked up by gnupg.user_id == list_fqdn. The plugin can have it's own db although this means data duplication. A solution I see would be to allow plugins to store arbitrary metadata in the core db attached to certain models. So that the mailinglist model would have another attribute plugin_data of PickleType which would really be a dict with the plugin name as keys, with additional api allowing a plugin to access it's metadata stored in this object.
Extending the Postorious and Hyperkitty interface is unsolved. Since plugins cannot add routes to the REST api. A solution I see that would be general and also require little changes to core would be to let plugins add custom routes to /plugins/<plugin_name>/... which could be then accessed either from a Postorious plugin or a custom app. Maybe with falcon.API.add_sink()?
Even when extending the REST api at core side is possible as just described, that still leaves integrating into HyperKitty and Postorius somehow, out-of-tree, as a django app. Looking at how HyperKitty and Postorius plug into each other currently, we see direct references at each other in the templates:
https://gitlab.com/mailman/postorius/blob/master/src/postorius/templates/pos...
{% if 'hyperkitty' in INSTALLED_APPS %}
{# insert hyperkitty link #}
{% endif %}- While that might be okay for integrating HyperKitty and Postorius together, and it's even used at very few places and seems to not be the best solution, it's not practically usable for integrating this plugin. Ideal integration will encrypt messages when sending them to HyperKitty while showing them decrypted to users. I am thinking of having a custom encrypted HyperKitty archiver that will handle encrypting with proper archive key at the core side and a custom django app at the HyperKitty side that will receive the message, decrypt it and pass it to HyperKitty decrypted. That however leaves the messages decrypted in HyperKitty's DB so doesn't really fulfill project requirements. In addition a custom DB engine can be used in HyperKitty's settings that wraps whatever DB engine and encrypts/decrypts objects on the fly, although I am not sure of the doability of this.
## Installation
- Just installing the plugin into the same virtualenv as the Mailman instance, and setting up the proper options in mailman.cfg should work.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi Jan!
Congratulations on being selected!
Note that Mailman is a PSF suborg, so you have to comply with PSF reporting standards. I'll get back to you if you need to do anything more. The only thing I can think of offhand is to register with the PSF "planet" (or whatever they're using to aggregate blog posts). I *think* that's not even ready yet (or maybe they set it up today).
My apologies for disappearing on you after the application period, it's been very busy for me (most of my teaching responsibilities are in spring term, April-June, and startup is alway demanding).
A few comments on the content.
Jan Jancar writes:
### Current state
Relevant mailman-developers thread from GSoC 2015: https://mail.python.org/pipermail/mailman-developers/2015-March/024477.html
Example plugin:
Thanks for the summary. I was wondering what people meant by "plugin", now I know! (I wouldn't call this a "plug-in architecture" to be honest; I'll be interested to study your improvements.)
### Proposed changes
- Add configuration option similar to config.styles.paths to Handlers, Rules, Chains etc.. Which Mailman will use to look for components, so that plugins can just be accessible on the python path and not necessarily inside the Mailman package.
Not enough, IMO.
Or use just one path per plugin and a standardized plugin structure:
Yes, more structure, please.
- Let plugins add Pipelines the same way they can add Handlers etc...
OK.
- Let plugins subscribe to receive events.
OK.
- Let List creator specify List Style when creating it through Postorius.
AFAIK this is already an RFE.
- Allow multiple callables in pre_hook and post_hook run in order specified.
Seems reasonable, you'll need to talk to Barry about that. (Of course you can always implement this with a hook-managing hook function. So it's a matter of abackward compatibility.
Steve
On 05/22/2017 07:53 PM, Stephen J. Turnbull wrote:
Hi Jan!
Congratulations on being selected!
Thanks! Very happy to be selected.
Note that Mailman is a PSF suborg, so you have to comply with PSF reporting standards. I'll get back to you if you need to do anything more. The only thing I can think of offhand is to register with the PSF "planet" (or whatever they're using to aggregate blog posts). I *think* that's not even ready yet (or maybe they set it up today).
I only found: https://wiki.python.org/moin/SummerOfCode/Expectations although having a look at planetpython it seems my site can be integrated into it, so no problem there. Should I send an integration request?
My apologies for disappearing on you after the application period, it's been very busy for me (most of my teaching responsibilities are in spring term, April-June, and startup is alway demanding).
Know that very well myself, as I struggle to finish all school / research related tasks before the coding period starts.
- Let List creator specify List Style when creating it through Postorius.
AFAIK this is already an RFE.
Now that I looked at list styles more closely the way they apply all of their parent classes in their apply method makes using them for plugins kind of heavy handed. Let's say that my encrypted lists plugin wants to work through a list style, which I proposed in my second update, as it's a really nice mechanism for this. Now there are two built-in styles "legacy-default" and "legacy-announce", because of the way list styles compose I will have to create two list styles myself, for "legacy-default" and "legacy-announce". If more styles are added, each of them will have to have a corresponding "encrypted" variant, and this applies to any plugin, or even to core styles and their combination.
It might make more sense to have a list of styles associated with a mailing list, which are linearly applied on list creation. The compound styles("legacy-default" and "legacy-announce") would be possible the same way they are now, so something like setting "legacy-default" and "encrypted-mixin" styles on a list would make it an encrypted mailing list.
With the current way, the amount of different list-styles would get rather large very quick. Might not be a problem since the amount of list styles I can imagine that would make sense combining is rather low. But just by adding another style, something like "AnonymousListStyle" would require creating 4 styles: anonymous-default anonymous-encrypted-default anonymous-announce anonymous-encrypted-announce and so on. The next style if combinable with all the other styles would need 8 styles.
- Allow multiple callables in pre_hook and post_hook run in order specified.
Seems reasonable, you'll need to talk to Barry about that. (Of course you can always implement this with a hook-managing hook function. So it's a matter of abackward compatibility.
As you pointed out out-of-list, there is a MR #264 already for this feature, along with a signal passing between core and plugins, so sending a link for the list:
https://gitlab.com/mailman/mailman/merge_requests/264
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all.
I have created a repository to keep the original proposal for this project, as well as the current working version and similar documentation / specs at:
https://gitlab.com/J08nY/pgpmailman-proposal
The repo contains up-to-date changes I propose to Mailman core to accommodate an encrypted lists plugin as well as other possible future plugins. (in core_changes.md) It also contains a current working sketch of the plugin (in plugin.md). Comments are very appreciated.
I wrote a post on the current issues I am facing on integrating the encrypted lists plugin with Postorius and HyperKitty:
https://neuromancer.sk/article/4
Integrating with Postorius and HyperKitty
Since a plugin-like out-of-tree approach is required for implementing encrypted lists into Mailman, a straight forward integration into Postorius and HyperKitty (as first proposed) by making them "aware" of the encrypted lists plugin is not possible.
Thus a new approach for providing their functionality and conforming to the project requirements is necessary. I see three possible pathways forward and a middle-ground between them.
## Standalone django app
A new django app will be created, using django-mailman3 as a base, that will implement all the web based functionality for encrypted mailing lists, such as:
- Displaying the List key for all public encrypted mailing lists.
- List key management for list admins
- User key management
- Encrypted archives, that are server unencrypted (effectively replaces HyperKitty for encrypted lists)
This app will then be run besides Postorius.
## A fork/patchset approach
This approach will create a fork of Postorius and HyperKitty that will integrate changes necessary for the encrypted lists plugin seamlessly. Thus users wanting to use encrypted mailing lists will have to setup Postorius and HyperKitty from this fork.
## Wrenching it in
This approach tries to integrate all of the functionality using configurable options of Postorius and HyperKitty. For example storing messages encrypted could be done via a custom django.db.backend. Receiving messages encrypted could be done via a small custom django app that will receive them, decrypt and pass to HyperKitty decrypted.
## A middle-ground
Somewhat of a middle ground seems to be most sensible. A standalone app will be necessary to provide functionality that is simply not possible to be integrated into Postorius and HyperKitty sensibly. This app will mostly provide key management (user and list), receive the messages encrypted and so on. However Postorius and HyperKitty will work with the least amount of "wrenching it in" as possible.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all,
ad the first ~week of the GSoC period passes I am reporting the current state of my work, comments appreciated! See below or at:
https://neuromancer.sk/article/5
============================= GSoC 2017 - About a week in
This week has been quite productive. I started laying the groundwork for the encrypted lists plugin by making changes from core_changes to different components of both Mailman Core, Mailman-Client and Postorius. I will describe the MRs and ideas behind them here.
Mailman Core
Move pipelines to their own package and instantiate dynamically
This MR makes Pipelines in Mailman Core work in the same dynamic way as Handlers, Rules, Chains and other similar Mailman's components work.
Add plugin infrastructure
This MR is the biggest one and certainly the most important one for the pgpmailman plugin and other potential Mailman plugins. It adds a notion of a plugin to Mailman Core. To explain what I mean by a plugin, it's best to look at its example configuration:
[plugin.example]
path: example_plugin
class: example_plugin.plugin.ExamplePlugin
enable: yes
configuration: python:example_plugin.config.exampleThis configuration defines a plugin called example (section name). Which resides in the example_plugin package with the example_plugin.plugin.ExamplePlugin class implementing the IPlugin interface, which I will describe in a moment. This plugin is enabled and can use the path example_plugin.config.example.cfg to load it's configuration.
Alright now what does this actually do / change / provide in Mailman Core?
The plugin's path package is searched for pluggable components if the plugin is enabled and has a non-empty path value. A new function similar to find_components is introduced and used, find_pluggable_components. The components are searched for in subpackages named after what is being searched, common to Mailman Core:
- example_plugin.rules -> IRule
- example_plugin.chains -> IChain
- example_plugin.commands -> IEmailCommand
- example_plugin.handlers -> IHandler
- example_plugin.styles -> IStyle
- example_plugin.pipelines -> IPipeline (after [mailman!287] lands)
The plugin's class is instantiated if the plugin is enabled and has a non-empty class value (and implements the IPlugin interface). The IPlugin interface has several methods:
pre_hook() -> Is called after instantiation but before Mailman's db initialization, really just moved the existing pre_hook configurable callable into a plugin infrastructure. Return value unused.
post_hook() -> Is called after pre_hook and after initialization of Mailman's db and other dynamic components such as commands, rules, handlers... Return value unused.
rest_object() -> Is called whenever a REST client calls the /plugins/<plugin.name>/... route, to obtain an object which will respond to REST queries for the plugin rooted at aforementioned route. This method can return None to signify the plugin doesn't want to expose any REST routes, in which case any calls by clients return 404. Otherwise the object should use the routing system used by Mailman's REST api (@child(), on_get and so on).
Mailman also sets the name attribute on the plugin object after instantiation, before pre_hook to the plugin's name in configuration.
The plugin can use the path in it's configuration option to load it's configuration if it needs a config file. The same path resoultion rules apply as for the archivers configuration.
Add description attribute to styles
Mailing list styles currently have an attribute for name, which for the default styles is:
legacy-announce
legacy-defaultWhich is not particularly human-readable, well it is, but doesn't look like something to be shown in a web ui such as Postorius. Mapping these style names to their descriptions in Postorius is possible, however since the styles are dynamically found and instantiated, it's not something practically doable.
So this MR adds a description attribute to IStyle and adds the appropriate descriptions to the default styles. Also exposes said description to REST clients.
Mailman-Client
Add bindings for specifying list style
This MR adds missing access to the lists/styles REST api endpoint and another argument style_name to the Mailman-Client's create_list call, which is already recognized in Core and will create a new list with selected style. This will then be used by Postorius.
Add plugin bindings
This MR depends on mailman!288 as it really just exposes the new REST api added with plugins to Mailman-Client. Since this new api gives plugins control over all the requests to paths after their root, the api in Mailman-Client can not be as high-level and abstracted as it is for lists and other core features. It provides access to plugin's configuration and a Plugin.call(path, data, method) similar to Connection.call.
Postorius
Add list style selection to list creation view
This MR depends on mailmanclient!28 and mailman!289. It adds another field to the list creation form, prompting the user to select a list style, with the default being pre-selected and list style descriptions being shown.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi Jan,
I haven't had a chance to look at the MRs yet, as it's my first week at a new job. But I have to say that your descriptions below look really great; I'm impressed at how well your design fits in with Mailman overall, and how clearly you've described them. I'm not sure I could have come up with anything better!
I have just a few questions, though we can hash out specifics on the MRs.
On Jun 08, 2017, at 12:45 AM, Jan Jancar wrote:
Add plugin infrastructure
This MR is the biggest one and certainly the most important one for the pgpmailman plugin and other potential Mailman plugins. It adds a notion of a plugin to Mailman Core. To explain what I mean by a plugin, it's best to look at its example configuration:
[plugin.example] path: example_plugin class: example_plugin.plugin.ExamplePlugin
I'm curious as to why you define both a path and a class, where the latter
has the former as the first component. I'm not making a judgment (and haven't
looked at the branch yet), so I'm just wondering what led you to this decision.
Mailing list styles currently have an attribute for name, which for the default styles is:
legacy-announce legacy-defaultWhich is not particularly human-readable, well it is, but doesn't look like something to be shown in a web ui such as Postorius. Mapping these style names to their descriptions in Postorius is possible, however since the styles are dynamically found and instantiated, it's not something practically doable.
So this MR adds a description attribute to IStyle and adds the appropriate descriptions to the default styles. Also exposes said description to REST clients.
Have you thought about how these descriptions will be internationalized? We'll clearly want to present style descriptions in the natural language of the admins who are creating new lists with a particular style.
Cheers, -Barry
Hi Barry!
On 06/10/2017 05:19 AM, Barry Warsaw wrote:
Hi Jan,
I haven't had a chance to look at the MRs yet, as it's my first week at a new job. But I have to say that your descriptions below look really great; I'm impressed at how well your design fits in with Mailman overall, and how clearly you've described them. I'm not sure I could have come up with anything better!
That's great to hear! Thank you and best of luck at your new job.
I have just a few questions, though we can hash out specifics on the MRs.
On Jun 08, 2017, at 12:45 AM, Jan Jancar wrote:
Add plugin infrastructure
This MR is the biggest one and certainly the most important one for the pgpmailman plugin and other potential Mailman plugins. It adds a notion of a plugin to Mailman Core. To explain what I mean by a plugin, it's best to look at its example configuration:
[plugin.example] path: example_plugin class: example_plugin.plugin.ExamplePlugin
I'm curious as to why you define both a
pathand aclass, where the latter has the former as the first component. I'm not making a judgment (and haven't looked at the branch yet), so I'm just wondering what led you to this decision.
I wanted the possibility of configuration, where for example the plugin
package might have more IPlugin classes with different behavior and so
the site admin might choose the plugin class he wants to use (a kind of
plugin super-package). Or really just to not force the plugin creator
to put the plugin class in some specific path with relation to the
general plugin path which is used for loading components as the plugin
class is a specific kind of component (only one per plugin).
It might be fine to just drop the class part, load it in a similar way
as other components (so it will be in the .plugin subpackage of the
plugin package) and just enforce that one plugin entry in config can
provide only one IPlugin implementation.
Mailing list styles currently have an attribute for name, which for the default styles is:
legacy-announce legacy-defaultWhich is not particularly human-readable, well it is, but doesn't look like something to be shown in a web ui such as Postorius. Mapping these style names to their descriptions in Postorius is possible, however since the styles are dynamically found and instantiated, it's not something practically doable.
So this MR adds a description attribute to IStyle and adds the appropriate descriptions to the default styles. Also exposes said description to REST clients.
Have you thought about how these descriptions will be internationalized? We'll clearly want to present style descriptions in the natural language of the admins who are creating new lists with a particular style.
Hmm haven't thought about it yet, as I thought it was quite similar to attributes of other Mailman's components, however this is indeed different as it's not shown only to site admins but list admins / through Postorius. Will see what's possible there.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all.
Sending an update here along with a new blog post. https://neuromancer.sk/article/6
============================== GSoC 2017 - About the plugin
As last time I described outstanding PRs to Mailman Core and other Mailman's components, this time I am going to describe the plugin I am building on top of those components to enable PGP encrypted mailing lists.
In fact there are two plugins to present, with the first one being a general and quick plugin example that showcases how the new plugin API might be used and also provides some nice services for the PGP plugin.
mailman-rest-events
This plugin subscribes itself to receive events in Mailman Core and
sends them to urls configured. It is a variation of what
mailman!264 is
intending to do, based on the plugin API implemented in my MRs. It
currently offers only basic functionality, a URL is specified in config
per endpoint along with an API key (very similar to
mailman-hyperkitty) and a regex of event class names that will be sent.
The data sent currently is only the event class name in a JSON object.
Config can have many endpoints specified with the form shown below.
[endpoints.django_mm3] url: http://localhost/django/api api_key: Something_secret_here events: .*
Results in a call such as this one:
POST /django/api?key=Something_secret_here HTTP/1.1 ... Content-Type: application/json
{"event": "DomainCreatedEvent"}
The plugin is currently very bare bones and proper serialization of
events is underway. I am thinking of recursively serializing the
non-private event attributes until an easily serializable object is
reached, such as an Address or a MailingList or anything that has a
unique string associated with it and can be queried using the REST API
and such a string. So for example:
{ "name": "DomainCreatedEvent", "event": { "domain": "example.com" } }
or
{ "name": "AddressVerificationEvent", "event": { "address": "anne@example.org" } }
This plugin will be used by the mailman-pgp plugin to send events to django-mailman3 which will distribute them to Postorius, HyperKitty and the django app created to enable PGP encrypted mailing lists.
mailman-pgp
First of all, I think I came up with a name that better fits with mailman-hyperkitty being an archiver "plugin" and in general with how plugins for projects are named in python (flask_ext_... and so on).
As the general functionality and layout of the plugin was already described I will mostly just summarise issues I am currently working on tackling.
low level PGP work
I originally proposed using python-pgp (https://github.com/mitchellrj/python-pgp https://pypi.python.org/pypi/py-pgp/0.0.1) to use to parse and create OpenPGP packets, as other libraries mostly offer only a high-level API or are not so feature rich. However this library seems quite un-maintained, so either a working fork should be established or a different library used. I am currently looking at both options. Will contact the original author as well as the current pypi package owner and see what's possible.
public key on subscription
Since a PGP enabled mailing list needs to know the public key of a subscribed address at the point when moderator approval is required, some mechanism to do so is necessary. This needs to happen before moderator approval, as the moderator / list owner should be able to see / download the provided key to verify it (or at least its fingerprint). Currently I see two options, an extension of the proposed key command or some changes to the current Subscription and Unsubscription workflows.
Extending the proposed key command would let the user use said command to send and confirm his public key, before a subscription to an encrypted list. This is similar to how schleuder does it. They essentially run somewhat of a mailing key-server where the users might submit keys via custom headers in message body.
However the current SubscriptionWorkflow and UnsubscriptionWorkflow
are really close to being able to require a custom additional step
during which the subscribers key is received and confirmed. Maybe making
[Un]SubscriptionWorkflow pluggable components, that can be set on a
per-list basis, similar to owner/posting chains or pipelines.
The second solution is definitely a more heavy-handed one, although might be a better one in the long run. As it allows plugins to provide [un]subscription steps which might be a nice use for plugins in Mailman Core. I can imagine for example subscription verification where the plugin verifies that the subscriber is a customer of some company by showing the user a confirmation token in the customer system etc..
require confirmation for all commands
To address replay attacks where an attacker listens on a user performing
a command by sending it to the list_id-confirm address and then sends
the same message again in the future to replay the command, all commands
will need to require confirmation. This stops the attacker since he
cannot obtain the cleartext of the message containing the confirmation
token, or even if he could, he cannot forge the signature on the
confirmation message (leaving endpoint security out-of-scope). To reuse
the confirm command, Workflows and confirmation would have to be
refactored out from subscription management.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all.
This update goes with my latest blog post about finding a good PGP python library:
https://neuromancer.sk/article/7 (!the blog posts content is not equal to this post!)
Along with that I also implemented list keypair generation in the mailman_pgp plugin. It is configurable (key type, key size) in the plugin config and currently produces keys such as this one:
pub rsa4096/0x539FD576AA401697 2017-06-19 [SC] 5B810BA0660C2EBCF46B3354539FD576AA401697 uid [ unknown] Zzz <zzz-request@example.com> uid [ unknown] Zzz <zzz@example.com> sub rsa4096/0x0AC655A0580B8CE6 2017-06-19 [E]
This keypair generation is done via spawning another process on encrypted list creation. Since the process might take quite some time, it would create problems if, for example, the encrypted list was created by the REST runner which would then go off to create the keypair for a few seconds while requests keep timing out.
I also implemented PGP/MIME and inline PGP utilities. They can check whether a message is signed/encrypted as per RFC1847+RFC3156 for the former, and per some custom strict rules for the latter(inline PGP). They can verify/decrypt both PGP/MIME and inline PGP messages. These use the PGPy library I found and blogged about.
After reading through some more of the discussion on mailman-developers around GSoC 2013 OpenPGP integration, I've chosen to refine some of the handling of both signatures and encryption in the PGP plugin, to address some of the questions/concerns. Here I specify a high level overview of signature handling, i.e. what should happen, not how should it be implemented.
Signature handling
Per-list configuration
Option Default
- unsigned_msg_action bounce
- inline_pgp_action None (pass-through)
- expired_sig_action bounce
- revoked_sig_action bounce
- duplicate_sig_action bounce
- malformed_sig_action bounce
- strip_original_sig False
- sign_outgoing False
Posting to
# Assumes mailman_pgp already has a confirmed users public key.
User signs his message, either via PGP/MIME or inline PGP.
Sends to list posting address.
mailman_pgp
checks if the message is for a PGP enabled mailing list. If not, passed it directly to the default incoming runner. Else go-on.
checks if the message is signed or not. If unsigned, the
unsigned_msg_actionis taken. Else go-on.checks if the message is inline PGP signed. If so, the
inline_pgp_actionis taken. Else go-on.checks if the signature verifies with the users public key. If not, the
malformed_sig_actionis taken. Else go-on.checks if the users public key that signed the message has been revoked or expired. If so the
revoked_sig_actionandexpired_sig_actionis taken, respectively. Else go-on.looks up the signature hash and timestamp in the collection of all of them mailman_pgp ever saw. If it finds it, then the
duplicate_sig_actionis taken. Else go-on. ^^ I am not sure about this step ^^. It was proposed in the 2013 discussion, to avoid replay attacks on posting (replay attacks on commands are solved by requiring confirmation).
If it reached this step, with no actions being taken then the message is considered a good posting as far as mailman_pgp is concerned.
If it is inline signed, the signature ASCII-armor is moved from the message body and stored in
msgdataand the cleartext of the message remains in the message body.If it is MIME signed, the signature bodypart is "unwrapped" from the multipart/signed (as per RFC1847) with the signature ASCII-armor stored in
msgdataand only the inner part of the first payload as the message body.
Continue message processing as any other list.
Outgoing posts
If the list is configured to
strip_original_sig, go-on. If not attach the signature frommsgdataback to create a PGP/MIME signed message. (don't output inline PGP...)If the list is configured to
sign_outgoingadd another signature packet from of the mailing lists private key signing the message body, to the signature ASCII-armor.Send out.
Questions / Discussion points
Should mailman_pgp keep all the signature hashes and timestamps it received to stop replay attacks on posting? A replay attack on posting without it is trivial.
Should the users keys be per user or per address? From a PGP perspective per user makes more sense, as PGP keys have more user-ids, so a user just sets up his key once, and it covers addresses in all his user-ids. However per-address is more flexible, and the keypair must be confirmed per-address anyways. A subscription is also per-address(a user through a preferred address).
Aren't the options specified for signature handling too much? revoked and expired sig_action could be merged, maybe even malformed.
Should we "unwrap" the signed message, as I propose, before passing it to the rest of Mailman core?
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all!
Another blog post is up. Copy follows for discussion. https://neuromancer.sk/article/8
PGP handling progress
This week things moved along with the mailman-pgp plugin. As I finally had a good PGP library to work with as well as some basic plugin API patches outstanding to Mailman Core.
So with that I implemented large parts of the mailman_pgp.pgp package,
including tests, as well as a basic signature checking rule. Also, as
the package now contains tests, I added a gitlab CI config and started
running it.
repo @ neuromancer.sk repo @ gitlab
Wrappers
Created two classes, which wrap an email.message.Message instance (or
mailman.email.message.Message) and expose a common public PGP handling
API on said instance. One understands PGP/MIME (as per RFC1847 and
RFC3156) and the other somewhat understands inline PGP. They both give
methods to check presence of a PGP signed/encrypted message or PGP keys,
to verify signatures of a message, to decrypt it or collect the attached
keys. There are currently almost complete tests for handling signatures
and partial for encryption.
MIMEWrapper
This one was rather easy to implement, as RFC3156 and RFC1847 are both very precise but also rather simple standards (well compared that to 90+ page RFC4880 [OpenPGP]).
InlineWrapper
As there is really no "Inline PGP standard" this wrapper improvises with how it looks for signatures/messages/keys. In it's current form it only accepts a non-multipart message and looks for ASCII-armored blobs in it.
Signature rule
The concrete way of integrating the PGP plugin into Mailman Core has changed over time. The original version of my proposal planned on working in the Mailman Core tree and as that changed to a plugin, new requirements of integration popped up, us stuff couldn't just be changed for this one plugin to work.
Because of the nature of encryption the PGP plugin needs to get it's
hands on the messages before the rest of Mailmans rules. This could be
done with a rule in a custom chain, however as I understand rules, they
are, well, rules, they shouldn't change the message they are processing
in any way. In my understanding of this architecture, if Mailman was
written in a language with a const identifier, the message would have
it in the chain links process method. It would work as a rule, just
wouldn't feel nice and fit in with the architecture.
That's why I first wanted to implement all handling of incoming messages
in a custom incoming runner. This runners class is then set as the
runner for the in queue and the default runner is moved to another
queue, perhaps in_default. The custom incoming runner does it's thing
before passing messages back to the default incoming runners queue.
However only encryption has this issue of needing to change the message,
as rules and the rest need it in cleartext. So I decided to split off
signature handling to a custom rule, and only handle encryption in the
incoming runner.
I would like to get rid of the not-so-nice redirection of the default incoming runner altogether but I see no other way to decrypt the incoming message to a PGP enabled list before other rules get it.
The signature rule processes messages using the wrappers described above according to a per-list signature handling configuration:
Option | Default -------------------- | ------- unsigned_msg_action | bounce inline_pgp_action | None (pass-through) expired_sig_action | bounce revoked_sig_action | bounce malformed_sig_action | bounce duplicate_sig_action | bounce strip_original_sig | False sign_outgoing | False
I think the options are quite self-explanatory. They are checked in
order and taken if the message matches the option and the set action is
not None. So with the default configuration of the PGP enabled mailing
list, an unsigned message will be bounced, inline PGP is allowed,
signatures by expired keys are bounced (expired at time of checking),
signatures by revoked keys are bounced, signatures that don't verify are
bounced (maybe a better name for them than malformed?) and finally
signatures that the Mailman PGP plugin already processed are bounced as
well. This is to prevent replay attacks and is something I think people
would expect Mailman to do, although I am not sure if the plugin should
do it.
This signature rule is almost complete, with some edge-cases remaining and the checking of duplicate signatures also remaining.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all.
https://neuromancer.sk/article/9
The first evaluation period is here, I have already sent in my evaluation so I thought I would recap the state of the project.
During the first two weeks or so, I mostly focused on features in Mailman Core that would be necessary for making a plugin like this work, my MRs introduce an easy to use/configure plugin infrastructure. Where plugins can provide components and even custom REST routes in Mailman Core. This with just installing the plugin into the same environment as Mailman Core and some config additions. Also some minor additions of selecting the list style on creation in Postorius and appropriate changes in mailman-client.
During the last two weeks I focused on work on the mailman-pgp plugin itself, along with its helper plugin mailman-rest-events. The REST events plugin is finished, apart from some tests and bug-fixing and it does its job of sending Mailman Core events to RESTful application endpoints fine.
The mailman-pgp plugin currently has working list-key generation, decryption in incoming runner, signature checking rule and general inline and PGP/MIME message handling utilities. I also got its tests to work with Mailman Core test layers so it can be actually integration tested as it would work with Mailman Core. Along with that I setup Gitlab CI pipelines with coverage testing and currently tests pass!
As I worked in PGP handling in messages I needed a good python PGP library, which I found, but since some features mailman-pgp requires were not implemented/buggy I made quite a lot of PRs to this library as well.
Key management is the next thing I would like to work on after finishing tests of the current implementation. After that outgoing message encryption and signing. When these things work I would then move on to implement the appropriate things in the django-pgpmailman web app which hooks into Postorius and HyperKitty.
Overall I am quite confident in where the project is now. Seeing it run tests with Mailman Core running and pass (just some basic tests atm) feels good.
Cheers!
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
On Jun 28, 2017, at 01:49 PM, Jan Jancar wrote:
Overall I am quite confident in where the project is now. Seeing it run tests with Mailman Core running and pass (just some basic tests atm) feels good.
Thanks for the update, and really great work Jan! I'm slowly making my way through the changes to improve plugins to the core, but I really like the direction you've chosen.
Cheers, -Barry
Hi all!
https://neuromancer.sk/article/11
Another update
Just a quick update about the state of the project before I disappear from civilization for this weekend.
Pluggable workflows
https://gitlab.com/mailman/mailman/merge_requests/299
I have now finally got the pluggable workflows branch to test succesfully! With workflows being loaded dynamically, with workflow steps being saved completely, full backward compatibility, also database migrations and REST backward compatibility.
That branch is still missing more tests for diffcov to pass.
https://gitlab.com/J08nY/mailman/pipelines/9645615
^^python 3.4 CI envs are currently broken, since aiosmtpd 1.1 is
python 3.5 only, so don't mind that.
Key management
https://gitlab.com/J08nY/mailman-pgp/tree/feature/workflows
Building up on the pluggable workflows I have a base implementation of how the subscribers key gets to the mailing list and its moderator and also of how it's confirmed.
SMTPS + STARTTLS
https://gitlab.com/mailman/mailman/merge_requests/286
During this week I also got to add SMTP over SSL/TLS support to
aiosmtpd, a lib which Mailman uses for it's SMTP tests, so with that
the SMTPS+STARTTLS MR has tests which pass and I think is ready for
merging. Thanks Barry for the quick responses and help debugging over at
aiosmtpd!
Next up
For next week I plan on cleaning up the pluggable workflows branch, finishing the pre-subscription key management, implementing after-subscription key management as well as getting back on track with documenting the mailman-pgp architecture and changes at the pgpmailman-proposal repo, which has been lacking.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all!
Checking in with progress around the halfway point of the GSoC 2017 timeline.
https://neuromancer.sk/article/12
====================== The point of no return
This week is around the halfway of GSoC 2017 project timeline, hence the title.
Key management
As I set out to do, I have now implemented most of the after
subscription key management. The key change command. The pre
subscription key management is done via a custom dynamically loaded
subscription policy/workflow. This is possible with my
pluggable-workflows branch/MR which introduces dynamically loaded
workflows, which subscription and unsubscription policies are a part of.
Subscription
On subscription the PGP enabled mailing list needs a users public key. An example conversation establishing that follows (shows a list with moderation + pgp enabled subscription policy):
Mailman User
------- ----
list@example.com anne@example.com
<- subscribe 1.
2. key set "token1" ->
+ instructions to reply and attach user key
(signed by list)
<- key set "token1" 3.
(key attached as per RFC3156(PGP/MIME)
or inline and we do our best to find it.)
4. key confirm "token2" ->
+ statement containing "token2" and
user key fingerprint in body, with instructions
to sign it
(signed by list, encrypted to user key)
<- key confirm "token2" 5.
+ must contain signed statement from step 4
(required to be signed by user key)
6. ... the rest of the subscription steps (moderation) ...Steps 3 and 5 can be optionally encrypted to the list key.
This workflow gives "some" assurance that the user really has the key
she is subscribing with, however the trust in the address/user/key has
to come from the list moderator and both my proposal and implementation
rely on that. This workflow only proves that Mailman is communicating
with someone that has the user key (can decrypt "token2" and sign the
statement), not that that user is anne@example.com.
Key change
Works similar to the key set command on subscription. Works via a
custom workflow, but not a subscription workflow/policy, just a workflow.
Mailman User
------- ----
list@example.com anne@example.com
<- key change 1.
(new! key attached as per RFC3156(PGP/MIME)
or inline and we do our best to find it.)
2. key confirm "token1" ->
+ statement containing "token1" and
new! user key fingerprint in body, with instructions
to sign it
(signed by list, encrypted to new! user key)
<- key confirm "token1" 3.
+ must contain signed statement from step 2.
(required to be signed by old! user key)Steps 1 and 3 can again be optionally encrypted.
This ensures that only someone with possesion of the old user key can change it. It requires the user to decrypt with the new key and sign a statement with the new key fingerprint as well as the token in it with the old key.
PyPi package
mailman-pgp @ PyPi I have now created a PyPi package for the mailman-pgp plugin as it's maturing, and while still unusable I wanted the name.
Test coverage
I got test coverage of mailman-pgp to 95% which now actually means
something as large parts of the codebase are implemented and not just
stubs. Since this plugin requires features not merged into Mailman Core
(or PGPy) for pipelines to work I maintain forks of both with branches
that merge all required branches into a plugin branch for Mailman Core
and dev branch for PGPy.
Next up
Outgoing
I had a look at working in outgoing encryption of messages for pgp
enabled mailing lists. Encryption cannot be done as a handler, as it
requires to process the messages per-recipient or per-batch of
recipients. The only configurable/dynamic option currently in Mailman
Core is the [mta].outgoing callable. Which is not ideal since it
itself has more functionality than simply dispatching to a correct
Delivery instance based on the list configuration, that would have to
be duplicated in mailman-pgp. The IMailTransportAgentDelivery
interface looks promising though, so I'm thinking about refactoring the
outgoing callable a bit to allow setting a custom one without the
duplicities.
Signature hash tracking
To prevent replay attacks for things where always sending and requiring a signed confirmation token in response is not practically doable, the pgp plugin will keep signature hashes with timestamps and posssibly key fingerprints in a DB/some other quickly searchable structure, and will check signatures it receives against the hashes in that structure. It will then act accordingly, for example not allowing a user to post the same message (with the same signature) again, since it could have been replayed.
Documentation
I wanted to get the pgpmailman-proposal repository updated with current state/MRs but didn't get to it this week, so might find time for it over the weekend as well as for some code documentation.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all!
Another week, another project update.
https://neuromancer.sk/article/13
==================== Post title goes here
Signature hash tracking
It would be relatively easy to replay a signed message to a mailing list by a user as no kind of challenge-response is done on posting.
While signature replay checking is usually done on the end users point with against his keyring and messages he has so far received and their context, I think it is kind of expected of PGP enabled Mailman to also do this as it relays the messages.
On a successful posting to a PGP enabled mailing list (AcceptedEvent
to be precise), the message is searched for PGP signatures and their
digests and key fingerprints are added into the sighash table.
The signature rule then checks the digests from a posting against those
in the sighash table. If it finds it has previously accepted a message
with any of those hashes it takes the duplicate_sig_action which is
per-list configurable.
This of course means that if the duplicate_sig_action is not set to
Action.defer, which means the message gets rejected or dropped, that a
signature that was sent to a list cannot be sent again. Of course if a
user wants to send a signed message again, he can just resign it and
send it again, the hashes won't match. However sometimes it might be
useful to post the message as signed originally, for example to prove
something. However, I think it is worth it to keep this as a
configurable option. Maybe with another option disabling the collection
of signature hashes, for performance reasons.
Outgoing processing
Since a the plugin needs to process the messages for outgoing encryption
on a per-recipient or per-chunk basis, it couldn't be implemented as a
Handler, I thought about implementing it as a custom OutgoingRunner but
that didn't work out either. However the IMailTransportAgentDelivery
interface is great for what the plugin needs to do. So there are two
custom PGP enabled delivery classes that get selected by a custom [mta] outgoing callable, one for bulk delivery and other for individual delivery.
Documentation
I got the pgpmailman-proposal repository almost completely up to date
on changes to the plugin and my current proposed
Core/Client/Postorius/HyperKitty changes and MRs.
Next up
More testing
As I remembered to ignore coverage of tests it dropped to 88% and I think it is currently in order to add more comprehensive tests for features implemented that check the edge cases which currently remain.
Setup live development instance
I believe I got pretty far into the development without having a proper live development instance of Mailman Core + plugins and Postorius + HyperKitty + Client, so I'm going to set that up now and test manually. With time I might set this up as a public site/mailing list server, to demonstrate the features of the PGP plugin.
Archivers
Currently thinking about implementing two encrypted archivers, one local, one remote. The local one would be similar to the prototype archiver but store messages encrypted, TBD how. The remote one will send the messages encrypted to a receiving archiver, the django-pgpmailman instance running next to HyperKitty.
django-pgpmailman
With most of the essential stuff in the core plugin done, the web ui part of development can begin.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all!
https://neuromancer.sk/article/14
================= Second evaluation
The second evaluation period came quite fast after the first one, nonetheless the project advanced much further so quick recap of its current state is in order.
Since first evaluation
SMTPS + STARTTLS
Finally with working tests after upstream fixes in aiosmtpd.
Pluggable components (plugins)
Rebased after the Click CLI processing branch was merged and got it to work nicely in the end. There was an issue with config processing, as plugins can now supply their own CLI commands and plugins are configured/specified in the Mailman core config, which can be supplied by a CLI option. So that created a bit of an argument processing/lazy-loading issue but looking some more at how Click works I was able to quickly resolve it. Config option is now an eager one and initializes Mailman before commands need to be listed/used.
Pluggable workflows
Also rebased and added tests to get diffcov to 100% and fix one of the migrations in that branch, as it was broken before.
Key management (plugin)
Implemented email commands for managing per-address PGP keys in plugin. It uses pluggable workflows to plug into the subscription process of a PGP enabled mailing list and requests the user key, also does mailback confirmation of it. The key is then available to the list moderator during subscription moderation, and more generally to the plugin for verifying signatures and encrypting. After subscription key management is also implemented where a user can change his set key, provided he can sign a challenge provided by the plugin with the old key. Key revocation handling is also necessary, but not yet done.
Outgoing processing (plugin)
Implemented custom Bulk and Individual delivery classes for PGP enabled lists in plugin. These deliveries optionally encrypt the mail to subscribers keys, and/or sign with the list key. The bulk one retains anonymity of subscribers as the keyids are zeroed out of the PKESK(Public Key Encrypted Symmetric Key) packets which OpenPGP implementations should handle as a wildcard keyid and try decrypting with all usable private keys. It is also almost as efficient as it can be, as it only encrypts the message with one session key per chunk and then encrypts said session key to recipients in the chunk. The signing is also configurable.
Signature hash tracking (plugin)
Implemented store of signature hashes from successful postings to a PGP enabled list which are then (optionally) used to stop replay of the same signature by Mailman in a future posting.
PyPI package (plugin)
Created the PyPI package for the PGP plugin.
Overall
I would like to be a bit further in the project at this point, however I am very optimistic about the next days and weeks. After resolving some issues and TODOs I have for the current plugin implementation and setting up the live Mailman instance with the PGP plugin, which should be up by the end of the week. I believe I can start work on the other side of the REST API, of somehow hooking PGP enabled archives/lists to Postorius and HyperKitty.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all!
Another week, another update.
https://neuromancer.sk/article/15
==================== GSoC 2017 - Progress
This week was tough but productive. Temperatures spiking to 34°C in my hometown have a really bad effect on my daily productivity.
Setup instance with PGP plugin
Finally got a complete mailman instance
setup and running with J08nY/mailman/plugin + J08nY/mailman-pgp/master
and J08nY/Postorius/plugin + J08nY/mailmanclient/plugin +
mailman/HyperKitty/master + mailman/django-mailman3/master. The plugin
branches merge MR branches that introduce the plugin infrastructure for
that particular Mailman component. For Mailman Core, the plugin branch
merges the pluggable-components, pluggable-workflows and
list-style-descriptions branches.
The pluggable-components one introduces the concept of a plugin to
Mailman Core and replaces the (pre|post)_hooks and is essential to let
site admins easily add plugins to Mailman Core by simply installing them
to the same environment as Mailman Core and some simple configuration to
enable. pluggable-workflows splits the subscription/unsubscription
monolithic workflows into composable workflows, that are also pluggable
by a plugin and set per-list. list-style-descriptions are exposed via
the REST api and Postorius uses them for displaying list style selection.
I even successfully created a PGP enabled discussion list through
Postorius. Subscribed to it by sending the subscription request,
confirming it, replying to the key set <token> challenge with key
attached, replying to the key confirm <token> with the challenge body
signed by the key being set. This would of course be followed by the
moderator verifying the supplied key in any real application of PGP
enabled lists, which is also supported.
The instance runs on a Raspberry Pi with 512MB RAM along with my web-server, mail-server and several other services, so don't expect lightning fast performance, or it being up anyway, reserving the right for any extended downtime ;).
Key revocation
Working on proper key revocation behavior from the PGP plugin took much of my week as getting this right is pretty hard and the OpenPGP revocation mechanism is quite complex. The usual workflow for just an ordinary key change was already presented in one of my previous posts. However if the user needs to revoke a key with a revocation signature, we cannot use the old key to perform the key change challenge. Also, the key revocation can be only partial, as in a subkey being revoked, and the key can still be used for encryption and signing, then it's usable for the PGP plugin and nothing needs to be done. This also gets more complex as when we allow a user to change his key without moderator approval, only with the challenge (which makes the user sign a challenge/statement signifying they are changing their key to the new one, by the old key). Then if the user revokes his former key using a reason for revocation that invalidates all signatures by that key(even former ones), we cannot trust the users current key, as the old one could have been compromised and used to set the new one.
For now, giving a mechanism for users to provide a revocation certificate that is verified merged with the key is implemented. If the revocation certificate revoked the key or a subkey/uid that makes the key not usable by the PGP plugin (the key can no longer be encrypted to or can sign) then the users key is reset and he/she has to send and confirm a new one with moderator approval necessary. That is almost completely implemented as it's almost the same as the subscription challenge.
More PGPy work
Necessary to make it usable, as for example, not having support for partial length headers would break handling of most messages encrypted with GPG as it likes to create plenty of packets with partial lengths. However, now I think that my development branch of PGPy is feature complete enough to support an instance of Mailman with the PGP plugin running.
Trello board
Setup a Trello board to better track the issues that I came up and keep my head sane (private board): https://neuromancer.sk/static/mailman_pgp_trello.png
Next up
Web UI integration
The original proposal proposed adding support for PGP enabled lists to Postorius and HyperKitty directly, now when mailman-pgp is dynamically enabled in Mailman Core a similar approach needs to be taken with the Postorius and HyperKitty integration.
Archiving
Thinking of doing local archiving very similar to the prototype archiver, encrypted by the list local-archive key. The remote archiving capability is a much tougher nut to crack and depends a lot on how the HyperKitty integration ends up looking.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all!
https://neuromancer.sk/article/16
================== Web UI integration
This post is about my current plans on how to implement the web ui part of PGP enabled Mailman. It strives to integrate into the Mailman Suite and use its features to the maximum possible degree.
General idea: Refactor general stuff to django-mailman3, to allow apps to hook up together in Mailman Suite easily, and then use that to hook up django-pgpmailman.
Features
Show PGP enabled public lists, with their key fingerprints, with the option to download their public keys, also show some of their configuration (so that subscribers can see that for example if they send a cleartext message to a list that requires encrypted messages, it will be bounced).
Enable list owner to configure the PGP related per-list configuration options.
Enable list owner to set/see the list key (private part). This is quite questionable and will have a site-level option to be turned off (the REST API will then not serve the list private key).
The same level of user key management as the key command allows, with
similar steps during key change/revocation.
Implementation
Another django app is installed in the same project as Postorius + HyperKitty, django-pgpmailman. This app provides a list of PGP enabled mailing lists and their PGP related management in a similar way Postorius does, also user key management.
There are few places where Postorius refers to HyperKitty and vice versa, for adding the appropriate links/icons to the navbar as well as for the user menu entries. These references will be refactored to some mechanism in django-mailman3, which will allow any installed django app to add it's entry to the navbar or the user menu. This will allow django-pgpmailman to hook up rather easily, without any direct references to it from Postorius/HyperKitty/django-mailman3.
Archiving
The archiving web UI is a tougher nut to crack. I either have to develop a custom PGP mail archive frontend and integrate it similar to the PGP list management app, or integrate with HyperKitty transparently, so that archives are received encrypted, stored encrypted, and yet served to subscribers in clear. Developing a custom app is quite unrealistic and it would lack most HyperKitty functions.
However hooking up an encrypted message store to HyperKitty is also non-trivial, as HyperKitty is strongly tied with storing messages in it's database and using a django Model to represent a message.
I currently have no realistic ideas, one that comes to mind, is to create a custom django database backend, that somehow stores everything encrypted, but thats a very unwieldy solution that likely won't work well.
Other progress
Fixed many little issues with the PGP plugin and PGPy. Got it to work quite nicely, below you can see a message received by a subscriber, by a PGP enabled discussion list, encrypted to his key, as shown by Thunderbird with the EnigMail plugin:
https://neuromancer.sk/static/mailman_pgp_message.png
Also finally merged the finished key revoke command to mailman-pgp/master.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all!
Another progress report, on the PGPMailman web ui status.
=============== Web UI progress
django-pgpmailman progress
Successfully created the mail list views. Inspired heavily by Postorius, to get the same look, both in templates and views. There is a list index view, which lists only PGP enabled lists, and their key fingerprints. This also allows one to download the list key as it's linked from the list key fingerprint. The list name link leads to a list settings/info view. The info tab is available to any logged in user, while the settings are list owner only. All the per-list PGP settings are configurable there.
List index screenshot: https://neuromancer.sk/static/mailman_pgp_web_index.png
List encryption settings screenshot: https://neuromancer.sk/static/mailman_pgp_web_settings.png
django-mailman3 template chunks
In order to make plugging the django-mailman3 based apps together and deduplicate some of their code, as well as to integrate the django-pgpmailman app into any Postorius + HyperKitty project I refactored the direct references of Postorius to HyperKitty and vice versa.
This is done in the template chunk MR. It introduces a new template tag in django-mailman3, which is intended to be used by all django-mailman3 based apps to let other installed apps add their entries to the navbar and user menu. Which I are two main ways Postorius and HyperKitty reference each other.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
Hi all.
https://neuromancer.sk/article/18
===================== Final work submission
As the GSoC 2017 final evaluation period just ended, my final work product is finally submitted. This post is a summary of my final work product.
Mailman-pgp
- https://gitlab.com/J08nY/mailman-pgp
- https://mailman-pgp.readthedocs.io/en/latest/
- Plugin for Mailman Core.
- Enables creating a PGP mailing list, which has a list key, can receive and serve messages encrypted, can sign and receive signed messages from subscribers.
- Creates the
keyemail command, which is used for per-address user key management. - Subscription to a PGP enabled mailing list the subscribing address to send and confirm an address public key, which the moderator must verify.
- Somewhat confirms the user has possession of the appropriate private key to the one sent on subscription.
- Has per-list settings for encryption/signatures/what to do with non encrypted / non signed messages, etc..
- Optionally exposes a REST API for list configuration.
- Has local archivers which can store the messages encrypted by the list key.
- Stores list and address keys in configurable key directories.
- requires (some not merged) MRs in Mailman Core
- additional MR (not required)
- Required branches are merged and maintained at [J08nY/mailman/plugin.
- To install, do
pip install mailman-pgp, warning: it will pull in a development version of Mailman Core and PGPy.
django-pgpmailman
- https://gitlab.com/J08nY/django-pgpmailman
- A Django app, uses django-mailman3 and mailmanclient, integrates well with Postorius and HyperKitty.
- Provides management of PGP enabled mailing lists to the list owner,
and of PGP related subscription settings to the subscriber.
- requires (currently not merged) MRs in mailmanclient, django-mailman3, Postorius and HyperKitty
mailman-rest-events
- https://gitlab.com/J08nY/mailman-rest-events
- A plugin for Mailman Core that turned out to be unnecessary for the working of django-pgpmailman, but implemented a similar feature as this MR.
- This plugin sends the events (and some information about them) from Mailman Core to a list of configurable endpoints using JSON in HTTP POST requests.
Other contributions
- mailmanclient/split-sources, merged
- postorius/list-style-selection
- Many many PRs to PGPy, a Python only implementation of OpenPGP. 19 PRs and counting. As PGPy was not and still is not feature complete in regards to RFC4880 I found out many times that it's missing features/bugs broke mailman-pgp CI. It would not make sense fixing them locally, both from a software design perspective and open source software one aswell.
Overall
I think I met almost all goals that the project idea required and my original proposal stated, with the noteworthy exception of remote archiving to HyperKitty which I just couldn't find a way to integrate.
Cheers,
Jan
/\ # PGP: 362056ADA8F2F4E421565EF87F4A448FE68F329D /__\ # https://neuromancer.sk /\ /\ # Eastern Seaboard Phishing Authority /__\/__\ #
participants (4)
-
Abhilash Raj -
Barry Warsaw -
Jan Jancar -
Stephen J. Turnbull