Re: [Mailman-Developers] Browser ID integration with Mailman

On Feb 16, 2012, at 10:23 PM, Aamir Khan wrote:
Hi Aamir,
I don't think much has been discussed about it, but maybe Florian has thought more about how to integrate it with the mm3 ui.
-Barry

Hi,
I don't think much has been discussed about it, but maybe Florian has thought more about how to integrate it with the mm3 ui.
I haven't had the opportunity to implement browserid on a website so far, but here are a few random thoughts on the subject:
Browserid is very convenient, but I wouldn't rely on it as the only means to log into the MM web ui. It relies on client-side code to be run, which has a lot of advantages usability-wise but also has some accessibility pitfalls. I don't think that's a reason *not* to implement browserid, but probably more as one of more auth-methods that site admins can choose when setting up their installation of mailman.
The server-side verification part of their example application [1] is done using node.js. I think creating a similar thing in python and making a django AuthBackend out of it could be a very nice task. A quick search brought up two django-browserid implementations ([2], [3]), but in both cases I'm not sure if this is how we would do it in our web ui. (Both are stand-alone reusable apps - I think we'd rather integrate something like that directly as part of our app code. But that's probably debatable...).
If this is too small a task for the whole summer I think it could be made part of a slightly larger login/user-related package. The wiki page for last year's GHC11 Open source day [4] describes some ideas that could fit in nicely (for example storing some additional user info, like their twitter account, ... see section "Subscriber info summary").
As I said these are random thoughts...
Cheers Florian
[1] https://github.com/lloyd/myfavoritebeer.org/ [2] https://github.com/mozilla/django-browserid [3] https://github.com/mozilla/django-browserid [4] http://wiki.list.org/display/DEV/Notes+from+the+GHC11+Open+Source+Day

On Friday, February 17, 2012 20:41 CET, "Florian Fuchs" <f@state-of-mind.de> wrote:
Sorry, I posted the second link twice. That should be:
https://github.com/omab/django-social-auth
Florian

On Feb 17, 2012, at 1:41 PM, Florian Fuchs wrote:
I agree. It is a reasonable scheme to support, but we need to have "authentication" done on a "plug-in" basis. Structurally, the underlying design should be completely agnostic to the login mechanism.
Since Django already has a User model with all kinds of permissions infrastructure, we should be leveraging off of that (extending the User model as needed) for our UI.
MM has two kinds of authentication to perform. BrowserID "solves" both at the same time. First, it establishes that the person on the website is a particular user. But it also "proves" that the email address "belongs" to that user. Thus, with BrowserID, we can allow the user to join a mailing list without having to send the confirming email transaction.
On the other hand, that confirming email transaction is basically what the browserid.org site does. Thus, with little effort, MM could also act as a secondary authority for the BrowserID protocol.
I'll "debate" the approach. IMHO, we should be leveraging the Django user model and we should not be reimplementing it. If the existing BrowserID implementations fail to provide the necessary functionality, then we should either "fix" one of them or write another implementation which is still generic and could be reused by others.
Similarly, authentication schemes such as Kerberos should be able to plug in without caring that they are being used for MM or some other application.
In keeping with current events:
As I said these are random thoughts...
Is there enough entropy? :)
Richard

Hi Richard,
On Friday, February 17, 2012 21:59 CET, Richard Wackerbarth <richard@NFSNet.org> wrote:
I agree. It is a reasonable scheme to support, but we need to have "authentication" done on a "plug-in" basis. Structurally, the underlying design should be completely agnostic to the login mechanism.
Since Django already has a User model with all kinds of permissions infrastructure, we should be leveraging off of that (extending the User model as needed) for our UI.
Agreed. Django's User model most definitely covers all our needs. The thing I am not sure about is: What kind of user info ends up in the core DB and what should be stored in the web ui db. In theory we don't need any kind of permanent storage in the web ui, because the core already has a pretty comprehensive user model (storing user names, email addresses, roles, memberships etc). OTOH it could make great sense to enhance this data with all kinds of crazy things that could be stored in the web ui. Or to store some user data temporarily to reduce API calls. But this means we'd have two different data sources since the web ui should not access the core db directly.
MM has two kinds of authentication to perform. BrowserID "solves" both at the same time. First, it establishes that the person on the website is a particular user. But it also "proves" that the email address "belongs" to that user. Thus, with BrowserID, we can allow the user to join a mailing list without having to send the confirming email transaction.
Yep, very convenient. The process would be initiated via the client-JS and verified server-side in the django-backend. Without ever touching the REST API.
On the other hand, that confirming email transaction is basically what the browserid.org site does. Thus, with little effort, MM could also act as a secondary authority for the BrowserID protocol.
I think Barry has had some similar ideas...
I agree, no re-implementing the User model! In fact, I think we mean the same thing, because django's custom auth backends are simply a way to handle auth externally *and* tie that to the existing user/permissions system. Of course, if we implement such a custom backend we should do it in a generic and reusable way. I just think making a full-blown django app for what is usually just a .py file might be a bit overkill... But as I said: debatable... ;-) I guess in the end it's not a question of whether or not we make use of the Django User model. (At least to me) the question is: Do we use it to store user data permamently in the web ui or just temporarily on a request/session basis. Or a mix of both.
Well, not completely random, I guess ;-)
Florian

On Feb 21, 2012, at 02:00 AM, Florian Fuchs wrote:
Yep, it's a big question. Something could clearly make sense in the core, e.g. gpg keys for authentication or encryption. Other, specifically ui-type data might not make as much sense in the core. Even if we came up with a way to extend the core schema and store the data in the core, how would you access it? There would probably have to be a generic REST API for getting key/value data associated with the user in and out of the core.
I've have a similar question when it comes to authorization. Just where to do we keep the authorization mapping users and roles to actions they can perform? I'm very close to punting on this for 3.0 beta (and thus likely for 3.0 final), though I'm also happy to revisit it during the Pycon sprint (likely for a hypothetical 3.1).
Cheers, -Barry

On Feb 20, 2012, at 7:28 PM, Barry Warsaw wrote:
In general, I think that this will be needed. Consider what happens when we try to add a "plugin" optional function such as a different archiver.
In general, it will need to have some database stored parameters. Do we then add yet another database? I think not. A generic key/value mechanism in the existing database would provide the capability without requiring a custom change to the API.
For the short term, I would let the UI become the repository for such information. For the longer term, see my remarks in reply to Florian's message which you are referencing.
Richard

On Feb 21, 2012, at 08:10 AM, Richard Wackerbarth wrote:
Another way to do this, now that we have (what I think is a fairly robust) schema migration, you could imagine the plugin would make available a few migrations which would add the necessary data to the existing database. There might be problems if you're adding fields to existing model objects, but adding purely new tables shouldn't be too much of a problem I'd think.
Yep, thanks. -Barry

On Feb 25, 2012, at 3:37 PM, Barry Warsaw wrote:
First, I would be quite concerned if any plugins alter existing database tables. I see nothing but nightmares trying to avoid conflicting interactions, naming conventions, etc. From a maintenance point-of-view, adding auxiliary tables rather than additional columns in existing tables is much more manageable.
When it comes to migrations, we need to assure that migration sequences from separate plugins remain commutative. We will have no control over the order in which a particular installation will adopt particular plugins.
Where possible, I think that it is preferable to avoid add database tables if the scope of reference for those tables extends beyond the plugin itself. In order to bridge between the UI and the MM core servers, the REST interface would also have to be extended.
On the other hand, implementing the parameter store for a plugin by adding rows to a common parameter database would be able to utilize an existing interface.
Parameters are just key/value pairs that apply to a particular context. Generically, that context is a tuple of [user, list, ...] . If we add "plugin" to that context key, we have can have a generic store that does not need to be changed when a plugin is added.

Hi,
On Tuesday, February 21, 2012 02:28 CET, Barry Warsaw <barry@list.org> wrote:
Hmm, that doesn't sound like you particularly like that scenario. ;-) I'm not sure if it's a really good idea to use the core as a general data store. If user data (or even list data) is augmented with information the core doesn't need to operate, why not use a second data source. After all we have a system at hand (django) that's designed to do that.
I kind of liked the idea of an optional "bare-bones" version web ui that doesn't need anything more than the core API to do its tasks. But of course there are many use cases beyond that.
I've have a similar question when it comes to authorization. Just where to do we keep the authorization mapping users and roles to actions they can perform?
I'm not sure if that's what you mean, but I would say that depends on who the main authority is for, say, assigning a role to a certain user for instance. What if I revoke the moderator role from a member via the command line (I don't think that's currently possible, but let's say it is...). The core would have no way of actively inform the web ui (or possibly all the many different web ui installations, apps etc...) that this privilege has been revoked. So I'd say this is some information the web ui would have to request from the core.
Aahhh, the Pycon sprint... :-)
Cheers Florian

On Feb 23, 2012, at 11:04 PM, Florian Fuchs wrote:
Not so much. ;)
Exactly.
Yep. TBH, this is something I've been struggling with for a long while now. Remember I've advocated for keeping all those decisions out of the core, but there has been valid push-back on that (admittedly somewhat lazy ;) stance.
Now, I think we can get notifications pushed from the core to the ui if we wanted, assuming the ui had some service to notify. See my previous message on events. But in some cases, like the one you mention above, I think it would also be fine for the ui to query the core to gather the information necessary to decide whether an action is allowed. E.g. the ui could ask the core, via REST, "is this person an owner of that mailing list?".
There are upsides and downsides to this. For example, do we want to keep the logic for authorization in the web ui? That would mean if you integrate the core with your own custom web ui, then it wouldn't gain any of the benefit of the authorization logic in the Django app. OTOH, that might be just what you want, and I can imagine a site like Launchpad would prefer that, since it has its own user model. Ignoring that whole morass does keep the core lean and mean, which I rather like, especially because I want to release this beast RSN. ;)
So I'm very tempted to punt on the whole authentication/authorization issue as much as possible in the core for the 3.0 release, but I'm also happy to keep the discussion open for possible changes for the future. I think the integration work going on between the web ui and the core will give us some much needed practical experience here. Right now, it's all too theoretical for me to make much headway on.
Fun, fun!
Note too that I will be giving a talk on MM3 at Pycon, but fortunately I only have to fill 30m. :) I want to be able to release MM3 very soon after the sprints.
-Barry

Hi,
Awesome! So the wui would implement a simple REST API for the core to talk to. Ideally it should be possible to "register" more than one web ui (the official ui might not be the only application using the Mailman API), so multiple parties can be notified of changes. Also, we should probably think about how fine-grained those "invalidate"-notifications can be. It's probably not necessary to invalidate the whole cache every time something changes in the core.
I wonder what the main use cases are for custom web uis. My guess is that many ui-creators might not build the whole thing anew, but create smaller applications for single tasks like (un)subscribing to lists etc. In larger integration scenarios, auth might be taken care of at some other stage (a company intranet for instance or a content management system).
Note too that I will be giving a talk on MM3 at Pycon, but fortunately I only have to fill 30m. :)
I know - I'll definitely be there! (Always wanted to know what this MM3 thing is and what it can do! ;-)
Florian

On Feb 20, 2012, at 7:00 PM, Florian Fuchs wrote:
Agreed. Django's User model <snip>
OTOH it could make great sense to enhance this data with all kinds of crazy things that could be stored in the web ui. Or to store some user data temporarily to reduce API calls. But this means we'd have two different data sources since the web ui should not access the core db directly.
Let me suggest that we "step back" a bit from the implementation and look at the infrastructure architecture.
The Web UI is a Django site. Therefore, (IMHO), we should build it as one. That means that we have some django apps that represent the MM functionality. They have models and views just like any other django site. The models for the users, lists, etc. just "happen" to represent the same data structure as that used by the MM core.
The database access becomes a stack. The UI always accesses its data through its model. The implementation of the access could be directly as a table on a local database (as django is normally used) or it could use the REST interface to utilize the MM core as its data store. I would also introduce a transparent caching layer which would be optional. I suspect that (almost all of) the part of the MM database that is accessed by the UI could be classified as "read mostly" and that we could cache the information on a per-session basis, thus avoiding many of the accesses thru the REST interface.
Of course, if we implement such a custom backend we should do it in a generic and reusable way. I just think making a full-blown django app for what is usually just a .py file might be a bit overkill... But as I said: debatable... ;-)
I'm not sure what you mean by "full-blown django app". Even though an app must have models, views, templates, etc., those can usually be empty placeholder files.
I guess in the end it's not a question of whether or not we make use of the Django User model. (At least to me) the question is: Do we use it to store user data permamently in the web ui or just temporarily on a request/session basis. Or a mix of both.
We will definitely need some persistent storage that is beyond the functionality provided by the current MM core. Whether that storage is provided locally by a UI-owned database or the core database is extended to provide the backing store should be configurable at a later time.
Another approach would be to migrate all of the storage to a separate module and let both the UI and the "core" interface to it. The REST interface could also be layered on top of it to provide an alternate API that could be bypassed by the core and/or the UI. This would make provide a clean place to insert "foreign" databases as might be maintained by existing enterprise infrastructures.
Richard
Richard

Hi,
On Tuesday, February 21, 2012 14:59 CET, Richard Wackerbarth <richard@NFSNet.org> wrote:
The database access becomes a stack. The UI always accesses its data through its model. The implementation of the access could be directly as a table on a local database (as django is normally used) or it could use the REST interface to utilize the MM core as its data store.
Yep, it obviously makes sense to separate data from business logic. And ideally the view code doesn't look (much) different whether it accesses a model stored in a db or a rest resource. (BTW, I recently started rewriting this part of the prototype in that sense.) But I'm not sure I would see database models as exactly equivalent to REST resources.
I would also introduce a transparent caching layer which would be optional. I suspect that (almost all of) the part of the MM database that is accessed by the UI could be classified as "read mostly" and that we could cache the information on a per-session basis, thus avoiding many of the accesses thru the REST interface.
Yep. But we need to decide how tolerant we are when it comes to inconsistencies between the state of the core data and a web ui session. A list of mailing lists might be stored on a per session basis - if a new list is created by a third party during a session, it's probably tolerable not to have that new list appear immediately. It's a bit trickier when it comes to privileges like moderation etc. that should probably take effect immediately (especially when revoked).
Of course, if we implement such a custom backend we should do it in a generic and reusable way. I just think making a full-blown django app for what is usually just a .py file might be a bit overkill... But as I said: debatable... ;-)
I'm not sure what you mean by "full-blown django app". Even though an app must have models, views, templates, etc., those can usually be empty placeholder files.
By that I meant: Something people installing it have to take care of by tweaking their settings.INSTALLED_APPS tuple. If it's part of the package, that doesn't need to be done. Although having an extra app for that could make sense if the different auth methods should be strictly pluggable.
Cheers Florian

On Feb 23, 2012, at 4:46 PM, Florian Fuchs wrote:
If I am not mistaken, the INSTALLED_APPS as defined in the django instance settings, is just a convenient way to switch on/off django apps whose code is present. I think that an enabled app can effectively override the tupe and cause other apps to behave as if they had been listed in the settings.
However, I am not sure that to do so would be a good idea. The person who installs a plugin is going to have to do a number of things. In particular, he will, presumedly, have to do more than just add the code to the python path. He will modify some configuration parameter on the core side in order to have the plugin have any effect on the mail traffic.
Since we have separated the mail handling from the administrative interface (UI), he will also need to both make code available and alter some configuration setting on the UI side.
Just as in the django distribution, where the "admin" interface is optional, adding references to our plugins SHOULD involve adding lines to the settings files, etc. Note that we could provide a script that would automate those changes. However, as a system administrator, I prefer to be told in the installation instructions what lines I need to change, and why. I am leery of scripts that "magically" change configurations on my system.
Richard

On Feb 23, 2012, at 11:46 PM, Florian Fuchs wrote:
I wonder if a push model could be used in those cases where changes to the core database need to be immediately reflected in the ui. For example, the core utilizes zope.events as a hook mechanism to notify other parts of the core when certain things happen. For the most part, I'm just using this in the test suite, but I think it would make sense to hook events to notify external systems (e.g. the web ui, or its cache, etc.) when certain things happen.
Cheers, -Barry

Florian Fuchs writes:
Agreed. Django's User model most definitely covers all our needs.
What version of Django's User model are you using? The vanilla User is actually rather limited. Or do you mean as augmented by UserProfiles? If the latter, do you have a specific schema for the Profile in mind, or are you just referring to its flexibility?
We do, I think. I think the core DB should be constant across Mailman installations, and restricted to fields useful in "forum" administration (ie, not restricted to mailing lists, although that will be the main purpose of Mailman 3 at its initial release). The web UI DB storage would be optional and flexible. But all fields and tables available to the web UI should be accessed by the same APIs in developing an instance of the web UI.
Also, some fields of the core DB may be provided by a third party in vegetative state (eg, a personnel department). We may want to allow the web UI to augment, edit, or override some of those fields in presentation.
But this means we'd have two different data sources since the web ui should not access the core db directly.
Yes, the core DB and the web UI's persistent storage would be different. But clients of the web UI's API need not know that, or do they?

On Wednesday, February 22, 2012 07:49 CET, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
By "User model" I didn't mean the actual vanilla User model (sorry for not being precise). I meant its flexibility and that it can be easily enhanced. Also how it's automatically added to the request object and accessible from the templates etc.
Good point.
If they become aware of that, something probably went wrong... ;-)
Florian

On Feb 22, 2012, at 03:49 PM, Stephen J. Turnbull wrote:
The counter argument is that the core's database should only contain the data that is needed to do its function. A leaner core schema means it will be easier to migrate and maintain as time goes on. It also reduces bloat in the REST API, even if we provided a generic key/data API (which frankly I don't like).
E.g. if personnel department were central to the core's functionality, by all means, let's add it to the schema and expose it in the right places within the REST resource model. If not, then it really shouldn't be part of the core's schema.
Cheers, -Barry

On Feb 23, 2012, at 5:43 PM, Barry Warsaw wrote:
The counter argument is that the core's database should only contain the data that is needed to do its function.
Yet another argument would be: The MM suite can be partitioned logically into a message handler, an archiver, an administration-by-mail interface, a web-based interface, and a roster (et. al.) store.
Perhaps the core should store only that data which it, and no other part of the suite utilizes. Upon receipt of a message, if appropriate, it would forward the message to appropriate subsystems. If it is to be distributed to a roster, it would obtain the current roster from the storage subsystem.
The web UI, administration-by-mail subsystem and the corporate personnel department would interact with the data store in order to make changes to the rosters.
Perhaps this concept is in keeping with the "leaner core schema" approach. It also has the advantage that the data store is isolated from the MM message handler. That would make it easier for "large corporation" to implement the store within their existing corporate database.
Richard

Hi,
I don't think much has been discussed about it, but maybe Florian has thought more about how to integrate it with the mm3 ui.
I haven't had the opportunity to implement browserid on a website so far, but here are a few random thoughts on the subject:
Browserid is very convenient, but I wouldn't rely on it as the only means to log into the MM web ui. It relies on client-side code to be run, which has a lot of advantages usability-wise but also has some accessibility pitfalls. I don't think that's a reason *not* to implement browserid, but probably more as one of more auth-methods that site admins can choose when setting up their installation of mailman.
The server-side verification part of their example application [1] is done using node.js. I think creating a similar thing in python and making a django AuthBackend out of it could be a very nice task. A quick search brought up two django-browserid implementations ([2], [3]), but in both cases I'm not sure if this is how we would do it in our web ui. (Both are stand-alone reusable apps - I think we'd rather integrate something like that directly as part of our app code. But that's probably debatable...).
If this is too small a task for the whole summer I think it could be made part of a slightly larger login/user-related package. The wiki page for last year's GHC11 Open source day [4] describes some ideas that could fit in nicely (for example storing some additional user info, like their twitter account, ... see section "Subscriber info summary").
As I said these are random thoughts...
Cheers Florian
[1] https://github.com/lloyd/myfavoritebeer.org/ [2] https://github.com/mozilla/django-browserid [3] https://github.com/mozilla/django-browserid [4] http://wiki.list.org/display/DEV/Notes+from+the+GHC11+Open+Source+Day

On Friday, February 17, 2012 20:41 CET, "Florian Fuchs" <f@state-of-mind.de> wrote:
Sorry, I posted the second link twice. That should be:
https://github.com/omab/django-social-auth
Florian

On Feb 17, 2012, at 1:41 PM, Florian Fuchs wrote:
I agree. It is a reasonable scheme to support, but we need to have "authentication" done on a "plug-in" basis. Structurally, the underlying design should be completely agnostic to the login mechanism.
Since Django already has a User model with all kinds of permissions infrastructure, we should be leveraging off of that (extending the User model as needed) for our UI.
MM has two kinds of authentication to perform. BrowserID "solves" both at the same time. First, it establishes that the person on the website is a particular user. But it also "proves" that the email address "belongs" to that user. Thus, with BrowserID, we can allow the user to join a mailing list without having to send the confirming email transaction.
On the other hand, that confirming email transaction is basically what the browserid.org site does. Thus, with little effort, MM could also act as a secondary authority for the BrowserID protocol.
I'll "debate" the approach. IMHO, we should be leveraging the Django user model and we should not be reimplementing it. If the existing BrowserID implementations fail to provide the necessary functionality, then we should either "fix" one of them or write another implementation which is still generic and could be reused by others.
Similarly, authentication schemes such as Kerberos should be able to plug in without caring that they are being used for MM or some other application.
In keeping with current events:
As I said these are random thoughts...
Is there enough entropy? :)
Richard

Hi Richard,
On Friday, February 17, 2012 21:59 CET, Richard Wackerbarth <richard@NFSNet.org> wrote:
I agree. It is a reasonable scheme to support, but we need to have "authentication" done on a "plug-in" basis. Structurally, the underlying design should be completely agnostic to the login mechanism.
Since Django already has a User model with all kinds of permissions infrastructure, we should be leveraging off of that (extending the User model as needed) for our UI.
Agreed. Django's User model most definitely covers all our needs. The thing I am not sure about is: What kind of user info ends up in the core DB and what should be stored in the web ui db. In theory we don't need any kind of permanent storage in the web ui, because the core already has a pretty comprehensive user model (storing user names, email addresses, roles, memberships etc). OTOH it could make great sense to enhance this data with all kinds of crazy things that could be stored in the web ui. Or to store some user data temporarily to reduce API calls. But this means we'd have two different data sources since the web ui should not access the core db directly.
MM has two kinds of authentication to perform. BrowserID "solves" both at the same time. First, it establishes that the person on the website is a particular user. But it also "proves" that the email address "belongs" to that user. Thus, with BrowserID, we can allow the user to join a mailing list without having to send the confirming email transaction.
Yep, very convenient. The process would be initiated via the client-JS and verified server-side in the django-backend. Without ever touching the REST API.
On the other hand, that confirming email transaction is basically what the browserid.org site does. Thus, with little effort, MM could also act as a secondary authority for the BrowserID protocol.
I think Barry has had some similar ideas...
I agree, no re-implementing the User model! In fact, I think we mean the same thing, because django's custom auth backends are simply a way to handle auth externally *and* tie that to the existing user/permissions system. Of course, if we implement such a custom backend we should do it in a generic and reusable way. I just think making a full-blown django app for what is usually just a .py file might be a bit overkill... But as I said: debatable... ;-) I guess in the end it's not a question of whether or not we make use of the Django User model. (At least to me) the question is: Do we use it to store user data permamently in the web ui or just temporarily on a request/session basis. Or a mix of both.
Well, not completely random, I guess ;-)
Florian

On Feb 21, 2012, at 02:00 AM, Florian Fuchs wrote:
Yep, it's a big question. Something could clearly make sense in the core, e.g. gpg keys for authentication or encryption. Other, specifically ui-type data might not make as much sense in the core. Even if we came up with a way to extend the core schema and store the data in the core, how would you access it? There would probably have to be a generic REST API for getting key/value data associated with the user in and out of the core.
I've have a similar question when it comes to authorization. Just where to do we keep the authorization mapping users and roles to actions they can perform? I'm very close to punting on this for 3.0 beta (and thus likely for 3.0 final), though I'm also happy to revisit it during the Pycon sprint (likely for a hypothetical 3.1).
Cheers, -Barry

On Feb 20, 2012, at 7:28 PM, Barry Warsaw wrote:
In general, I think that this will be needed. Consider what happens when we try to add a "plugin" optional function such as a different archiver.
In general, it will need to have some database stored parameters. Do we then add yet another database? I think not. A generic key/value mechanism in the existing database would provide the capability without requiring a custom change to the API.
For the short term, I would let the UI become the repository for such information. For the longer term, see my remarks in reply to Florian's message which you are referencing.
Richard

On Feb 21, 2012, at 08:10 AM, Richard Wackerbarth wrote:
Another way to do this, now that we have (what I think is a fairly robust) schema migration, you could imagine the plugin would make available a few migrations which would add the necessary data to the existing database. There might be problems if you're adding fields to existing model objects, but adding purely new tables shouldn't be too much of a problem I'd think.
Yep, thanks. -Barry

On Feb 25, 2012, at 3:37 PM, Barry Warsaw wrote:
First, I would be quite concerned if any plugins alter existing database tables. I see nothing but nightmares trying to avoid conflicting interactions, naming conventions, etc. From a maintenance point-of-view, adding auxiliary tables rather than additional columns in existing tables is much more manageable.
When it comes to migrations, we need to assure that migration sequences from separate plugins remain commutative. We will have no control over the order in which a particular installation will adopt particular plugins.
Where possible, I think that it is preferable to avoid add database tables if the scope of reference for those tables extends beyond the plugin itself. In order to bridge between the UI and the MM core servers, the REST interface would also have to be extended.
On the other hand, implementing the parameter store for a plugin by adding rows to a common parameter database would be able to utilize an existing interface.
Parameters are just key/value pairs that apply to a particular context. Generically, that context is a tuple of [user, list, ...] . If we add "plugin" to that context key, we have can have a generic store that does not need to be changed when a plugin is added.

Hi,
On Tuesday, February 21, 2012 02:28 CET, Barry Warsaw <barry@list.org> wrote:
Hmm, that doesn't sound like you particularly like that scenario. ;-) I'm not sure if it's a really good idea to use the core as a general data store. If user data (or even list data) is augmented with information the core doesn't need to operate, why not use a second data source. After all we have a system at hand (django) that's designed to do that.
I kind of liked the idea of an optional "bare-bones" version web ui that doesn't need anything more than the core API to do its tasks. But of course there are many use cases beyond that.
I've have a similar question when it comes to authorization. Just where to do we keep the authorization mapping users and roles to actions they can perform?
I'm not sure if that's what you mean, but I would say that depends on who the main authority is for, say, assigning a role to a certain user for instance. What if I revoke the moderator role from a member via the command line (I don't think that's currently possible, but let's say it is...). The core would have no way of actively inform the web ui (or possibly all the many different web ui installations, apps etc...) that this privilege has been revoked. So I'd say this is some information the web ui would have to request from the core.
Aahhh, the Pycon sprint... :-)
Cheers Florian

On Feb 23, 2012, at 11:04 PM, Florian Fuchs wrote:
Not so much. ;)
Exactly.
Yep. TBH, this is something I've been struggling with for a long while now. Remember I've advocated for keeping all those decisions out of the core, but there has been valid push-back on that (admittedly somewhat lazy ;) stance.
Now, I think we can get notifications pushed from the core to the ui if we wanted, assuming the ui had some service to notify. See my previous message on events. But in some cases, like the one you mention above, I think it would also be fine for the ui to query the core to gather the information necessary to decide whether an action is allowed. E.g. the ui could ask the core, via REST, "is this person an owner of that mailing list?".
There are upsides and downsides to this. For example, do we want to keep the logic for authorization in the web ui? That would mean if you integrate the core with your own custom web ui, then it wouldn't gain any of the benefit of the authorization logic in the Django app. OTOH, that might be just what you want, and I can imagine a site like Launchpad would prefer that, since it has its own user model. Ignoring that whole morass does keep the core lean and mean, which I rather like, especially because I want to release this beast RSN. ;)
So I'm very tempted to punt on the whole authentication/authorization issue as much as possible in the core for the 3.0 release, but I'm also happy to keep the discussion open for possible changes for the future. I think the integration work going on between the web ui and the core will give us some much needed practical experience here. Right now, it's all too theoretical for me to make much headway on.
Fun, fun!
Note too that I will be giving a talk on MM3 at Pycon, but fortunately I only have to fill 30m. :) I want to be able to release MM3 very soon after the sprints.
-Barry

Hi,
Awesome! So the wui would implement a simple REST API for the core to talk to. Ideally it should be possible to "register" more than one web ui (the official ui might not be the only application using the Mailman API), so multiple parties can be notified of changes. Also, we should probably think about how fine-grained those "invalidate"-notifications can be. It's probably not necessary to invalidate the whole cache every time something changes in the core.
I wonder what the main use cases are for custom web uis. My guess is that many ui-creators might not build the whole thing anew, but create smaller applications for single tasks like (un)subscribing to lists etc. In larger integration scenarios, auth might be taken care of at some other stage (a company intranet for instance or a content management system).
Note too that I will be giving a talk on MM3 at Pycon, but fortunately I only have to fill 30m. :)
I know - I'll definitely be there! (Always wanted to know what this MM3 thing is and what it can do! ;-)
Florian

On Feb 20, 2012, at 7:00 PM, Florian Fuchs wrote:
Agreed. Django's User model <snip>
OTOH it could make great sense to enhance this data with all kinds of crazy things that could be stored in the web ui. Or to store some user data temporarily to reduce API calls. But this means we'd have two different data sources since the web ui should not access the core db directly.
Let me suggest that we "step back" a bit from the implementation and look at the infrastructure architecture.
The Web UI is a Django site. Therefore, (IMHO), we should build it as one. That means that we have some django apps that represent the MM functionality. They have models and views just like any other django site. The models for the users, lists, etc. just "happen" to represent the same data structure as that used by the MM core.
The database access becomes a stack. The UI always accesses its data through its model. The implementation of the access could be directly as a table on a local database (as django is normally used) or it could use the REST interface to utilize the MM core as its data store. I would also introduce a transparent caching layer which would be optional. I suspect that (almost all of) the part of the MM database that is accessed by the UI could be classified as "read mostly" and that we could cache the information on a per-session basis, thus avoiding many of the accesses thru the REST interface.
Of course, if we implement such a custom backend we should do it in a generic and reusable way. I just think making a full-blown django app for what is usually just a .py file might be a bit overkill... But as I said: debatable... ;-)
I'm not sure what you mean by "full-blown django app". Even though an app must have models, views, templates, etc., those can usually be empty placeholder files.
I guess in the end it's not a question of whether or not we make use of the Django User model. (At least to me) the question is: Do we use it to store user data permamently in the web ui or just temporarily on a request/session basis. Or a mix of both.
We will definitely need some persistent storage that is beyond the functionality provided by the current MM core. Whether that storage is provided locally by a UI-owned database or the core database is extended to provide the backing store should be configurable at a later time.
Another approach would be to migrate all of the storage to a separate module and let both the UI and the "core" interface to it. The REST interface could also be layered on top of it to provide an alternate API that could be bypassed by the core and/or the UI. This would make provide a clean place to insert "foreign" databases as might be maintained by existing enterprise infrastructures.
Richard
Richard

Hi,
On Tuesday, February 21, 2012 14:59 CET, Richard Wackerbarth <richard@NFSNet.org> wrote:
The database access becomes a stack. The UI always accesses its data through its model. The implementation of the access could be directly as a table on a local database (as django is normally used) or it could use the REST interface to utilize the MM core as its data store.
Yep, it obviously makes sense to separate data from business logic. And ideally the view code doesn't look (much) different whether it accesses a model stored in a db or a rest resource. (BTW, I recently started rewriting this part of the prototype in that sense.) But I'm not sure I would see database models as exactly equivalent to REST resources.
I would also introduce a transparent caching layer which would be optional. I suspect that (almost all of) the part of the MM database that is accessed by the UI could be classified as "read mostly" and that we could cache the information on a per-session basis, thus avoiding many of the accesses thru the REST interface.
Yep. But we need to decide how tolerant we are when it comes to inconsistencies between the state of the core data and a web ui session. A list of mailing lists might be stored on a per session basis - if a new list is created by a third party during a session, it's probably tolerable not to have that new list appear immediately. It's a bit trickier when it comes to privileges like moderation etc. that should probably take effect immediately (especially when revoked).
Of course, if we implement such a custom backend we should do it in a generic and reusable way. I just think making a full-blown django app for what is usually just a .py file might be a bit overkill... But as I said: debatable... ;-)
I'm not sure what you mean by "full-blown django app". Even though an app must have models, views, templates, etc., those can usually be empty placeholder files.
By that I meant: Something people installing it have to take care of by tweaking their settings.INSTALLED_APPS tuple. If it's part of the package, that doesn't need to be done. Although having an extra app for that could make sense if the different auth methods should be strictly pluggable.
Cheers Florian

On Feb 23, 2012, at 4:46 PM, Florian Fuchs wrote:
If I am not mistaken, the INSTALLED_APPS as defined in the django instance settings, is just a convenient way to switch on/off django apps whose code is present. I think that an enabled app can effectively override the tupe and cause other apps to behave as if they had been listed in the settings.
However, I am not sure that to do so would be a good idea. The person who installs a plugin is going to have to do a number of things. In particular, he will, presumedly, have to do more than just add the code to the python path. He will modify some configuration parameter on the core side in order to have the plugin have any effect on the mail traffic.
Since we have separated the mail handling from the administrative interface (UI), he will also need to both make code available and alter some configuration setting on the UI side.
Just as in the django distribution, where the "admin" interface is optional, adding references to our plugins SHOULD involve adding lines to the settings files, etc. Note that we could provide a script that would automate those changes. However, as a system administrator, I prefer to be told in the installation instructions what lines I need to change, and why. I am leery of scripts that "magically" change configurations on my system.
Richard

On Feb 23, 2012, at 11:46 PM, Florian Fuchs wrote:
I wonder if a push model could be used in those cases where changes to the core database need to be immediately reflected in the ui. For example, the core utilizes zope.events as a hook mechanism to notify other parts of the core when certain things happen. For the most part, I'm just using this in the test suite, but I think it would make sense to hook events to notify external systems (e.g. the web ui, or its cache, etc.) when certain things happen.
Cheers, -Barry

Florian Fuchs writes:
Agreed. Django's User model most definitely covers all our needs.
What version of Django's User model are you using? The vanilla User is actually rather limited. Or do you mean as augmented by UserProfiles? If the latter, do you have a specific schema for the Profile in mind, or are you just referring to its flexibility?
We do, I think. I think the core DB should be constant across Mailman installations, and restricted to fields useful in "forum" administration (ie, not restricted to mailing lists, although that will be the main purpose of Mailman 3 at its initial release). The web UI DB storage would be optional and flexible. But all fields and tables available to the web UI should be accessed by the same APIs in developing an instance of the web UI.
Also, some fields of the core DB may be provided by a third party in vegetative state (eg, a personnel department). We may want to allow the web UI to augment, edit, or override some of those fields in presentation.
But this means we'd have two different data sources since the web ui should not access the core db directly.
Yes, the core DB and the web UI's persistent storage would be different. But clients of the web UI's API need not know that, or do they?

On Wednesday, February 22, 2012 07:49 CET, "Stephen J. Turnbull" <stephen@xemacs.org> wrote:
By "User model" I didn't mean the actual vanilla User model (sorry for not being precise). I meant its flexibility and that it can be easily enhanced. Also how it's automatically added to the request object and accessible from the templates etc.
Good point.
If they become aware of that, something probably went wrong... ;-)
Florian

On Feb 22, 2012, at 03:49 PM, Stephen J. Turnbull wrote:
The counter argument is that the core's database should only contain the data that is needed to do its function. A leaner core schema means it will be easier to migrate and maintain as time goes on. It also reduces bloat in the REST API, even if we provided a generic key/data API (which frankly I don't like).
E.g. if personnel department were central to the core's functionality, by all means, let's add it to the schema and expose it in the right places within the REST resource model. If not, then it really shouldn't be part of the core's schema.
Cheers, -Barry

On Feb 23, 2012, at 5:43 PM, Barry Warsaw wrote:
The counter argument is that the core's database should only contain the data that is needed to do its function.
Yet another argument would be: The MM suite can be partitioned logically into a message handler, an archiver, an administration-by-mail interface, a web-based interface, and a roster (et. al.) store.
Perhaps the core should store only that data which it, and no other part of the suite utilizes. Upon receipt of a message, if appropriate, it would forward the message to appropriate subsystems. If it is to be distributed to a roster, it would obtain the current roster from the storage subsystem.
The web UI, administration-by-mail subsystem and the corporate personnel department would interact with the data store in order to make changes to the rosters.
Perhaps this concept is in keeping with the "leaner core schema" approach. It also has the advantage that the data store is isolated from the MM message handler. That would make it easier for "large corporation" to implement the store within their existing corporate database.
Richard
participants (4)
-
Barry Warsaw
-
Florian Fuchs
-
Richard Wackerbarth
-
Stephen J. Turnbull