Mailman 3 April 2013 - Mailman-Developers

Re: [Mailman-Developers] Architecture for extra profile info
by Barry Warsaw 19 Apr '13

19 Apr '13

My main suggestion for now is to be very careful and not over-engineer the user database component. Provide something minimal that fits the bill and has a minimum of security, e.g. basic-auth over localhost, and possibly https. For now, I think it would be fine as a Django app if that makes things easier, but also remember how much pain we had at the sprint trying to get Postorius and HyperKitty deployed together (how's that coming by the way?). OTOH, do the easiest thing that will allow our GSoC students to succeed but that doesn't box us in later. E.g. providing a REST API makes sense, and it's okay if there aren't fancy UI to change the schema (unless it's easy using Django). Eventually OAuth is a good idea and I'm not aware of anything else that fits the bill as well, for authenticated scripting of REST APIs. But we probably don't need it for now. One important requirement is that for any data that is kept in both the core and the user database, we must have a way of keeping them in sync. The easiest way of doing that I think is to allow two way communication between them so that if data changes in the core (e.g. an address gets verified by reply instead of link-click), the core can inform the user database of this event. Eventually, we can think about how the core would just share that information or delegate to the user database, but for now, and for the GSoC students, it's probably overkill. -Barry

2 1

Re: [Mailman-Developers] Architecture for extra profile info
by Stephen J. Turnbull 19 Apr '13

19 Apr '13

Main comment: Sounds like LDAP to me. Florian Fuchs writes: > 5) It should implement an oAuth provider. I don't see this. Mailman is an auth consumer. The only people Mailman can provide auth for are the site admins. Everybody else is more or less untrustworthy. I can see that there are applications where it would be useful to have an auth provider bundled with Mailman, but I think implementing it is somebody else's job. > This could be used for API authenticaion and to log into > Postorius/Hyperkitty I think generic auth provider is overkill for these purposes, and a trap for anybody who thinks we know enough about crypto/security to do this stuff well.

5 7

Re: [Mailman-Developers] anti-spam filter
by Patrick Ben Koetter 19 Apr '13

19 Apr '13

* Pratik Sarkar <iampratiksarkar(a)gmail.com>: > Hi guys, > Is the anti-spam/abuse filter still being seriously considered > as a gsoc project this year? I - personal opinion - think adding a anti-spam/abuse filter is a good idea. I do however stronly oppose against a hardcoded implemenation that e.g. integrates SpamAssassin only or a new Bayesian filter. Why? (Please don't take this personal...) Reinvention I think it is a waste of time to reinvent (Bayes) something that has been there for many years. Proprietary implementation I also think it is no good service to those who will also want to expand Mailmans capabilities to do a product specific implementation instead of building a generic interface. I think, if Mailman should become able to filter spam and viruses, we should use a standardized interface (e.g. MILTER, SMTP/LMTP transport) that allows _any_ kind of mail filter to be hooked into the receiving process. Then anyone would be able to add their secret sauce when it comes to process email. This way the (new) Bayesian filter, which has been suggested on the list, could interface with Mailman too. p@rick -- [*] sys4 AG http://sys4.de, +49 (89) 30 90 46 64 Franziskanerstraße 15, 81669 München Sitz der Gesellschaft: München, Amtsgericht München: HRB 199263 Vorstand: Patrick Ben Koetter, Axel von der Ohe, Marc Schiffbauer Aufsichtsratsvorsitzender: Joerg Heidrich

6 10

Re: [Mailman-Developers] Architecture for extra profile info
by Richard Wackerbarth 19 Apr '13

19 Apr '13

On Apr 18, 2013, at 1:54 PM, Florian Fuchs <flo.fuchs(a)gmail.com> wrote: > 2013/4/18 Richard Wackerbarth <rkw(a)dataplex.net>: >> Whoa! Perhaps I don't understand oAuth. I thought that oAuth (and persona, kerberos, etc.) were protocols whereby one system (the provider) furnishes credentials for a second system (the client) to some third system (the consumer). >> >> By configuration, the consumer trusts that the provider has verified the client's identity and furnished appropriate credentials. >> Thus, when the client presents credentials in an interaction with the consumer, the consumer provides services on the basis of the credentials. >> >> If we assume that we distribute the MM implementation to include more than the two (core and web UI) systems by having, for example, a user manager, there might be an argument for passing around such credentials. > > I was primarily thinking about the (future) authenticated REST API in > Postorius. In that scenario a third party web app that we don't know > would request an access token from the user profile store (provider) > as well a user's email address. It would then use that access token > for requests to the Postorius API (consumer). > > If the instance of the user store does not act as provider, we would either: > > 1) effectively require every api user to have an account with some > other oauth provider. > > or > > 2) use some other authentication method. > > > 1) is odd in my opinion. Mailman is free software. And there are not > too many oauth providers that match that philosophy. > 2) would be totally great if we found something that doesn't require a > user to provide his credentials to said 3rd party app. Look at what happens now. The user contacts Postorius and presents either some oAuth/BrowserID-style credential or it presents a username/password. In exchange, Postorius issues a local credential (session key). When the username/password path is chosen, Postorius acts as an agent for the user and verifies the login. In this case, Postorius must "see" the underlying credential (username/password pair) to be able to present it to the authenticator. In the oAuth path, Postorius never sees the underlying credential, but exchanges the presented one for its local credential. Since we consider the user manager to be a part of the MM complex, what have we gained by hiding the underlying credential from the web interface? Therefore, I would suggest that we need to be able to accept third-party (oAuth) credentials and also locally presented ones. I don't see where we gain an advantage by exposing the authenticator as a separate oAuth provider. > I don't know. Maybe I'm missing something. And of course I agree with > Stephen: If it's too hairy in terms of security, we should not do it. > > Anyway, we're talking about something that is absolutely not needed > for what we want to achieve *right now*, which is a profile data store > that Postorius/HK/etc can access from localhost (or maybe from an > internal network. Or IP-restricted through SSL. Or... ?). > > Florian

3 3

Re: [Mailman-Developers] Build a mailman3 development virtualmachine with Vagrant and Chef
by Mark Sapiro 19 Apr '13

19 Apr '13

Xu Wang wrote: >I'm not sure I have the permission to edit the wiki page. If anyone with >the permission and know how, please feel free to add the link. As it says at the upper left of the wiki home (dashboard) page "to add or edit content you must sign up and log in, and you must also request write permission for your user name by sending a note to the Mailman Steering Committee. (sorry it's the only way to control wiki spam)." You now have permission. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 1

Re: [Mailman-Developers] Architecture for extra profile info
by Barry Warsaw 19 Apr '13

19 Apr '13

On Apr 18, 2013, at 09:39 PM, Florian Fuchs wrote: >I think in this case it makes sense to spend a more time on the API as >seen from the outside (urls, methods, json responses) than the actual >implementation. If the API is good, it's totally acceptable if the >underlying implementation will be redesigned later. +1 -Barry

1 0

Re: [Mailman-Developers] Architecture for extra profile info
by Richard Wackerbarth 19 Apr '13

19 Apr '13

On Apr 18, 2013, at 1:19 AM, Florian Fuchs <flo.fuchs(a)gmail.com> wrote: > Hi, > > some first thoughts on it: > > 1) It should be self-contained. > Meaning: It should not depend on any > mailman/mailman.client/postorius/hyperkitty packages. Agreed > 2) Like the core, it should implement a Python-based webserver. > It doesn't need to run on Ports 80/443, so we don't have to care about > one of the popular web servers already listening to those ports. > Also, we definitely don't want admins to have to follow another "how > to setup mailman.users with Apache/mod_wsgi" howto. > It should just run if someone says "start" (Maybe it can even hook > into $ bin/mailman start? I know, that contradicts item 1. ...). Where it runs is something that should be easily configurable as a part of installation. To simplify installation, I would advocate that all of the MM subsystem interfaces become self-documentating. By doing so, much of the configuration could be done through a "discovery" mechanism. That would provide an easy way to have modular plug-in extensions. It would also provide a way for Postorius to automatically adapt to the addition of configuration settings or user fields. > 3) It doesn't need Django. > Since it will not deliver any HTML (except an oAuth login form -- see > 5.) and it doesn't need to be integrated into any existing web site, > we can choose a more light-weight framework. Here I take exception. Dismissing Django is a restriction that unnecessarily affects the ease of implementation and, in the common case, complicates the integration of the functionality. Although it could be implemented without Django, it could also be implemented as a Django "app". An instance of a django server can then serve the functionality. As an alternative, where appropriate, this "app" would directly "drop in" to an instance of Postorius or an enterprise website. One of the advantages of Django is that it can be used as a rapid prototyping mechanism. Simplified interfaces to the data are "free" and more elaborate ones can be added in an incremental fashion. Also, rather than writing custom modules for things such as authentication and REST interfaces, there is the large community of third-party extensions which readily integrate to provide that functionality. > 4) Adding new content types for user records should be easy, but > clearly defined. > We don't know what information applications need to store. Icons, > essays, avatars, IRC handles, Twiter names, ... > So we might think about using a schema-less database, but: We don't > want to make it possible to just manipulate the result JSON and POST > it back to the resource, possible deleting things other apps need. So > adding new information types should be a separate process. > > 5) It should implement an oAuth provider. > This could be used for API authenticaion and to log into > Postorius/Hyperkitty (even on other servers. Hint: Reputation!) > We won't need this for a first prototype though. For now we're > probably fine with 6) > > 6) Like the core it should be accessible with BasicAuth from localhost. > Ideally, in the future, it should be accessible both via BasicAuth > from localhost and via oAuth from the outside world... > > Please correct me where I'm being stupid! > > Florian > 2013/4/18 Terri Oda <terri(a)zone12.com>: >> Background for folk new to this discussion: >> >> Currently, all user information is stored in Mailman core, but it's minimal: >> a real name, a set of email addresses, subscription info, and preferences. >> Barry suggests that it should stay minimal: only the things Mailman needs to >> know to correctly deliver mail (which actually doesn't include "real name" >> but let's leave that as a legacy item for the moment) I would advocate that this "User" module make it appear as if stores the entire "record" for the user. In the implementation, it could actually store parts of the user information in multiple databases (one of which could be the MM-core). This would also allow the option of having the MM-core become a client of this User module, just as it now relies on an external message store. >> It's pretty likely that future features of Mailman will want to attach extra >> information to users. Some of it will be social-y stuff like user icons for >> HyperKitty to display in the archives. Other things include metrics like >> "when did this person last post to the list?" or "how many posts have they >> made over the lifetime of this list?" One thing I know of is that Systers >> requires a short essay for all new subscribers, explaining why they want to >> join the list. (And they're considering porting this feature to Postorius, >> which means we potentially want an answer to "where will the extra profile >> data get stored?" before their students start coding.) >> >> So... >> >> I think we've sort of agreed that it would be best if whatever we built just >> had a rest interface and hyperkitty/postorius/whatever would talk to it >> through there, and could share data that way, but we need a simple prototype >> that folk (particularly students) will be able to start using, and there's >> still some internal architecture decisions that need to be made. >> >> Does anyone have time to build such a thing or write up some short >> architectural documents so a student could build such a thing in relatively >> short order? It doesn't have to be the perfect final design, but we >> probably need a basic starter api for adding, accessing, editing and >> possibly even removing profile data. >> >> Thoughts? >> >> Terri

4 5

Re: [Mailman-Developers] Build a mailman3 development virtual machine with Vagrant and Chef
by Terri Oda 19 Apr '13

19 Apr '13

Yeay, thanks! If you've got time, could you make sure your vm is linked on the mailman wiki docs? You can replace the comment I have on the GSoC page for sure, and there's probably a good place to link it in a couple "setting up your dev environment" pages. Terri On 13-04-18 1:39 AM, Xu Wang wrote: > Hi there, > > Although mailman3 installation is well documented, but putting everything > together is still not a trivial job. > > To ease the task, I made a mailman3 chef cookbook and vagrant file: > > https://github.com/xuwang/mailman3-vbox > > If you already had virtualbox/vagrant installed on your mac or pc, build a > working mailman3 vm should only take a few minutes. > > I'm sure there will be things I missed or got it wrong, but it just a > starter. > > Hope this helps, > > Xu Wang > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers(a)python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/terri%40zone12.com > > Security Policy: http://wiki.list.org/x/QIA9 >

2 1

Re: [Mailman-Developers] Architecture for extra profile info
by Richard Wackerbarth 18 Apr '13

18 Apr '13

Whoa! Perhaps I don't understand oAuth. I thought that oAuth (and persona, kerberos, etc.) were protocols whereby one system (the provider) furnishes credentials for a second system (the client) to some third system (the consumer). By configuration, the consumer trusts that the provider has verified the client's identity and furnished appropriate credentials. Thus, when the client presents credentials in an interaction with the consumer, the consumer provides services on the basis of the credentials. If we assume that we distribute the MM implementation to include more than the two (core and web UI) systems by having, for example, a user manager, there might be an argument for passing around such credentials. However, the design that has been followed thus far does not have the client communicating directly with the consumer system. Instead, the web UI interacts as an agent for the client. In this model, we have implicitly trusted the agent to properly represent the client and also screen client requests in accordance with system policy. Thus, although we need some level of authentication of the agent, there is no need for third party credentials such as those implemented in oAuth. A connection on "localhost" is a form of credential. Trusting that the OS design restricts access to the connection, and trusting the applications running on that host provides a level of identification and trust. There is no reason why alternate channels cannot be substituted as long as a means of identification (such as shared secrets) is utilized. In those cases, the security of the communication channel and the trustworthiness of the agent system need to be considered. However, in a logical sense, the interaction is the same as one using the localhost channel. On Apr 18, 2013, at 4:27 AM, Florian Fuchs <flo.fuchs(a)gmail.com> wrote: > 2013/4/18 Stephen J. Turnbull <stephen(a)xemacs.org>: >> Florian Fuchs writes: >> >>> 5) It should implement an oAuth provider. >> >> I don't see this. Mailman is an auth consumer. The only people >> Mailman can provide auth for are the site admins. Everybody else is >> more or less untrustworthy. >> >> I can see that there are applications where it would be useful to have >> an auth provider bundled with Mailman, but I think implementing it is >> somebody else's job. >> >>> This could be used for API authentication and to log into >>> Postorius/Hyperkitty >> >> I think generic auth provider is overkill for these purposes, and a >> trap for anybody who thinks we know enough about crypto/security to do >> this stuff well. > > I agree it's probably not easy. And, yes, maybe we need someone to > help us with that. > > But maybe we can take a moment to think about the usefulness of such a > feature and the possibilities this might open up, rather than > dismissing the use of a certain technology right off the bat. If we're > unsure we can implement this in a secure way, we can still say no to > this. Also, who says such a feature would be enabled by default? We > can add it as an "experts only" thing and leave it up to the admins to > make sure they use it in a secure environment. > > I remember several discussions during PyCon(s) and on IRC where > scenarios of different mailman instances talking to each other came > up. Of course that doesn't mean implementing a generic oAuth provider > is the only answer to this. If there are better and easier solutions, > fine. > > Luckily going beyond localhost isn't something we need for the > prototype that Terri suggested to be built for GSoC. > > Florian > _______________________________________________ > Mailman-Developers mailing list > Mailman-Developers(a)python.org > http://mail.python.org/mailman/listinfo/mailman-developers > Mailman FAQ: http://wiki.list.org/x/AgA3 > Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ > Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/rkw%40dataplex.net > > Security Policy: http://wiki.list.org/x/QIA9

3 8

Re: [Mailman-Developers] GSOC 2013 : Authenticated REST-API in Postorius/Django
by Barry Warsaw 18 Apr '13

18 Apr '13

On Apr 18, 2013, at 01:56 PM, Rahul Gaur wrote: >Talking about authentication, django-restframework has a wrapper for OAuth2 >out of the box, but I will have to check how to use non form data sources. I keep hearing terrible things about OAuth2. OAuth1 is nice and simple and there are plenty of good libraries in many different languages to support it, so we want to think about which version of OAuth to support carefully. Maybe there are security issues involved. >I know the support for python 3 is essential to the project, but as of now >sticking I would prefer to sticking to Python 2.7.x for now. As a practical matter, MM3.0 will be Python 2.7. The core can't be ported until restish is ported, and I don't have high hopes for Storm ever being ported, so we'd have to also switch to SQLAlchemy or some other Python 3 compatible ORM (if there is another one). That's way too much work for the 3.0 release. What I would say though, is to be as careful as you can so that you will be able to port to Python 3 when the time comes. This means making your code `python2.7 -3` clean and being *very* certain of your bytes vs. strings story. Essentially, you want to write your code to be bi-lingual without of course using any Python 3 idioms. This page may help: https://wiki.ubuntu.com/Python/3 -Barry

1 0