Let's try to package mailman3 in Debian!

Hey,
As a Debian fanboy and also a mailman addict, I'd like to try packaging it in Debian. As I'm not a DD and as I'm not in a hurry, I'd like to take some time to think about what would be relevant.
For that, I'd like to have some help. And I was suggested to come here and ask.
I decided to use https://gitlab.com/groups/mailman as a basis for my work.
First of all, to create the source package, I've to think about what to put in it.
It seems that if I want to have a good source package, I need 5 repos :
- Mailman
- HyperKitty
- Postorius
- HyperKitty - MailMan Plugin
- MailmanClient
My first question is "Am I right?". And the second one is whould I consider looking into mailman-suite-doc and also have it in the source package?
On another way, I also see that there is a standalone postorius repo and also some django project files for HyperKitty in another one. I have the impression it is more designed for people to work on as standalone - forkable projects for other features, and so I wouldn't need them in a Debian package.
Am I also right?
Thanks for your answer, I hope I'll be able to make this package look as you'd like. :)
Cheers,
-- PEB

Pierre-Elliott Bécue writes:
In the spirit of the refactoring, I would say mailman-core (mailman + mailmanclient), hyperkitty, and postorious should be three separate packages in Debian. Exactly which repos are needed, I'm not sure.
Mailman3 (or mailman-suite) would be a trivial package that depends on all of the above.
No, Debian breaks out doc packages anyway. I think as long as upstream is a separate repo, you may as well just have a separate source package for mailman-suite-doc, too.
No, IIRC these are variants which don't need a separate webserver such as Apache (convenient for testing), but they're still closely tied to mailman core and mailmanclient.

On Fri, Sep 11, 2015 at 11:26:44AM +0900, Stephen J. Turnbull wrote: the hyperkitty mailman plugin
- I would make a mailman-core package, everything depends on that
- mailmanclient would be a package providing the python bindings
- postorius would be a package depending on mailman-core and mailmanclient
- hyperkitty would be a package depending on mailman-core and mailmanclient, it should also inlcude
- mailman-deploy (or some other name) would contain a django skeleton project. It would have optional dependencies on postorius and hyperkitty.
- mailman-suite would depend on all of the packages
All django projects can be run without a separate webserver. However, this should _not_ be used in production enviornments.
I am deploying several django projects using nginx and uwsgi, works like a charm. I would either link to or provide documentation about different options for deploying django projects. Anything applicable to django is applicable to postorius and hyperkitty
Since postorius and hyperkitty are django apps, you are going to need a django project to deploy them. The *_standalone repositories are such projects to deploy postorius or hyperkitty. In a production environment you would probably want to have both in one django project.
I would provide a skeleton with comments about what options need to be changed to get to a sane production system.
I am currently working on writing such a project skeleton that would be able to deploy postorius or hyperkitty or both. I will publish a repo on gitlab when it's functional

Simon Hanna writes:
Actually mailman-core is fully functional all alone.
That's not really true. Sure, you can telnet to the REST port and speak REST by hand, but that's hardly pleasant. At a minimum, you need mailmanclient. (There will be alternatives, there was a GSoC project to provide a client that integrates with Node.js. But at the moment that's not production-ready.)
Also, all this talk may be somewhat premature, as at present there's no authentication in the REST protocol, so everything needs to be on the same host. Andrew Stuart's authenticating proxy will help with that, but AFAIK it's not integrated yet. It may be worth waiting for that (and/or Mailman 3.1 with the migration story) before doing betas of the Debian package(s).

On ven. 11 sept. 2015 à 15:49:48, Stephen J. Turnbull wrote:
Good afternoon (UTC+0200),
Thanks for all your answers, I'm sorry not having answered earlier, but I was having an Internet free weekend.
Thanks for all your suggestions, I'll think about different sources packages, except maybe for mailman-core plus client?
Stephen, I was considering that 3.1 would be a better one to release in debian, but maybe starting before would be good to see how things would go. As I said, I did not for now packaged any big project, and I'm not a Debian Dev, so I might be slower than a well trained uploading DD.
If you think I'm wrong, I'm eager to wait, if not, maybe I should start slowly and try understand all dependencies and create a good bunch of sources packages so that we'll be able to go faster when 3.1 is out.
Thanks, and cheers,
-- PEB

On Sep 11, 2015, at 11:26 AM, Stephen J. Turnbull wrote:
Mailman3 (or mailman-suite) would be a trivial package that depends on all of the above.
Yep, though I'd go with something like mailman3-suite. Or you can reserve the 'mailman3' source package name for the umbrella package that depends on all the sub-components (e.g. mailman3-core, mailman3-client, etc.)
Cheers, -Barry

On Sep 11, 2015, at 12:49 AM, Pierre-Elliott Bécue wrote:
As a Debian fanboy and also a mailman addict, I'd like to try packaging it in Debian.
Excellent! Do check wnpp to make sure there's not already an ITP for Mailman 3. Note too that there is already a Mailman 2 package, so you might want to contact the pkg-mailman-hackers@lists.alioth.debian.org mailing list.
Yes, those are a good start. I'm in the "separate source packages" camp for each component. Since each upstream does separate tarball releases, separate source packages make the most sense, IMHO. Definitely for core and client. You'll probably want to use 'mailman3-' as a prefix for the source packages, e.g. mailman3-core, mailman3-client, etc.
You'll probably want to talk to some Debian Django developers on the best way to package Postorius and HK.
mailman-suite-doc doesn't have anything in it right now.
Cheers, -Barry

Le vendredi 11 septembre 2015 à 00:49:44+0200, Pierre-Elliott Bécue a écrit :
[packaging mailman3]
Hey,
Here is an update.
For now on, I focused on mailman3-core package in order to get good practices working.
One can find my work here : https://github.com/P-EB/mailman3-core
I'm working on having working systemd/sysv services and after that it'll be a good idea to try installing the package and see how it goes.
I also have to design a good config file for debian, (see debian/contrib/mailman.cfg in master branch). Any suggestion is welcome!
Cheers
-- PEB

Le lundi 23 novembre 2015 à 02:56:27+0100, Pierre-Elliott Bécue a écrit :
Small bump here, I'd appreciate if somebody finds the time to tell me two things:
- Is my config file enough for a start ?
- Is my systemd service good to have mailman started properly ?
I'd rather be sure it's okay before writing a sysv service.
You can find contrib files in debian/contrib.
Cheers :)
-- PEB

Le lundi 14 décembre 2015 à 17:01:57+0100, Pierre-Elliott Bécue a écrit :
Hello!
Before requesting for sponsorship, and packaging officially the other components of mailman3, I'd like some "testers" for the core package I built, in order to be sure that it works, and that I will not introduce some stupid caveats on the packaging of the other components.
.deb can be found here: http://peb.pimeys.fr/mailman/mailman3-core/ git repo can be found there: https://gitlab.pimeys.fr/PEB/mailman3-core and there: https://github.com/P-EB/mailman3-core
Any volunteer welcome! Please, I need your help, I cannot review my work alone! :)
-- PEB

Pierre-Elliott Bécue <becue@crans.org> wrote:
Hi Pierre,
I just tried to have a look in a stretch docker image.
There are three .deb files on your webserver
mailman3-core_3.0.0-1_all.deb mailman3-core_3.0.0-3_all.deb mailman3-core_3.0.0-3_amd64.deb
It is not immediately obvious which one to test, since -1 has actually a newer timestamp.
Mailman Core 3.0.1 and 3.0.2 have been released in the meantime
it does not work on stretch/sid because it is not compatible with Python 3.5, see https://gitlab.com/mailman/mailman/issues/181
it does not install on Jessie (even with jessie-backports because of missing dependencies) (just a note because of not working on stretch either)
The systemd unit won't work because the parameter for the configuration file is '-C', not '-c'
installing python3.4 and changing the shebang in /usr/bin/mailman leads to the following error
root@4ca9477a166c:~# /usr/bin/mailman -C /etc/mailman3/mailman.cfg start Starting Mailman's master runner /usr/bin/python3.4: can't open file '/var/lib/mailman/bin/master': [Errno 2] No such file or directory
The files are installed in /var/lib/mailman3/bin, but bin_dir in /etc/mailman3/mailman.cfg points to /var/lib/mailman/bin ... both directories are wrong from my POV, it should probably be /usr/lib/mailman3 or something like that
Fixing this up leads to
root@4ca9477a166c:~# /usr/bin/mailman -C /etc/mailman3/mailman.cfg start Starting Mailman's master runner Traceback (most recent call last): File "/var/lib/mailman3/bin/master", line 9, in <module> load_entry_point('mailman==3.0.0', 'console_scripts', 'master')() File "/usr/lib/python3/dist-packages/mailman/bin/master.py", line 536, in main with open(config.PID_FILE, 'w') as fp: FileNotFoundError: [Errno 2] No such file or directory: '/run/mailman3/master.pid'
because the directory does not exist. Since you are shipping a systemd unit only you should probably ship a .tmpfile as well.
I now have a running daemon. I haven't done anything with Mailman3 so far, so I need to read up on that first.
Bernhard

Le samedi 20 février 2016 à 19:52:49+0000, Bernhard Schmidt a écrit :
The good version is actually the 3.0.0-1, sorry for the remaining stuff, I removed it.
Yes, I'll ask to mailman devels how clever it'd be to take 3.0.2 for packaging now.
Yeah, during my tries I didn't met this issue. I guess it's because 3.5 wasn't in. I'll work on it as I'm not fully aware of how to enforce 3.4.
Backporting all mailman3 suite will not be simple I guess, but I intend doing so when and if my packaging for sid succeeds.
My bad, I really thought I've fixed it before sending this email.
During my discussions with Barry Warsaw, we agreed that except the mail binary, the others shouldn't be in /usr/bin as their name is "too generic".
This error is mailman3.cfg was in the same stack of patches that should have been committed before my email (with -C option)
During my initial test, I tried to run mailman3 standalone, and the directory stayed, so I never met the error. Fix committed. Thanks.
I now have a running daemon. I haven't done anything with Mailman3 so far, so I need to read up on that first.
Thanks for reviewing, sorry for the loss of time I induced.
-- PEB

Pierre-Elliott Bécue writes:
In the spirit of the refactoring, I would say mailman-core (mailman + mailmanclient), hyperkitty, and postorious should be three separate packages in Debian. Exactly which repos are needed, I'm not sure.
Mailman3 (or mailman-suite) would be a trivial package that depends on all of the above.
No, Debian breaks out doc packages anyway. I think as long as upstream is a separate repo, you may as well just have a separate source package for mailman-suite-doc, too.
No, IIRC these are variants which don't need a separate webserver such as Apache (convenient for testing), but they're still closely tied to mailman core and mailmanclient.

On Fri, Sep 11, 2015 at 11:26:44AM +0900, Stephen J. Turnbull wrote: the hyperkitty mailman plugin
- I would make a mailman-core package, everything depends on that
- mailmanclient would be a package providing the python bindings
- postorius would be a package depending on mailman-core and mailmanclient
- hyperkitty would be a package depending on mailman-core and mailmanclient, it should also inlcude
- mailman-deploy (or some other name) would contain a django skeleton project. It would have optional dependencies on postorius and hyperkitty.
- mailman-suite would depend on all of the packages
All django projects can be run without a separate webserver. However, this should _not_ be used in production enviornments.
I am deploying several django projects using nginx and uwsgi, works like a charm. I would either link to or provide documentation about different options for deploying django projects. Anything applicable to django is applicable to postorius and hyperkitty
Since postorius and hyperkitty are django apps, you are going to need a django project to deploy them. The *_standalone repositories are such projects to deploy postorius or hyperkitty. In a production environment you would probably want to have both in one django project.
I would provide a skeleton with comments about what options need to be changed to get to a sane production system.
I am currently working on writing such a project skeleton that would be able to deploy postorius or hyperkitty or both. I will publish a repo on gitlab when it's functional

Simon Hanna writes:
Actually mailman-core is fully functional all alone.
That's not really true. Sure, you can telnet to the REST port and speak REST by hand, but that's hardly pleasant. At a minimum, you need mailmanclient. (There will be alternatives, there was a GSoC project to provide a client that integrates with Node.js. But at the moment that's not production-ready.)
Also, all this talk may be somewhat premature, as at present there's no authentication in the REST protocol, so everything needs to be on the same host. Andrew Stuart's authenticating proxy will help with that, but AFAIK it's not integrated yet. It may be worth waiting for that (and/or Mailman 3.1 with the migration story) before doing betas of the Debian package(s).

On ven. 11 sept. 2015 à 15:49:48, Stephen J. Turnbull wrote:
Good afternoon (UTC+0200),
Thanks for all your answers, I'm sorry not having answered earlier, but I was having an Internet free weekend.
Thanks for all your suggestions, I'll think about different sources packages, except maybe for mailman-core plus client?
Stephen, I was considering that 3.1 would be a better one to release in debian, but maybe starting before would be good to see how things would go. As I said, I did not for now packaged any big project, and I'm not a Debian Dev, so I might be slower than a well trained uploading DD.
If you think I'm wrong, I'm eager to wait, if not, maybe I should start slowly and try understand all dependencies and create a good bunch of sources packages so that we'll be able to go faster when 3.1 is out.
Thanks, and cheers,
-- PEB

On Sep 11, 2015, at 11:26 AM, Stephen J. Turnbull wrote:
Mailman3 (or mailman-suite) would be a trivial package that depends on all of the above.
Yep, though I'd go with something like mailman3-suite. Or you can reserve the 'mailman3' source package name for the umbrella package that depends on all the sub-components (e.g. mailman3-core, mailman3-client, etc.)
Cheers, -Barry

On Sep 11, 2015, at 12:49 AM, Pierre-Elliott Bécue wrote:
As a Debian fanboy and also a mailman addict, I'd like to try packaging it in Debian.
Excellent! Do check wnpp to make sure there's not already an ITP for Mailman 3. Note too that there is already a Mailman 2 package, so you might want to contact the pkg-mailman-hackers@lists.alioth.debian.org mailing list.
Yes, those are a good start. I'm in the "separate source packages" camp for each component. Since each upstream does separate tarball releases, separate source packages make the most sense, IMHO. Definitely for core and client. You'll probably want to use 'mailman3-' as a prefix for the source packages, e.g. mailman3-core, mailman3-client, etc.
You'll probably want to talk to some Debian Django developers on the best way to package Postorius and HK.
mailman-suite-doc doesn't have anything in it right now.
Cheers, -Barry

Le vendredi 11 septembre 2015 à 00:49:44+0200, Pierre-Elliott Bécue a écrit :
[packaging mailman3]
Hey,
Here is an update.
For now on, I focused on mailman3-core package in order to get good practices working.
One can find my work here : https://github.com/P-EB/mailman3-core
I'm working on having working systemd/sysv services and after that it'll be a good idea to try installing the package and see how it goes.
I also have to design a good config file for debian, (see debian/contrib/mailman.cfg in master branch). Any suggestion is welcome!
Cheers
-- PEB

Le lundi 23 novembre 2015 à 02:56:27+0100, Pierre-Elliott Bécue a écrit :
Small bump here, I'd appreciate if somebody finds the time to tell me two things:
- Is my config file enough for a start ?
- Is my systemd service good to have mailman started properly ?
I'd rather be sure it's okay before writing a sysv service.
You can find contrib files in debian/contrib.
Cheers :)
-- PEB

Le lundi 14 décembre 2015 à 17:01:57+0100, Pierre-Elliott Bécue a écrit :
Hello!
Before requesting for sponsorship, and packaging officially the other components of mailman3, I'd like some "testers" for the core package I built, in order to be sure that it works, and that I will not introduce some stupid caveats on the packaging of the other components.
.deb can be found here: http://peb.pimeys.fr/mailman/mailman3-core/ git repo can be found there: https://gitlab.pimeys.fr/PEB/mailman3-core and there: https://github.com/P-EB/mailman3-core
Any volunteer welcome! Please, I need your help, I cannot review my work alone! :)
-- PEB

Pierre-Elliott Bécue <becue@crans.org> wrote:
Hi Pierre,
I just tried to have a look in a stretch docker image.
There are three .deb files on your webserver
mailman3-core_3.0.0-1_all.deb mailman3-core_3.0.0-3_all.deb mailman3-core_3.0.0-3_amd64.deb
It is not immediately obvious which one to test, since -1 has actually a newer timestamp.
Mailman Core 3.0.1 and 3.0.2 have been released in the meantime
it does not work on stretch/sid because it is not compatible with Python 3.5, see https://gitlab.com/mailman/mailman/issues/181
it does not install on Jessie (even with jessie-backports because of missing dependencies) (just a note because of not working on stretch either)
The systemd unit won't work because the parameter for the configuration file is '-C', not '-c'
installing python3.4 and changing the shebang in /usr/bin/mailman leads to the following error
root@4ca9477a166c:~# /usr/bin/mailman -C /etc/mailman3/mailman.cfg start Starting Mailman's master runner /usr/bin/python3.4: can't open file '/var/lib/mailman/bin/master': [Errno 2] No such file or directory
The files are installed in /var/lib/mailman3/bin, but bin_dir in /etc/mailman3/mailman.cfg points to /var/lib/mailman/bin ... both directories are wrong from my POV, it should probably be /usr/lib/mailman3 or something like that
Fixing this up leads to
root@4ca9477a166c:~# /usr/bin/mailman -C /etc/mailman3/mailman.cfg start Starting Mailman's master runner Traceback (most recent call last): File "/var/lib/mailman3/bin/master", line 9, in <module> load_entry_point('mailman==3.0.0', 'console_scripts', 'master')() File "/usr/lib/python3/dist-packages/mailman/bin/master.py", line 536, in main with open(config.PID_FILE, 'w') as fp: FileNotFoundError: [Errno 2] No such file or directory: '/run/mailman3/master.pid'
because the directory does not exist. Since you are shipping a systemd unit only you should probably ship a .tmpfile as well.
I now have a running daemon. I haven't done anything with Mailman3 so far, so I need to read up on that first.
Bernhard

Le samedi 20 février 2016 à 19:52:49+0000, Bernhard Schmidt a écrit :
The good version is actually the 3.0.0-1, sorry for the remaining stuff, I removed it.
Yes, I'll ask to mailman devels how clever it'd be to take 3.0.2 for packaging now.
Yeah, during my tries I didn't met this issue. I guess it's because 3.5 wasn't in. I'll work on it as I'm not fully aware of how to enforce 3.4.
Backporting all mailman3 suite will not be simple I guess, but I intend doing so when and if my packaging for sid succeeds.
My bad, I really thought I've fixed it before sending this email.
During my discussions with Barry Warsaw, we agreed that except the mail binary, the others shouldn't be in /usr/bin as their name is "too generic".
This error is mailman3.cfg was in the same stack of patches that should have been committed before my email (with -C option)
During my initial test, I tried to run mailman3 standalone, and the directory stayed, so I never met the error. Fix committed. Thanks.
I now have a running daemon. I haven't done anything with Mailman3 so far, so I need to read up on that first.
Thanks for reviewing, sorry for the loss of time I induced.
-- PEB
participants (5)
-
Barry Warsaw
-
Bernhard Schmidt
-
Pierre-Elliott Bécue
-
Simon Hanna
-
Stephen J. Turnbull