Mailman-Developers October 2019

mailman-developers@python.org

5 participants
6 discussions

GSOD Update
by ariessa.norramli＠gmail.com 25 Nov '19

25 Nov '19

Hi everyone, My Week 1 update can be viewed here: https://medium.com/@ariessa_norramli/google-season-of-docs-week-1-update-43… Week 2 update will be up by Saturday, 11pm, GMT8+. Thank you. Best Regards, Ariessa Norramli

1 11

Re: [Mailman-i18n] Mailman 3 Internationalization setup
by Abhilash Raj 17 Oct '19

17 Oct '19

On Thu, Oct 17, 2019, at 6:55 AM, Rémi Oudin wrote: > Hi! > > It seems that the projects are not public anymore. > Should we ask you to give us access for the translations? It should be, I haven't changed anything in past few days. What makes you think they aren't public? Are you not able to visit their page? Just as side note, the URL for Postorius is now changed to https://hosted.weblate.org/projects/gnu-mailman/postorius/ Everything else is same. > > Cheers, > -- > Guinness > > Attachments: > * signature.asc -- thanks, Abhilash Raj (maxking)

1 0

Improving the speed of mailman import21
by Abhilash Raj 08 Oct '19

08 Oct '19

Hi All, This morning, I set out to improve the performance of "mailman import21" command. If you have used it in the past, you will know that it is slow. Until now, I never had an idea about why? Here were my ideas: - Too many database calls and sqlite3 being the usual self Although, I forgot that it is slow irrespective of the database backend. Maybe we are doing way too many queries? - Too many string comparisons We all know string comparisons are slow, but how slow could they be? - Something wasteful being done over and over again. Here is a rough estimate of the time it takes to import mailman2.1's config.pck for two lists: 151 members: 58 seconds 1429 members: 9 minutes This is quote slow, 9 minutes is a lot. So, I set out to do the usual python profiling using the standard library `cProfile` module and only wrapped it around `mailman.utilities.importers._import_roster`. That method is the slowest one since if you have run the the command, you know it takes the maximum amount of time importing the list of members. Without even looking at the entire output, the problem was apparent and none of the ones that I guessed before: ncalls tottime percall cumtime percall filename:lineno(function) 1 0.009 0.009 50.692 50.692 /home/maxking/Documents/mm3/core/src/mailman/utilities/importer.py:600(_import_roster) 151 0.001 0.000 45.691 0.303 /home/maxking/Documents/mm3/core/src/mailman/utilities/passwords.py:35(encrypt) 90% of the time is spent trying to encrypt user passwords, for each of the imported member. Well, duh, encryption is an expensive operation and when you do that once per-imported member, it is definitely going to be slow. Mailman 3 uses passlib[1] for crypto and so I set out to figure out if there is a hashing algorithm which can do this much faster and perhaps has a C library wrapper that we can use to speed things up. I settled on argon2 cipher with a supporting library argon2_cffi. Then I changed the config and tried the imports again: 151 members: 15.884 seconds 1429 memebrs: 2minutes 29 seconds That was a significant improvement over the previous numbers. Although, another interesting fact is the user passwords are kind of useless in Mailman 3. In Mailman 2 you had to setup a password or one was auto-generated for you per-list and you needed that to login to the web ui. However, in Mailman 3, the passwords (in Core's database) aren't used for logging in since Web Frontend stores the authentication tokens (social auth or passwords). In fact, the users who sign up first time on Mailman 3 probably don't ever have a password set in Mailman Core's database. So, I commented out the code that actually imports the password(src/mailman/utilities/importer.py#L663-664) and the import speed improved even more, obviously: 151 members: 4 seconds 1429 members: 57 seconds I am hoping that I can commit the change with the commented out code, unless I am reminded of a use for the passwords in Core's database. Then, it might be a bit more of work trying to figure out another way to improve the speed. Thanks for reading up! [1]: https://passlib.readthedocs.io/en/stable/narr/quickstart.html#making-a-deci… -- thanks, Abhilash Raj (maxking)

4 6

Mailman 3 Internationalization setup
by Abhilash Raj 07 Oct '19

07 Oct '19

Hi All, Mailman 3 is now setup for continuous internationalization. We are using Hosted Weblate[1], which is the hosted service for the Weblate project[2]. It is actually quite nice and I've been learning how to properly document a Django project from it's documentation. Which brings me to it's documentation. Weblate has great documentation for translators[2]. The UI pretty simple and should be easy to navigate around. We have setup the workflow in a way that each commit in Gitlab generates a webhook and updates Weblate. Weblate is then able to pull all the changes in real time. To pull translations back into Gitlab, I have a custom script running every night that pulls translations (as commits) and generates pull requests against the upstream repos. Weblate has the ability to automatically push the translations back, but for security reasons, I didn't want to give a 3rd party service direct push access to our repos. I've been working on a Patch to Weblate[3], which would allow it to send translations as Merge Requests (as compared to direct-push). After it is merged and deployed at hosted.weblate.org, I hope I can retire my custom script. We have setup the following projects: - Mailman Core (https://hosted.weblate.org/projects/gnu-mailman/mailman/) - Postorius* (https://hosted.weblate.org/projects/gnu-mailman/translations/) - Hyperkitty (https://hosted.weblate.org/projects/gnu-mailman/hyperkitty/) - Django-mailman3 (https://hosted.weblate.org/projects/gnu-mailman/django-mailman3/) There should be empty .po files generated for most languages supported by Django and Mailman Core. If there are any questions or bugs related to source strings, please send out an email to mailman-developers(a)python.org. The current translation %age is pretty low, hopefully we'll be ale to get it close to Mailman 2 sometime soon :) [1]: https://hosted.weblate.org/projects/gnu-mailman/ [2]: https://docs.weblate.org/en/latest/user/translating.html [3]: https://github.com/WeblateOrg/weblate/pull/3024 [*] Yeah, the URL for Postorius is a bit weird, because it was called "translations" before. At the time, I didn't know the difference between a project and a translation component in Weblate terminologies. And they don't give out permissions to change the URL, so we're maybe stuck with it. -- thanks, Abhilash Raj (maxking)

1 0

Mailman 3 installation documentation
by Abhilash Raj 07 Oct '19

07 Oct '19

Hi All, I have been trying to work on revamping the documentation site we have at https://docs.mailman3.org. It is on https:// but I am not sure why the http->https redirect doesn't work in Readthedocs. I have set it to "serve always from https". Anyway, it might not look too different right now, but there are some small re-organizations. I have been learning some new ReST and Sphinx tricks and been trying to cross-reference and link various sections and settings. The docs site shows up Mailman's logo now (I didn't even know we could do that ;-). There is a place[1] for all supported Django settings that Postorius and Hyperkitty have. I am also planning to add some common Django settings that often people have to lookup and is not fun for people from non-python backgrounds. Hopefully, it will make lives easier for administrators. I have reverted my own stupid decision to use "sudo pip3 install" in the docs to instead use virtualenv. The installation docs are going to grow to be more comprehensive. I am also working on mailman-web project[3], which should simply installation process for all the web components using a single project and provide better guidance on Django's settings file management and howto run manage.py commands. I have added a `mailman-web` command, which, for those of you familiar with internal details of Django, is going to be our custom manage.py script. It sets up settings correctly and would hopefully reduce the pain of using `django-admin` and/or finding out the right path for `manage.py` script. It is still not ready for prime time, but it is usable. Early adopters can play with it if they like! I would really appreciate if people who have tried to manually install Mailman 3 in the past would share their experiences about what they think could be documented better. You can reply here or open an issue[2]. The user documentation are going to follow, hopefully soon :) [1]: https://docs.mailman3.org/en/latest/config-web.html#configure-postorius-hyp… [2]: https://gitlab.com/mailman/mailman-suite-doc/issues [3]: https://gitlab.com/mailman/mailman-web -- thanks, Abhilash Raj (maxking)

1 0

Umbrella Settings
by sandeep kumar 01 Oct '19

01 Oct '19

Hi Team I just want to know whether the umbrella list option is enabled in Maiman 3 or not. I have seen in the below link that umbrella list options are going to be replaced with new ones. https://www.gnu.org/software/mailman/mailman-admin/node12.html Is there any alternative for using this setting in MM3. -- Regards Sandeep Kumar +91-9642669192

3 2

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

1998

1997

Mailman-Developers October 2019