On Thu, Oct 17, 2019, at 6:55 AM, Rémi Oudin wrote:
> Hi!
>
> It seems that the projects are not public anymore.
> Should we ask you to give us access for the translations?
It should be, I haven't changed anything in past few days. What makes you think they aren't public? Are you not able to visit their page?
Just as side note, the URL for Postorius is now changed to
https://hosted.weblate.org/projects/gnu-mailman/postorius/
Everything else is same.
>
> Cheers,
> --
> Guinness
>
> Attachments:
> * signature.asc
--
thanks,
Abhilash Raj (maxking)
Hi All,
This morning, I set out to improve the performance of "mailman import21" command. If you have used it in the past, you will know that it is slow. Until now, I never had an idea about why? Here were my ideas:
- Too many database calls and sqlite3 being the usual self
Although, I forgot that it is slow irrespective of the database
backend. Maybe we are doing way too many queries?
- Too many string comparisons
We all know string comparisons are slow, but how slow could they be?
- Something wasteful being done over and over again.
Here is a rough estimate of the time it takes to import mailman2.1's config.pck for two lists:
151 members: 58 seconds
1429 members: 9 minutes
This is quote slow, 9 minutes is a lot. So, I set out to do the usual python profiling using the standard library `cProfile` module and only wrapped it around `mailman.utilities.importers._import_roster`. That method is the slowest one since if you have run the the command, you know it takes the maximum amount of time importing the list of members.
Without even looking at the entire output, the problem was apparent and none of the ones that I guessed before:
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.009 0.009 50.692 50.692 /home/maxking/Documents/mm3/core/src/mailman/utilities/importer.py:600(_import_roster)
151 0.001 0.000 45.691 0.303 /home/maxking/Documents/mm3/core/src/mailman/utilities/passwords.py:35(encrypt)
90% of the time is spent trying to encrypt user passwords, for each of the imported member. Well, duh, encryption is an expensive operation and when you do that once per-imported member, it is definitely going to be slow.
Mailman 3 uses passlib[1] for crypto and so I set out to figure out if there is a hashing algorithm which can do this much faster and perhaps has a C library wrapper that we can use to speed things up. I settled on argon2 cipher with a supporting library argon2_cffi. Then I changed the config and tried the imports again:
151 members: 15.884 seconds
1429 memebrs: 2minutes 29 seconds
That was a significant improvement over the previous numbers.
Although, another interesting fact is the user passwords are kind of useless in Mailman 3. In Mailman 2 you had to setup a password or one was auto-generated for you per-list and you needed that to login to the web ui. However, in Mailman 3, the passwords (in Core's database) aren't used for logging in since Web Frontend stores the authentication tokens (social auth or passwords). In fact, the users who sign up first time on Mailman 3 probably don't ever have a password set in Mailman Core's database.
So, I commented out the code that actually imports the password(src/mailman/utilities/importer.py#L663-664) and the import speed improved even more, obviously:
151 members: 4 seconds
1429 members: 57 seconds
I am hoping that I can commit the change with the commented out code, unless I am reminded of a use for the passwords in Core's database. Then, it might be a bit more of work trying to figure out another way to improve the speed.
Thanks for reading up!
[1]: https://passlib.readthedocs.io/en/stable/narr/quickstart.html#making-a-deci…
--
thanks,
Abhilash Raj (maxking)
Hi All,
Mailman 3 is now setup for continuous internationalization. We are using Hosted Weblate[1], which is the hosted service for the Weblate project[2]. It is actually quite nice and I've been learning how to properly document a Django project from it's documentation.
Which brings me to it's documentation. Weblate has great documentation for translators[2]. The UI pretty simple and should be easy to navigate around.
We have setup the workflow in a way that each commit in Gitlab generates a webhook and updates Weblate. Weblate is then able to pull all the changes in real time. To pull translations back into Gitlab, I have a custom script running every night that pulls translations (as commits) and generates pull requests against the upstream repos.
Weblate has the ability to automatically push the translations back, but for security reasons, I didn't want to give a 3rd party service direct push access to our repos. I've been working on a Patch to Weblate[3], which would allow it to send translations as Merge Requests (as compared to direct-push). After it is merged and deployed at hosted.weblate.org, I hope I can retire my custom script.
We have setup the following projects:
- Mailman Core (https://hosted.weblate.org/projects/gnu-mailman/mailman/)
- Postorius* (https://hosted.weblate.org/projects/gnu-mailman/translations/)
- Hyperkitty (https://hosted.weblate.org/projects/gnu-mailman/hyperkitty/)
- Django-mailman3 (https://hosted.weblate.org/projects/gnu-mailman/django-mailman3/)
There should be empty .po files generated for most languages supported by Django and Mailman Core. If there are any questions or bugs related to source strings, please send out an email to mailman-developers(a)python.org.
The current translation %age is pretty low, hopefully we'll be ale to get it close to Mailman 2 sometime soon :)
[1]: https://hosted.weblate.org/projects/gnu-mailman/
[2]: https://docs.weblate.org/en/latest/user/translating.html
[3]: https://github.com/WeblateOrg/weblate/pull/3024
[*] Yeah, the URL for Postorius is a bit weird, because it was called "translations" before. At the time, I didn't know the difference between a project and a translation component in Weblate terminologies. And they don't give out permissions to change the URL, so we're maybe stuck with it.
--
thanks,
Abhilash Raj (maxking)
Hi All,
I have been trying to work on revamping the documentation site we have at https://docs.mailman3.org. It is on https:// but I am not sure why the http->https redirect doesn't work in Readthedocs. I have set it to "serve always from https".
Anyway, it might not look too different right now, but there are some small re-organizations. I have been learning some new ReST and Sphinx tricks and been trying to cross-reference and link various sections and settings. The docs site shows up Mailman's logo now (I didn't even know we could do that ;-).
There is a place[1] for all supported Django settings that Postorius and Hyperkitty have. I am also planning to add some common Django settings that often people have to lookup and is not fun for people from non-python backgrounds. Hopefully, it will make lives easier for administrators.
I have reverted my own stupid decision to use "sudo pip3 install" in the docs to instead use virtualenv. The installation docs are going to grow to be more comprehensive.
I am also working on mailman-web project[3], which should simply installation process for all the web components using a single project and provide better guidance on Django's settings file management and howto run manage.py commands.
I have added a `mailman-web` command, which, for those of you familiar with internal details of Django, is going to be our custom manage.py script. It sets up settings correctly and would hopefully reduce the pain of using `django-admin` and/or finding out the right path for `manage.py` script. It is still not ready for prime time, but it is usable. Early adopters can play with it if they like!
I would really appreciate if people who have tried to manually install Mailman 3 in the past would share their experiences about what they think could be documented better. You can reply here or open an issue[2].
The user documentation are going to follow, hopefully soon :)
[1]: https://docs.mailman3.org/en/latest/config-web.html#configure-postorius-hyp…
[2]: https://gitlab.com/mailman/mailman-suite-doc/issues
[3]: https://gitlab.com/mailman/mailman-web
--
thanks,
Abhilash Raj (maxking)
Hi Team
I just want to know whether the umbrella list option is enabled in Maiman 3
or not.
I have seen in the below link that umbrella list options are going to be
replaced with new ones.
https://www.gnu.org/software/mailman/mailman-admin/node12.html
Is there any alternative for using this setting in MM3.
--
Regards
Sandeep Kumar
+91-9642669192
