Mailman 3 December 2014 - Mailman-Developers

Re: [Mailman-Developers] Python 3
by Barry Warsaw 27 Dec '14

27 Dec '14

On Dec 26, 2014, at 10:32 PM, Aurelien Bompard wrote: >Right, I'm having issues too with the config file not being initalized. >I'll have a closer look. Awesome, thanks. >I'm not exclusively using the REST API though. I'm importing a couple >interfaces, mostly the archiver interface. I'm also using the custom Message >class a lot in the tests. And I'm importing a couple database types for >compatibility in my own schemas (Enum and UUID) I wonder if it would be possible to refactor some of this code into separate libraries, which would reduce the surface area for bilingual support. The config object is probably the tricky part here, but also the Message class has a lot less in it now that we don't have to worry about all that unicode stuff. Really, it's essentially just the .sender and .senders properties and some backward compatible pickle support (unnecessary atm for MM3 and probably not much needed going forward, unless the stdlib Message object in some future Python grows internal attributes that break unpickling - I'll certainly test that against what will be Python 3.5). >But I think the main problem is the import of mailman's config object in the >class that implements the IArchiver interface. I don't believe there's >another way to get the configuration. But there could be! I took a very quick look at HK (but not KittyStore) and it doesn't look like you need much. What if I added a REST API to access the core's system configuration settings? They would be read-only, and I would only start with the minimum of what HK needs. I think it's probably dangerous in the long term to expect the core's internal config API to remain stable, so accessing that through REST would provide the guarantees you need, in addition to decoupling HK from the core. >And now that I think of it, the archiver interface will be imported by >Mailman core, and will thus run with a Python3 interpreter. As a result, all >of KittyStore must at least be Python3 compatible. Yep, I missed that. Is it possible to feed HK messages out-of-process? E.g. via the command line? If so, then we write HK's IArchiver implementation to make those out of process calls, and we can even pull that bit into the core. I would be happy to help with that. Cheers, -Barry

3 4

Re: [Mailman-Developers] Python 3
by Barry Warsaw 27 Dec '14

27 Dec '14

On Dec 26, 2014, at 11:02 PM, Aurelien Bompard wrote: >> But there could be! I took a very quick look at HK (but not KittyStore) >> and it doesn't look like you need much. What if I added a REST API to >> access the core's system configuration settings? > >That would work. It turned out to be almost ridiculously easy, so that probably means it's a good idea <wink>. Quoting from the documentation: ==================== System configuration ==================== The entire system configuration is available through the REST API. You can get a list of all defined sections. >>> dump_json('http://localhost:9001/3.0/system/configuration') http_etag: ... sections: ['antispam', 'archiver.mail_archive', 'archiver.master', ... You can also get all the values for a particular section. >>> dump_json('http://localhost:9001/3.0/system/configuration/mailman') default_language: en email_commands_max_lines: 10 filtered_messages_are_preservable: no http_etag: ... layout: testing noreply_address: noreply pending_request_life: 3d post_hook: pre_hook: sender_headers: from from_ reply-to sender site_owner: noreply(a)example.com Dotted section names work too, for example, to get the French language settings section. >>> dump_json('http://localhost:9001/3.0/system/configuration/language.fr') charset: iso-8859-1 description: French enabled: yes http_etag: ... -----end quote----- Of course, these configuration resources are read-only. Also, I am deprecating the `/<api>/system` resource path; for backward compatibility this will still return the system versions, but you'll notice that the self_link now points to `<api>/system/versions` which is the new canonical resource path. I'll remove the old path when the API version gets rev'd. This makes a good point though: I'm not outright discouraging applications from using core as a library, and in fact that was part of the point of making everything importable via the `mailman` package. But I'm far from ready to make any guarantees of stability for the internal API, including interfaces. The REST API however I *am* making guarantees about, as evidenced by using an API version at the root of the resource tree. Note that configuration values are verbatim strings. You might need to convert the value to some typed data, e.g. [mailman]pending_request_lifetime has a value of '3d' which is shorthand for a timedelta. Clients must convert those data types themselves. Similarly for ints, booleans, etc. Multiline or whitespace separated strings values must be .split() by the client. But you *do* have access to all configuration settings now, although only on the py3 branch atm. Cheers, -Barry

1 0

Re: [Mailman-Developers] Python 3
by Barry Warsaw 26 Dec '14

26 Dec '14

On Dec 26, 2014, at 11:02 PM, Aurelien Bompard wrote: >> But there could be! I took a very quick look at HK (but not KittyStore) >> and it doesn't look like you need much. What if I added a REST API to >> access the core's system configuration settings? > >That would work. Cool, let me see what I can whip up. >> Yep, I missed that. Is it possible to feed HK messages out-of-process? >> E.g. via the command line? > >Not now. Would an LMTP server be appropriate? If so, could Mailman's internal >LMTP server be split off into a Py2-compatible library? What protocol would >be best suited in your opinion? LMTP might be a very interesting way to do it. The core's LMTP server is actually a relatively thin layer over the stdlib's smtpd library, and not much really had to be changed in the Py3 port (IIRC, the error status messages changed from bytes to strs, but that should be relatively easy to straddle via the six package). The relevant API is IArchiver.archive_message() which takes an mlist and a msg, since that's the API that sends the message to the archiver. .permalink() would be implemented in the core. It looks like HK/KS just needs the fqdn listname of the destination list, plus the message. The latter would be available over LMTP of course, and while the list name could usually be dug out of the message headers, it would probably be best to provide it in a special Mailman header. The alternative of course is adding a REST API to HK, but that seems a little heavier weight for what would probably be just one API (for now anyway). What do you think? Cheers, -Barry

1 0

Re: [Mailman-Developers] Python 3
by Barry Warsaw 26 Dec '14

26 Dec '14

On Dec 26, 2014, at 04:38 PM, Aurelien Bompard wrote: >This should be pretty easy, something like the following command should work: >alembic -c src/mailman/config/alembic.cfg revision --autogenerate -m "python3 >port" I'm having some trouble with this because the system alembic doesn't have access to the mailman source tree, which it needs (you get import errors). I tried to run alembic out of the .tox virtualenv, but that fails because the mailman config system needs to be initialized. Early on in the SA branch, there was an alembic subcommand to the mailman command. I don't think I want that, but I think we need something that calls into alembic programmatically, but with sys.path set up correctly, and the mailman system properly initialized. Maybe a 'withlist' script will do the trick. Anyway, I'll spend some time working on this and follow up here if needed. > >[...] then I'll merge this to trunk and do a 3.0b5 release as a > Python >3-only application. Mailman 3 will not be bilingual so Python 2 > support >will be dropped. > >This means I must now port HyperKitty and Kittystore to Python3 Oh gosh, no! I'm sorry I was imprecise, but I meant only that Mailman 3 *core* would be Python 3. I certainly would never make you guys do all that busy work without consulting you first. So for the record, HyperKitty and Postorious can absolutely port on their own time frame, and that need not at all block a release of the full suite. One of the advantages of accessing the core through the REST API is that it doesn't matter what clients like HyperKitty and Postorious are written in. >verified (I believe it imports the testing framework from mailman.client and >thus must be ported too). We've always said that mailman.client should be bilingual. It should be usable in either Python 2 or Python 3 scripts. I took a quick look at the current mailman.client trunk, and yes its test suite will have to be fixed, but I will take it upon myself to do that. If that's the only hangup, then I'm not worried. >And I thought the 3.0 release was almost there. I must say this is rather >discouraging. Hopefully the above provides some clarity and dis-disencouragement ;). Cheers, -Barry

2 1

Re: [Mailman-Developers] Python 3
by Andrew Stuart 26 Dec '14

26 Dec '14

Is there anything I can do to help with these? If there’s a task list maybe I can work out what I could do. >>This means I must now port HyperKitty and Kittystore to Python3, check that >>we don't use Py2-only libraries, etc. >>And mailman.client must be ported too, since it does import stuff from >>mailman for testing purposes. I think Postorious is safe, but that must be >>verified (I believe it imports the testing framework from mailman.client >>and thus must be ported too). On 27 Dec 2014, at 2:38 am, Aurelien Bompard <aurelien(a)bompard.org> wrote: > One of my top priorities has been to port Mailman 3 (core) to Python 3. > This > work is now complete, and ready to be merged into trunk. No doubt bugs > still > lurk, but at least the entire test suite is passing. Congrats ! * A few database columns have changed from LargeBinary to Unicode. I have > not > yet figured out how to add the Alembic schema migrations to handle this, > but > I'll do so before the release. > This should be pretty easy, something like the following command should work: alembic -c src/mailman/config/alembic.cfg revision --autogenerate -m "python3 port" [...] then I'll merge this to trunk and do a 3.0b5 release as a > Python 3-only application. Mailman 3 will not be bilingual so Python 2 > support will be dropped. > Wow. I am very surprised. So we went from "Python3 compatibility is not on the blocker list for Mailman 3.0" to "Mailman 3.0 will only work on Python 3". That's quite a change. This means I must now port HyperKitty and Kittystore to Python3, check that we don't use Py2-only libraries, etc. And mailman.client must be ported too, since it does import stuff from mailman for testing purposes. I think Postorious is safe, but that must be verified (I believe it imports the testing framework from mailman.client and thus must be ported too). And I thought the 3.0 release was almost there. I must say this is rather discouraging. Aurélien _______________________________________________ Mailman-Developers mailing list Mailman-Developers(a)python.org https://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: https://mail.python.org/mailman/options/mailman-developers/andrew.stuart%40… Security Policy: http://wiki.list.org/x/QIA9

2 1

Re: [Mailman-Developers] Error installing mailman.client due to "Aurélien"
by Florian Fuchs 16 Dec '14

16 Dec '14

Am 12.12.2014 um 00:00 schrieb Aurelien Bompard: > At first I thought I had introduced a regression.... but it's actually > just my name :-) > Don't blame me, I didn't choose it! That's a great example why French accents are great: First, in French you always know how a word is pronounced. And second, in Python they shame you into correctly decoding/encoding your I/O. :-) > (Florian, can you fix that? And by the way my last name is "Bompard", > without the "A") A fix is in the trunk. And, argh!, sorry I used your IRC nick as your last name! ;-) Florian

1 0

Error installing mailman.client due to "Aurélien"
by Andrew Stuart 11 Dec '14

11 Dec '14

Hello, Referring to transcript below, the word Aurélien seems to be causing a unicode error when setting up mailman client. When I cut it out, it works. thanks (venv2.7)ubuntu@server01:~$ bzr branch lp:mailman.client You have not informed bzr of your Launchpad ID, and you must do this to write to Launchpad or access private data. See "bzr help launchpad-login". Branched 58 revisions. (venv2.7)ubuntu@server01:~$ cd mailman.client/ (venv2.7)ubuntu@server01:~/mailman.client$ ls bin COPYING.LESSER distribute_setup.py Makefile MANIFEST.in README.txt setup.cfg setup_helpers.py setup.py src _static template.py (venv2.7)ubuntu@server01:~/mailman.client$ python setup.py develop Traceback (most recent call last): File "setup.py", line 40, in <module> 'src/mailmanclient/NEWS.txt'), File "/home/ubuntu/mailman.client/setup_helpers.py", line 138, in long_description if not value.endswith(NL): UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 162: ordinal not in range(128) (venv2.7)ubuntu@server01:~/mailman.client$ vi src/mailmanclient/NEWS.txt (venv2.7)ubuntu@server01:~/mailman.client$ cat src/mailmanclient/NEWS.txt ======================= NEWS for mailman.client ======================= 1.0.0b1 (xxxx-xx-xx) * Addedd an improved test harness using WebTest. Contributed by Aurélien Abompard. 1.0.0a1 (2014-03-15) ==================== * Initial release. (venv2.7)ubuntu@server01:~/mailman.client$ vi src/mailmanclient/NEWS.txt (venv2.7)ubuntu@server01:~/mailman.client$ cat src/mailmanclient/NEWS.txt ======================= NEWS for mailman.client ======================= 1.0.0b1 (xxxx-xx-xx) 1.0.0a1 (2014-03-15) ==================== * Initial release. (venv2.7)ubuntu@server01:~/mailman.client$

2 1

Will the next beta release have both Falcon for REST and SQLAlchemy?
by Andrew Stuart 11 Dec '14

11 Dec '14

thanks Andrew

2 1

Re: [Mailman-Developers] Mailman keeps creating "var" directories
by Andrew Stuart 09 Dec '14

09 Dec '14

Hi Abhilash, >>>>> Where did you get this config from? Mailman supports INI-style config only. Actually I copied the directory specifications from the output of mailman info -v I can see now that isn’t a good idea as the format is not the same. Don’t know why I assumed it would be. Also I can see now the second part of my issue was that even after setting the directories correctly, I also has to specify which section of mailman.cfg should be used to define the directories. It’s working now thanks. Andrew On 6 Dec 2014, at 6:31 pm, Abhilash Raj <raj.abhilash1(a)gmail.com> wrote: Hi Andrew, On Sat, Dec 6, 2014 at 12:04 PM, Andrew Stuart <andrew.stuart(a)supercoders.com.au> wrote: When I run “mailman info”, it creates a “var” directory in the current directory. Yes this is the default behavior when you don't specify the paths explicitly in your mailman.cfg You can see in the following that I have mailman.cfg in my /etc You can also see that /etc/mailman.cfg defines locations for the various directories. You can see that prior to running mailman info there is no “var” directory, but there is afterwards. I’m curious to understand why it is doing this when apparently mailman is picking up the configuration file OK from /etc but ignoring its contents and then setting up a new var directory in the current directory. Any ideas welcome. (venv2.7)ubuntu@server01:~$ ls blankontheme.zip mailman mailman.client postorius postorius_standalone temp venv2.7 (venv2.7)ubuntu@server01:~$ grep _dir /etc/mailman.cfg #var_dir: var archive_dir = /home/ubuntu/mailman/var/archives bin_dir = /home/ubuntu/venv2.7/bin data_dir = /home/ubuntu/mailman/var/data etc_dir = /etc ext_dir = /home/ubuntu/mailman/var/ext list_data_dir = /home/ubuntu/mailman/var/lists lock_dir = /home/ubuntu/mailman/var/locks log_dir = /home/ubuntu/mailman/var/logs messages_dir = /home/ubuntu/mailman/var/messages queue_dir = /home/ubuntu/mailman/var/queue template_dir = /home/ubuntu/mailman/var/templates var_dir = /home/ubuntu/mailman/var Where did you get this config from? Mailman supports INI-style config only. My bet is that mailman is able to find your config file but is not able to parse it. What actually happens is that mailman has predefined paths for different scenarios which can be configured with layout variable in [mailman] section inside your mailman.cfg. The default is `dev` layout which creates the `var` directory in the current directory. You can either change the default `var_dir` in `dev` layout, below is a sample config for that: ``` [paths.dev] var_dir : <dir_of_your_choice> ``` or you can use some other paths defined in src/mailman/config/mailman.cfg under [paths.local], [paths.master], [paths.fhs] and change the default layout in your mailman.cfg like this: ``` [mailman] layout: local ``` Also my guess is that you can create your own path structure inside your mailman.cfg and use that layout with mailman. How to create that can be seen in `src/mailman/config/mailman.cfg`. thanks, Abhilash Raj

3 2

Re: [Mailman-Developers] Mailman keeps creating "var" directories
by Barry Warsaw 09 Dec '14

09 Dec '14

On Dec 07, 2014, at 01:23 PM, Abhilash Raj wrote: >And coming to your problem, since mailman is still in development the default >layout is `dev` which helps developers as `var` directory is stored right in >front of them. Once it is packaged and released for production you can expect >some default whose behavior is different, and there are already many options >that I mentioned before. Exactly so. It turns out that during development, it's pretty useful for the default layout to be 'dev' and for the 'dev' layout to put the var directory in the current working directory. I fully expect that system packagers would create an /etc/mailman.cfg that sets the default layout to something else, e.g. fhs on Linux systems. Andrew, I think this is the source of the confusion, i.e. the fact that 'dev' is the default layout. >That may be because of the lack of explicit documentation for production. I >hope once that comes up it will be clearer for newbies and also for others >as well. Agreed. I would love to see some better organization to the documentation too. This would be a great opportunity for someone to contribute. It's pretty easy to learn reStructuredText and Sphinx. Cheers, -Barry

1 0