Mailman 3 December 2014 - Mailman-Developers

Re: [Mailman-Developers] Mailman keeps creating "var" directories
by Andrew Stuart 07 Dec '14

07 Dec '14

Hi Abhilash, I very much like the capability of mailman to create its own “var” directory structure - most open source systems require me to spend hours with strace trying to work out what directories and permission an application needs in what location - I’m heartily sick of having my time wasted like that so this is very refreshing functionality. On the other hand it’s very confusing when a misconfiguration in the mailman.cfg results in a “var” directory being created in the current directory **which then gets picked up and used as the configuration**, so correcting your configuration in /etc/mailman.cfg then leads to a new problem, which is that your /etc/mailman.cfg is no longer being used *at all*. I love the creation of the var directories by mailman but it feels like a bit of a hand grenade with a short fuse. Whilst I was setting up my configuration I ended up with var directories all over the file system, without any understanding of why they were there. A trap for Mailman newbies. Perhaps it’s worth considering making creation of the “var” directory structure could be the result of an explicit option rather than a default action. thanks On 6 Dec 2014, at 6:31 pm, Abhilash Raj <raj.abhilash1(a)gmail.com> wrote: Hi Andrew, On Sat, Dec 6, 2014 at 12:04 PM, Andrew Stuart <andrew.stuart(a)supercoders.com.au> wrote: When I run “mailman info”, it creates a “var” directory in the current directory. Yes this is the default behavior when you don't specify the paths explicitly in your mailman.cfg You can see in the following that I have mailman.cfg in my /etc You can also see that /etc/mailman.cfg defines locations for the various directories. You can see that prior to running mailman info there is no “var” directory, but there is afterwards. I’m curious to understand why it is doing this when apparently mailman is picking up the configuration file OK from /etc but ignoring its contents and then setting up a new var directory in the current directory. Any ideas welcome. (venv2.7)ubuntu@server01:~$ ls blankontheme.zip mailman mailman.client postorius postorius_standalone temp venv2.7 (venv2.7)ubuntu@server01:~$ grep _dir /etc/mailman.cfg #var_dir: var archive_dir = /home/ubuntu/mailman/var/archives bin_dir = /home/ubuntu/venv2.7/bin data_dir = /home/ubuntu/mailman/var/data etc_dir = /etc ext_dir = /home/ubuntu/mailman/var/ext list_data_dir = /home/ubuntu/mailman/var/lists lock_dir = /home/ubuntu/mailman/var/locks log_dir = /home/ubuntu/mailman/var/logs messages_dir = /home/ubuntu/mailman/var/messages queue_dir = /home/ubuntu/mailman/var/queue template_dir = /home/ubuntu/mailman/var/templates var_dir = /home/ubuntu/mailman/var Where did you get this config from? Mailman supports INI-style config only. My bet is that mailman is able to find your config file but is not able to parse it. What actually happens is that mailman has predefined paths for different scenarios which can be configured with layout variable in [mailman] section inside your mailman.cfg. The default is `dev` layout which creates the `var` directory in the current directory. You can either change the default `var_dir` in `dev` layout, below is a sample config for that: ``` [paths.dev] var_dir : <dir_of_your_choice> ``` or you can use some other paths defined in src/mailman/config/mailman.cfg under [paths.local], [paths.master], [paths.fhs] and change the default layout in your mailman.cfg like this: ``` [mailman] layout: local ``` Also my guess is that you can create your own path structure inside your mailman.cfg and use that layout with mailman. How to create that can be seen in `src/mailman/config/mailman.cfg`. thanks, Abhilash Raj

2 1

Mailman keeps creating "var" directories
by Andrew Stuart 06 Dec '14

06 Dec '14

If I run the mailman command it keeps creating “var” directories in whatever the current directory is. Is that intentional? thanks

3 3

Using alembic autogenerate
by Adam Clark 05 Dec '14

05 Dec '14

Is there a procedure for autogenerating a database update using alembic? I haven't used alembic before, but through trial and error I arrived at this: > alembic -c mailman/config/alembic.cfg revision --autogenerate This fails with File "/workspace/test/mailman/libs/mailman/src/mailman/config/config.py", line 98, in __getattr__ return getattr(self._config, name) AttributeError: 'NoneType' object has no attribute 'database' So apparently mailman config wasn't being loaded. I added it to mailman/database/alembic/env.py: def run_migrations_online(): config.load() # Added url = expand(config.database.url, config.paths) ... This succeeded in producing a version file, but that version would delete everything: def upgrade(): ### commands auto generated by Alembic - please adjust! ### op.drop_table('message') op.drop_table('address') op.drop_table('autoresponserecord') op.drop_table('member') op.drop_table('ban') ... The root cause appeared to be that Model.metadata.sorted_tables returned an empty list. Manually triggering model reflection seemed to improve things; again in env.py: def run_migrations_online(): config.load() # Added url = expand(config.database.url, config.paths) engine = create_engine(url) Model.metadata.reflect(bind=engine) # Added connection = engine.connect() ... This now produces a correctly empty update against the existing code; the problem is if I change the code (and I verified that Model.metadata.sorted_tables reflects the changed model information) this *still* produces an empty update. I suspect that I have this all backwards, that this whole process is actually intended to run from within mailman rather than via the alembic command line, but I can't find where this would be. Thanks, Adam

1 0

Is cron used in Mailman 3?
by Andrew Stuart 03 Dec '14

03 Dec '14

Where can I find the documentation that explains the required configuration for cron in Mailman 3? thanks

1 0

Where can I find the Mailman 3 REST API specification?
by Andrew Stuart 03 Dec '14

03 Dec '14

is it documented anywhere? I’m happy to dig through code or tests as long as someone can point me in the direction of a definitive list of REST API methods. thanks

2 1

Re: [Mailman-Developers] About Mailman's unicode-enabled Message subclass
by Barry Warsaw 02 Dec '14

02 Dec '14

On Dec 02, 2014, at 11:26 AM, Aurelien Bompard wrote: >I kinda think so too, so I tried removing the conversion to unicode in get() >and __getitem__(): I have about 20 failures in the whole test suite, and a >couple UnicodeWarnings from SQLAlchemy. I'm going to try and fix these >failures and the code that expects unicode headers, and send you a MP How >about that? +1; I look forward to seeing it. I suspect those will be useful also with my experimental py3 branch. :) Cheers, -Barry

2 1

Re: [Mailman-Developers] About Mailman's unicode-enabled Message subclass
by Mark Sapiro 02 Dec '14

02 Dec '14

On 12/01/2014 08:45 AM, Aurelien Bompard wrote: > > I'm really interested in any insight on this issue. Thanks for reading all > that :-) I just took a quick look, but I think the problem is likely that mailman.email.message.Message overrides email.message.Message.__getitem__() as follows: def __getitem__(self, key): # Ensure that header values are unicodes. value = email.message.Message.__getitem__(self, key) if isinstance(value, str): return unicode(value, 'ascii') return value If we trace back a bit in the code, email.message.get_param(...) does for k, v in self._get_params_preserve(failobj, header): and _get_params_preserve(...) does value = self.get(header, missing) which because of our override decodes the entire "attachment; filename*=UTF-8''d%C3%A9jeuner.txt" string as ascii into u"attachment; filename*=UTF-8''d%C3%A9jeuner.txt" which is then parsed into the tuple (u'UTF-8', u'', u'd\xc3\xa9jeuner.txt'). It seems to me that any way to fix this is a horrible kludge. In particular, the value portion of the tuple has already been garbled, and I don't see a good way to fix it once it's been returned. I've looked, again briefly because after a short time my head spun out of control, at trying to fix it by overriding additional methods like _get_params_preserve(), but I didn't get far. -- Mark Sapiro <mark(a)msapiro.net> The highway is for gamblers, San Francisco Bay Area, California better use your sense - B. Dylan

2 1

Re: [Mailman-Developers] About Mailman's unicode-enabled Message subclass
by Barry Warsaw 01 Dec '14

01 Dec '14

On Dec 01, 2014, at 05:45 PM, Aurelien Bompard wrote: >I'm really interested in any insight on this issue. Thanks for reading all >that :-) To be honest, mailman.email.message.Message should really go away. It's a terrible, and old, hack. A few things *are* useful, such as pickle support (required for the qfiles to work), and the convenience properties .sender and .senders. Of course, the convenience subclasses (e.g. UserNotification and OwnerNotification) are helpful too, but in a different way. This is yet another reason why I'd like to get the core on Python 3. Then we can ditch most of this hack of a subclass and rely on the behavior of the Python 3 email package, which should DTRT. Not sure that's helpful for your immediate problem though. ;) Cheers, -Barry

1 0

About Mailman's unicode-enabled Message subclass
by Stephen J. Turnbull 01 Dec '14

01 Dec '14

Aurelien Bompard writes: > I'm really interested in any insight on this issue. Thanks for reading all > that :-) RFC 2047 and RFC 2231 should be treated as wire protocol, and handled at the bytes level. I haven't looked at this in a while, but I'm pretty sure Python 3's email module does this correctly. I would say that the routines that do this decoding should return bytes, and there should be called by the get() and __getitem__() methods. I don't have time to take a close look at the code until the weekend at least. Steve

1 0

About Mailman's unicode-enabled Message subclass
by Aurelien Bompard 01 Dec '14

01 Dec '14

Hi * ! I'm trying to fix bug 1060951 (that I reported), and no matter how I poke at it, I can't find the proper way to solve it, so I'd like to share my thoughts with you. https://bugs.launchpad.net/mailman/+bug/1060951 The issue is with our unicode-enabled subclass of Python's email.message.Message class. Our subclass converts headers to unicode in the get() and __getitem__() methods. However, when an email has an attachment, one of the ways of representing the filename is to urlencode it (RFC 2231). The urldecoding is thus done on an unicode string, which is then passed to email.utils.collapse_rfc2231_value and decoded to unicode again. Of course, this fails, an unicode string can't be decoded twice. I tried re-implementing more of the original Message class to avoid this second decoding, but then I get another problem: the filename has already been urldecoded. Here's an example to make it clearer: >>> m1 = email.message.Message() >>> m1["content-disposition"] = "attachment; filename*=UTF-8''d%C3%A9jeuner.txt" >>> m1.get_param("filename", header="content-disposition") ('UTF-8', '', 'd\xc3\xa9jeuner.txt') If I use mailman's Message implementation, I get: >>> m2 = mailman.email.message.Message() >>> m2["content-disposition"] = "attachment; filename*=UTF-8''d%C3%A9jeuner.txt" >>> m2.get_param("filename", header="content-disposition") (u'UTF-8', u'', u'd\xc3\xa9jeuner.txt') The only difference is that the resulting tuple contains unicode strings. Now when I try to pass this result to email.utils.collapse_rfc2231_value, like Message.get_filename() does: >>> p1 = m1.get_param("filename", header="content-disposition") >>> email.utils.collapse_rfc2231_value(p1) u'd\xe9jeuner.txt' >>> p2 = m2.get_param("filename", header="content-disposition") >>> email.utils.collapse_rfc2231_value(p2) TypeError: decoding Unicode is not supported And if I suppress the final decoding from what's already a unicode string, I get u'd\xc3\xa9jeuner.txt' And, as you can see, the original u'd\xe9jeuner.txt' is not the same as u'd\xc3\xa9jeuner.txt' (the second one is encoded twice). In the end, our subclass of Message can't extract non-ascii filenames. The bug 1060951 contains a testcase for this, it's a replacement for the src/mailman/email/tests/test_message.py : https://bugs.launchpad.net/mailman/+bug/1060951 I'm really interested in any insight on this issue. Thanks for reading all that :-) Aurélien

1 0