I'm going to split this into three separate threads with appropriate titles. This is #1.
Regarding optimization for constrained environments, I wrote:
But I'm pretty sure people have run Mailman 3 on a Raspberry Pi. How constrained an environment are you aiming for?
Justus Winter writes:
I had problems on my shared hoster that provided 1 gigabyte of RAM per user (I'm not a 100% on how they measure that).
OK, yes, my estimate says that's going to be too tight. I'm seeing about 80MB per runner with a full complement of processes without any slicing being 18. Some of that is shared (IIRC about 5MB/runner according to top), but that only buys back 1 runner's worth. There are several somewhat optional processes (the nntp, archive, command, and 2 WSGI processes for Mailman Web) but that's probably still not quite going to fit into 1GB.
Re polling queues:
the runners are polling their queues in loops. My installations that hardly see any traffic at all are all doing: do I have work, no, sleep 1, do I have work, no, sleep 1... I can see that this will amortize in big installations, but for small ones this is quite sad.
I guess, but if it doesn't show in the load average, I'm not sure why one should care. I don't know about your installation, but Mailman consumes less than 1% of CPU when idle as far as I can tell. For me to support a change here, either you'd need to show a non-negligible improvement or it would have to be "free" (see below).
- Improve latency of messages
What latency are you observing?
And even for big installations, or if we say that efficiency is not important, if a mail goes through the hands of three queue runners, the worst-case latency is three seconds in an otherwise idle installation! We can definitively improve upon that.
Who will notice? Is there anybody who cares about a 3s latency in list email? If there is, that would be a user-visible improvement to set against any increase in code complexity.
The key insight here is that emails in queues don't appear out of thin air, another runner is putting them there. If each runner that goes to sleep does so by waiting on a condition variable associated with its queue, and every runner that deposits a mail into the queue signals the sleeping runners, that latency goes away while at the same time improving efficiency by no longer having to poll the queue every second.
Thing is, email (and Mailman specifically) operates on a store-and- forward model. The queue file *must* be present for a runner to do its work, and conversely, if a file is present the runner has work to do. Polling is a little ugly, but it's a perfect fit for the problem semantically, and very simple to explain and implement.
If the condition-variable-based code is equally simple, equally reliable, and identical across our supported platforms, sure, that's worth looking at because we get your developer-visible efficiency enhancements for "free". But if any of those requirements fails, I would want to see an improvement in user-visible performance.
I will take a look at the work you mention, but it will be a couple of weeks at least before I have useful comments.
Still need some time for this, but I wanted to get some stuff out of my inbox. :-)