Re: [Mailman-Developers] Why both a .msg and .db in qfiles?
On Wed, 24 Jul 2002 16:20:49 -0600 Jason R Mastaler <Jason> wrote:
I'm curious why MM doesn't just store the rfc822 headers and message body in the .db pickle/marshal as well? Why maintain two files instead of just one?
One advantage of the flat text message file is that it can be edited by external tools (in my case XEmacs and various odd scripts) prior to being processed and released for broadcast by the moderation interface.
<<Yeah, I hand edit every post>>
The reason I ask is because I'm contemplating a similar approach for TMDA.
Mailman v2.1 offers a choice between the above approach and a pickle.
--
J C Lawrence
---------(*) Satan, oscillate my metallic sonatas.
claw@kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
J C Lawrence <claw@kanga.nu> writes:
Mailman v2.1 offers a choice between the above approach and a pickle.
Where is this configured in MM 2.1?
-- (http://tmda.net/)
"JRM" == Jason R Mastaler <jason@mastaler.com> writes:
>> Mailman v2.1 offers a choice between the above approach and a
>> pickle.
JRM> Where is this configured in MM 2.1?See SAVE_MSGS_AS_PICKLES in Mailman/Queue/Switchboard.py for whether re-queued messages get stored as a pickle or as plaintext.
See METADATA_FORMAT in Defaults.py/mm_cfg.py for the storage format for the metadata dictionary (I was mistaken that it's always a pickle, in fact it's usually a marshal, and METADATA_FORMAT=METAFMT_MARSHAL is probably the only one that works, see SF patch #567288).
-Barry
barry@python.org (Barry A. Warsaw) writes:
See METADATA_FORMAT in Defaults.py/mm_cfg.py for the storage format for the metadata dictionary (I was mistaken that it's always a pickle, in fact it's usually a marshal
And why use a marshal instead of a pickle? Better performance?
-- (http://tmda.net/)
"JRM" == Jason R Mastaler <jason@mastaler.com> writes:
>> See METADATA_FORMAT in Defaults.py/mm_cfg.py for the storage
>> format for the metadata dictionary (I was mistaken that it's
>> always a pickle, in fact it's usually a marshal
JRM> And why use a marshal instead of a pickle? Better
JRM> performance?For something as simple as a dictionary (which the metadata is), I believe so. I did some benchmarking quite a while back, but don't remember the details. Note though that marshals are easier to break in odd ways -- they're basically a tool too support .pycs so were never terribly robust. Pickles generally better for object trees too, especially if you might have cycles. OTOH, binary fast pickles for simple data structures might perform equally well.
-Barry
On Thu, 25 Jul 2002 14:57:01 -0400 Barry A Warsaw <barry@python.org> wrote:
For something as simple as a dictionary (which the metadata is), I believe so. I did some benchmarking quite a while back, but don't remember the details. Note though that marshals are easier to break in odd ways -- they're basically a tool too support .pycs so were never terribly robust. Pickles generally better for object trees too, especially if you might have cycles. OTOH, binary fast pickles for simple data structures might perform equally well.
Points briefly at Jelly and Banana:
http://twistedmatrix.com/products/spread
I've been pleased with the performance curves of both in early testing here.
--
J C Lawrence
---------(*) Satan, oscillate my metallic sonatas.
claw@kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
"JCL" == J C Lawrence <claw@kanga.nu> writes:
>> For something as simple as a dictionary (which the metadata
>> is), I believe so. I did some benchmarking quite a while back,
>> but don't remember the details. Note though that marshals are
>> easier to break in odd ways -- they're basically a tool too
>> support .pycs so were never terribly robust. Pickles generally
>> better for object trees too, especially if you might have
>> cycles. OTOH, binary fast pickles for simple data structures
>> might perform equally well.
JCL> Points briefly at Jelly and Banana:
JCL> http://twistedmatrix.com/products/spread
JCL> I've been pleased with the performance curves of both in
JCL> early testing here.Interesting stuff. I've known about Twisted for a while but hadn't realized they had those serialization protocols. I can see how that'd be useful for some distributed computing applications, especially when you're accepting hostile data.
In Mailman's case, I think pickles and marshals are probably fine. They have the advantage that they're built-in tools, and we don't have to worry about hostile data (in this context).
But thanks for the reference. I've had conversations with the Twisted guys at previous IPCs about Mailman and SMTP plugins. -Barry
On Thu, 25 Jul 2002 15:22:14 -0400 Barry A Warsaw <barry@python.org> wrote:
Interesting stuff. I've known about Twisted for a while but hadn't realized they had those serialization protocols. I can see how that'd be useful for some distributed computing applications, especially when you're accepting hostile data.
I have a little attention on down the road when we start exposing and distributing the process queue model more. The opportunities for hostile data insertion start growing as the system becomes more loosely coupled.
But thanks for the reference. I've had conversations with the Twisted guys at previous IPCs about Mailman and SMTP plugins.
Welcome.
--
J C Lawrence
---------(*) Satan, oscillate my metallic sonatas.
claw@kanga.nu He lived as a devil, eh?
http://www.kanga.nu/~claw/ Evil is a name of a foeman, as I live.
"JCL" == J C Lawrence <claw@kanga.nu> writes:
>> Interesting stuff. I've known about Twisted for a while but
>> hadn't realized they had those serialization protocols. I can
>> see how that'd be useful for some distributed computing
>> applications, especially when you're accepting hostile data.
JCL> I have a little attention on down the road when we start
JCL> exposing and distributing the process queue model more. The
JCL> opportunities for hostile data insertion start growing as the
JCL> system becomes more loosely coupled.Cool. -Barry
participants (3)
-
barry＠python.org -
J C Lawrence -
Jason R. Mastaler