-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Feb 21, 2008, at 10:48 AM, Mark Sapiro wrote:

The underlying cause may not be the same, but the problem is precipitated by a message in the list's lists/listname/digest.mbox file.

Just FTR, I really want to move MM3's digest mailbox to maildir format
instead of mbox. MM3 will require Python 2.5, so we can use it's
really great updated mailbox module to handle problems like this.
That's the plan anyway.

• -Barry

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAke9shwACgkQ2YZpQepbvXE64wCeJ5g98PTgMAJ+J5ipiU9DzOk+ ncsAnRJ5hG886mTjIpT6rcKyzRqfKRp5 =yZPy -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Feb 21, 2008, at 10:42 PM, Brad Knowles wrote:

On 2/21/08, Barry Warsaw wrote:

...

Just FTR, I really want to move MM3's digest mailbox to maildir
format instead of mbox. MM3 will require Python 2.5, so we can use it's really great updated mailbox module to handle problems like this. That's the plan anyway.

Can we at least use a hashed mail directory solution that doesn't
have massive scalability problems?

For the digests it probably doesn't matter because they'll never get
that big. I'm still planning on making this change for the queue
directories (though I haven't yet).

• -Barry

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAke+0koACgkQ2YZpQepbvXGeRgCfanXNjb4NYNzVnK7i7FM2TZpM NIMAnjcwkm0ElMle7RmHP65n7sIcqetD =Ndfd -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

I set the Reply-To to mailman-developers, since that's where we should
move this discussion to.

On Feb 22, 2008, at 11:14 PM, Brad Knowles wrote:

On 2/22/08, Barry Warsaw wrote:

...

...

Can we at least use a hashed mail directory solution that doesn't have massive scalability problems?

For the digests it probably doesn't matter because they'll never get that big. I'm still planning on making this change for the queue directories (though I haven't yet).

I thought we were talking about replacing the 7th edition mbox file
format for the raw or the similar mbox-like format for the cooked
archives. Using some sort of a hashed directory structure would
allow a lot more flexibility in terms of going in and deleting or
editing messages in the archive, along with many other benefits it
might bring.

However, if you imagine python-list with hundreds of thousands of
messages in the archive, there's just no way that could possibly
scale with Maildir, Maildir+, or any other solution that does not
enforce a good hashed directory scheme that is kept invisible to the
higher-level applications.

The archives is an entirely different subsystem, and without a
volunteer stepping forward (for which I've been practically begging
for years ;), I don't see us being able to do any architectural
changes to Pipermail, at least until after MM3 comes out, if ever.
That's the unfortunate reality, but I do have thoughts about our
archiver situation, which I'll outline at some future date.

Actually, what I was talking about was the temporary digest mailbox.
When Mailman receives messages destined for the list's digest, it
holds these messages in a mailbox, currently mbox format, because
that's all that older Python's supported. Then when Mailman is
building the digest, it reads messages from this mbox, and chucks it
when its done. So they never get that big, or live for that long.

Still, mbox presents its own problems, such as what Mark outlined.
One corrupted message, or a broken mbox delimiter horks the whole
digest. I think we can do better with maildir and for the digest
application, we don't have to worry about scalability.

• -Barry

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin)

iEYEARECAAYFAkfAILwACgkQ2YZpQepbvXGm2ACggEuyXvTC7iV/32tK7y435VyL lNsAn2UPZg9QNKZUkWwZHDzBmkPYbz02 =L7kZ -----END PGP SIGNATURE-----