[Web-SIG] WSGI -- usable for other protocols?

[Jean-Paul Calderone]

On Tue, 17 Oct 2006 18:19:10 PDT, Bill Janssen <janssen at parc.com> wrote:
>> might I suggest you contribute
>> to a project which sounds roughly equivalent to the one you're describing?
>>
>> http://divmod.org/trac/wiki/DivmodQuotient
>
>Just for fun, I grepped the sources for IMAP.  No hits.

Quite so.  We're currently most of the way through an (unfortunate) rewrite
to fix some database-related problems.  IMAP4 hasn't been high on the
port-list, so there's no IMAP4 code in the new codebase yet.  However,
Twisted's IMAP4 protocol implementation was developed for this project,
and IMAP4 is on our mind as we implement things, so adding it isn't going
to be obstructed by anything in Quotient (I would say "easy" but nothing
related to IMAP4 is easy).

>
>Seems like I'd spend more time understanding the framework system
>you're using than it would take me to write it from scratch.

Ahhh, I doubt it.  This isn't to say you wouldn't spend a while
understanding the framework, but writing it from scratch would take
longer.

>An IMAP server isn't hard.

Having spent fair chunks of the last several years implementing various
IMAP4 servers, I must disagree. :)  Unless you're happy with a
semi-protocol spec, semi-broken server that doesn't scale to a decent
number of messages, it's quite a haul.

>And I don't think the project is all that equivalent.

Strictly speaking, an IMAP4 server will be a subset of Quotient, and IMAP4
is by no means the main focus of Quotient, so maybe equivalent wasn't the
right word.

>
>Does Twisted support the use of PyLucene?

Quotient's using PyLucene for fulltext indexing already.  So... yes :)

>
>I basically want an IMAP server that supports the MH mail storage
>format, uses Lucene for indexing and search, and has the ability to do
>auto-filtering on a per-user basis with either MH procmail scripts or
>a Python script that uses a particular API.  I don't need an SMTP
>server, I don't need a Web interface to mail.

It's possible you'd be happier basing the IMAP4 server on Twisted's
protocol support, rather than starting from Quotient (although _I'd_ be
happier if you added IMAP4 support to Quotient ;).

Quotient uses a SQLite database for storage of structured data about
messages and a filesystem structure (currently not a great structure,
but it's fixable) for actual message files.

It supports per-user filtering rules (although not procmail based - and
the work done in this area so far is extremely minimal, basically it can
do substring matching on headers - expanding this would be pretty simple
though, Quotient is designed for this kind of thing).

The SMTP server can be turned off completely, although then you need
another mechanism for adding messages to the system (inotify + directory
would work, but you'll have to write that part).  For users, the web
interface is optional too, but various admin tasks may continue to require
some web interaction.

>
>If DivmodQuotient is anywhere close to that, I'll take a longer look.

It sounds like you might be happier starting from Twisted's IMAP4 code
rather than doing the work in Quotient, unless your requirements are
somewhat more flexible than I have gotten the impression that they are.

I _certainly_ would not recommend doing the protocol implementation from
scratch.  Using Twisted there at least is a complete win.

As for PyLucene in that scenario, there's no _direct_ support for gcj
threads in Twisted.  All of my work with PyLucene in Quotient has been
in the main thread in a child process of the main process (desirable
to avoid segfaulting the main server, mainly).

Note (in case it isn't obvious yet) I'm a developer on both Quotient and
Twisted, and I wrote pretty much all of Twisted's IMAP4 code.  It's possible
I'm slightly biased.  That said, lots of people have told me Twisted's IMAP4
implementation is the best they've worked with, that it saved their thesis,
project, company, life, etc. ;)

Jean-Paul