[Email-SIG] use_case.py

[R. David Murray]

On Mon, 25 May 2015 21:33:21 -0000, PyTonic <pytonic at i2pmail.org> wrote:
> On 05/25/2015 07:04 PM, PyTonic wrote:
> > I will post another message containing a simple use case for the new
> > interface which only streams, decodes and stores base64 encoded
> > payloads on the fly and uses the old method for everything else. It
> > additionally uses two more callables inside its Message subclass:
> > get_payload_file() and is_streamed().
> >
> > It also contains some comments about unresolved issues like how
> > decoding errors should be properly dealt with. And who is responsible
> > for catching exceptions raised by the new interfaces so they can't
> > break the FeedParser itself.

It is great that you are interested in working on this.  Providing a way
to process large emails without the current crazy memory consumption is
a goal of mine, and we'll happily work with you toward making that a
reality.  However, anything along these lines is going to be a new
feature, and therefore can only target 3.6 at this point, so any patch
proposals need to be against the default branch of the cpython
repository.

In Python3 we now have the policy framework.  I'm pretty sure it makes
sense to leverage that for the new internal API.  I agree that
feedparser itself will need some changes in order to make all this work
correctly.  (Also, feedparser has gotten a couple of non-trivial
performance enhancements in python3, so some of the code is different.)

Note that the generator needs similar changes, and that problem may be
much harder to solve, since the current algorithm is recursive and holds
*everything* in memory.  In addition, it seems like this would be a
natural (even necessary?) place to introduce the 'store bodies on disk'
interface that we've wanted for quite a while.

Can you take a look at the policy framework and reformulate your proposal
in light of that?  We can certainly work on this one piece at a time,
we just want to keep all of the moving parts in mind while we do so...

--David