[Email-SIG] [RFC] Payload class

[Menno Smits]

Hi list,

This post follows on from something I brought here up almost 2 years 
ago: http://mail.python.org/pipermail/email-sig/2004-November/000181.html

I finally have time to look at this again and have some (hopefully) 
better ideas on how to accomplish custom email payload storage. Where I 
  work we need to be able to handle huge email messages which don't 
always fit in RAM. We are using some pretty awful hacks on the Python 
email libs to store payloads on disk instead of in memory.

The attached patch against 4.0a2 is rough sketch of a relatively clean 
way to solve the problem. It is rough and incomplete; I've posted it 
here to get some feedback before I head too far down the path of 
implementing a particular solution. A simple demo script is also included.

The Message class has been modified so it can handle payloads that are 
either a string (as now) or an instance of a new Payload class. A 
iter_payload() method has been added to Message to allow streaming out 
of payload data (regardless of the payload type underneath).

I've included 2 sample Payload classes. One is a simple memory store. 
The other caches payloads to temporary files on disk; the payload 
doesn't sit in RAM. Future payload classes could:

- use mixed memory/disk storage, storing only large payloads on disk so 
there's minimal I/O overhead for small payloads

- cache the decoded copy of a payload so that decoding is only done once 
if the decoded payload is required multiple times

- do crazy things like storing payloads across a network.

The possibilities are endless :)

More work is required on the parsing side. The FeedParser needs to 
accept an optional Payload factory class and generate payloads of that 
type as it parses. This should be an easy change.

The Generator class also needs to be modified. It should use the new 
iter_payload() method so that payloads are not loaded into RAM if the 
payload is stored in a memory efficient way.

These changes are backwards compatible with the existing email API.

Thoughts/questions/flames?

Regards,
Menno Smits
-------------- next part --------------
A non-text attachment was scrubbed...
Name: email-payloads.patch
Type: text/x-patch
Size: 6356 bytes
Desc: not available
Url : http://mail.python.org/pipermail/email-sig/attachments/20060714/3cd657c4/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: test_payload.py
Type: text/x-python
Size: 360 bytes
Desc: not available
Url : http://mail.python.org/pipermail/email-sig/attachments/20060714/3cd657c4/attachment.py