sending a file chunk by chunk instead as a whole to a web server

[Ryan Kelly]

On Tue, 2010-08-03 at 10:45 +0530, Kushal Kumaran wrote:
> On Mon, Aug 2, 2010 at 12:22 PM, Sanjeeb <sanjeeb25 at gmail.com> wrote:
> > Hi,
> > I have a web client which send a file to a server as multipart form
> > data, the sending of data is from
> > http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/.
> >
> > I dont want to open the whole file to memory(at cliend end) and then
> > send, i just want to send part by part, say chunk of 1024 bytes to the
> > server and then assemble at the server end.
> >
> > Could some one suggest what would be the best way to do this?
> >
> 
> There's no reason why sending the whole file implies reading the whole
> file into memory at one time.  You can just read your desired chunk
> size and send it, then read the next chunk, and so on.  You might have
> to first find the total size to calculate what to set Content-Length
> to.

More concretely, you would restructure the encode_multipart_formdata()
function as a generator, yielding chunks of data to send one at a time.
Something like this:

    def post_multipart(host,selector,fields,files):
        ...snip...
        h.endheaders()
        for chunk in encode_multipart_formdata_chunks(fields,files):
            h.send(chunk)
        errcode, errmsg, headers = h.getreply()
        return h.file.read()


    def encode_multipart_formdata(fields,files):
        BOUNDARY = '----------ThIs_Is_tHe_bouNdaRY_$'
        CRLF = '\r\n'
        for (key, value) in fields:
            yield '--' + BOUNDARY
            yield 'Content-Disposition: form-data; name="%s"' % key
            yield ''
            yield value
        for (key, filename, value) in files:
            yield '--' + BOUNDARY
            yield 'Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename)
            ...etc...
            ...etc...

There are many improvements to make, but this should get you started.
For example, you'll need to calculate the total content-length rather
than just calling len(body) to obtain it.  That's left as an exercise to
the reader.


    Ryan


-- 
Ryan Kelly
http://www.rfk.id.au  |  This message is digitally signed. Please visit
ryan at rfk.id.au        |  http://www.rfk.id.au/ramblings/gpg/ for details

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://mail.python.org/pipermail/python-list/attachments/20100803/e7d192f6/attachment.sig>