On 5 Oct, 06:45 pm, seadog@sealabs.net wrote:
Hello everyone, I'm writing a client that is using t.w.client.downloadPage function to download a large chunk of data into a file from a webserver. This is working fine.
Now I want use the same function (or at least take advantage of the "save to file" functionality) and be able to send along with my request a large chunk of data, without storing that chunk into memory.
To clarify my need here is an example
(1) client -> POST <large data file> -> server (2) server computes (3) server -> <REPLY with large data file> -> client
the (3) step is done in a memory efficient way using t.w.c.downloadPage(). Right now I execute the (1) step using the "postData" argument of t.w.c.downloadPage after loading all data into memory.
Is there a way I can avoid loading all data into memory?
Twisted 9.0 introduced twisted.web.client.Agent which is intended to eventually replace getPage, downloadPage, and all the other stuff in the twisted.web.client module. If you're lucky, then Agent already supports all the features you're interested in, depending on Twisted 9.0 is okay for you, and you can switch to using it right away. There are some helpers still missing, so you'll find it a little clunkier to use than downloadPage, but it will let you send your upload without loading it all into memory. Otherwise, you might have to move to using HTTPClientFactory or HTTPDownloader directly (the factories used by getPage and downloadPage) in order to feed it data incrementally. Avoid this if you can, those two factories are a mess. Jean-Paul