Problems uploading data in web2
Hi I'm trying to use web2 to stream uploading of data, but I ran into some problems. First I tried building from the example in http://twistedmatrix.com/projects/web2/documentation/examples/demo.html, however args and files objects in the request did not show anything. However the length of stream was correct, and using tcpdump, showed the correct value. So I tried building my own uploader which look like this: class Uploader(resource.PostableResource): def http_POST(self, ctx): request = iweb.IRequest(ctx) self.parse_stream(request.stream) @defer.deferredGenerator def parse_stream(self, stream): from twisted.web2 import fileupload s = fileupload.parse_urlencoded_stream(stream) while True: datas = s.read() datas = fileupload.wait(datas) yield datas datas = datas.getResult() Using the 0.1.0 release of web2 didn't get past the yield line. Using latest from subversion (revision 15836), got past, but datas was None, even though data had been send. I've used both cURL (using the -d switch) and wget (using the --post-file switch) for uploading data, so I do not think the client as malfunctioning. What am I doing wrong? I'm aware that web2 is under development and all that, but I need streaming upload, so the web module is not an option. -- - Henrik
On Feb 6, 2006, at 5:52 AM, Henrik Thostrup Jensen wrote:
Hi
I'm trying to use web2 to stream uploading of data, but I ran into some problems. First I tried building from the example in http:// twistedmatrix.com/projects/web2/documentation/examples/demo.html, however args and files objects in the request did not show anything. However the length of stream was correct, and using tcpdump, showed the correct value. So I tried building my own uploader which look like this:
What are you actually trying to do? There's two different kinds of "streaming upload". The simpler is just a raw data stream. If you just want raw data, forget POST and associated form processing, and just use request.stream.read() in with a PUT method. That will work with curl -T. However, if you want it to be the result of a form post from a web browser, you'll need the MIME multipart support. The default POST support (with PostableResource) parses the multipart stream, and places all the results in memory or temporary files (depending on size), in the args & files dicts. It parses the data incrementally, but doesn't give control to your "render" method until all the data has been uploaded. If all you're looking for from a streaming upload is to not buffer everything in memory, this will do it for you. If that's not good enough for your application, if you really need access to the data as it's being uploaded, you'll need to use the lower level machinery in fileupload: the MultipartMimeStream class. You give this a raw data stream (request.stream), and the multipart boundary from the content-type, then call its read() method repeatedly to get back tuples of (fieldname, filename, ctype, dataStream). Then you use dataStream.read() to get back data blocks and can do whatever you like with them. I can help explain any of the alternatives in more detail if you let me know which you're really interested in. James
Hi James, thanks for your answer. On 2/6/06, James Y Knight <foom@fuhm.net> wrote:
On Feb 6, 2006, at 5:52 AM, Henrik Thostrup Jensen wrote:
I'm trying to use web2 to stream uploading of data, but I ran into some problems. First I tried building from the example in http://twistedmatrix.com/projects/web2/documentation/examples/demo.html, however args and files objects in the request did not show anything. However the length of stream was correct, and using tcpdump, showed the correct value. So I tried building my own uploader which look like this:
What are you actually trying to do? There's two different kinds of "streaming upload". The simpler is just a raw data stream. If you just want raw data, forget POST and associated form processing, and just use request.stream.read() in with a PUT method. That will work with curl -T.
This is what i want - raw data as I'm uploading files. I hacked up a small example and got it to work. I also tried POST for the fun of it, and that seems to work as well. Not sure what I got wrong before. Oh, and deferredGenerator is darned clever :-). I can help explain any of the alternatives in more detail if you let me know
which you're really interested in.
I think I got whats needed for now. Thanks for your time. -- - Henrik
participants (2)
-
Henrik Thostrup Jensen -
James Y Knight