large file upload in web2 (was Re: http.HTTPFactory(site) equivalent in web2.0.2?)
Did you create a bug about it when you found it in 0.1? If not could you? Because I didn't know about any such issue, and I'm guessing neither did anyone else. Assign it to me and I'll look into it this evening after work (any code you have to reproduce such an issue would be helpful it doesn't necessarily have to be in the form of a unittest but if you can do that it'd be awesome.)
I just found it yesterday (in 0.1) and thought I should upgrade to 0.2 before bugging anyone about it. I'd be happy to create a bug, but I have no idea how to do that ("TICKET_CREATE privileges are required to perform this operation", and I haven't been able to get myself over the wall of "where are the instructions on getting an account or gaining those privileges?")
Here, though, is a simple example illustrating the problem:
#!/usr/bin/python
import os from twisted.internet import reactor from twisted.web2 import server, http, resource,\ channel from twisted.web2 import http_headers, responsecode from twisted.web2 import iweb
SAVEDIR = "/tmp" READSIZE=8192
class UploadFile(resource.PostableResource): def render(self, ctx): request = iweb.IRequest(ctx) filename = request.files['filename'][0][0] file = request.files['filename'][0][2]
dest = os.path.join(SAVEDIR, filename) destfile = open(dest, 'wb') tot = 0 while True: buf = file.read(READSIZE) tot = tot + READSIZE print "reading %d..." % tot if buf == '': break destfile.write(buf) destfile.close() file.close()
msg = "saved %s to %s" % (filename, dest) print msg return http.Response(stream="%s" % msg)
class Toplevel(resource.Resource): addSlash = True def render(self, ctx): return http.Response(responsecode.OK, {'content-type': http_headers.MimeType('text', 'html')}, "Hello")
child_uploadfile = UploadFile()
if __name__ == "__main__": site = server.Site(Toplevel()) reactor.listenTCP(1080, channel.HTTPFactory(site)) reactor.run()
Run this script, then upload a file to it with the client of your choice. Here is an html form which you can use to do that:
<html> <form action="http://localhost:1080/uploadfile" enctype="multipart/form-data" method="post"> filename: <input type="file" name="filename"> <input type="submit" value="submit"> </form> </html>
You should be able to upload files smaller than 9M without any problem, and files larger than 10M not at all.
I can send you an httplib client as well, but at least this should be enough to get you started.
I'll go try again, now, to figure out how to get this filed properly in Trac (and maybe even as a unittest!).
Thanks! Lenny
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Lenny G Arbage wrote:
I just found it yesterday (in 0.1) and thought I should upgrade to 0.2 before bugging anyone about it. I'd be happy to create a bug, but I have no idea how to do that ("TICKET_CREATE privileges are required to perform this operation", and I haven't been able to get myself over the wall of "where are the instructions on getting an account or gaining those privileges?")
There is a register link.
Here, though, is a simple example illustrating the problem:
<snip>
while True: buf = file.read(READSIZE) tot = tot + READSIZE print "reading %d..." % tot if buf == '': break destfile.write(buf)
<snip>
Ok, while True is bad and blocking. I think you should look at twisted.web2.static.FileSaver, if it doesn' perform a function close enough to what you want it is a good example of the right way to handle uploaded files as far as reading. (Hint it's to use things in the twisted.web2.stream library) I expect your problems will go away but you might still want to file a documentation enhancement bug stating that how to do this kind of thing is completely undocumented.
- -David
- -- "Usually the protocol is this: I appoint someone for a task, which they are not qualified to do. Then, they have to fight a bear if they don't want to do it." -- Glyph Lefkowitz
On Jun 27, 2006, at 11:52 AM, Lenny G Arbage wrote:
Did you create a bug about it when you found it in 0.1? If not could you? Because I didn't know about any such issue, and I'm guessing neither did anyone else. Assign it to me and I'll look into it this evening after work (any code you have to reproduce such an issue would be helpful it doesn't necessarily have to be in the form of a unittest but if you can do that it'd be awesome.)
I just found it yesterday (in 0.1) and thought I should upgrade to 0.2 before bugging anyone about it.
The failing with > 10MB is intentional, but the hanging is not. It's supposed to return an error response to the client. In fileupload.py: 233, parseMultipartFormData has arguments which limit the number of fields and the size of the uploaded data. The default limit for size is 10MB.
Unfortunately, it looks like there is no way to pass in different limits to fileupload.parseMultipartFormData from where it's called in server.parsePOSTData.
So there's two bugs here: 1) hangs instead of returning a response. 2) limit is not configurable.
James
I'd be happy to create a bug, but I have no idea how to do that ("TICKET_CREATE privileges are required to perform this operation", and I haven't been able to get myself over the wall of "where are the instructions on getting an account or gaining those privileges?")
Ticket creation is disabled for anonymous visitors, but enabled for all registered accounts: just use the "Register" account on the upper right.
After you do that, you may want to click on the "User Configuration" link nearby, and insert a valid email. Then, by using your login name in the Cc: ticket field, you will receive email at each ticket change.
For anonymous visitors, the "New Ticket" link is not shown; you possibly got the above error by directly going to the URL:
http://twistedmatrix.com/trac/newticket
while not logged in.
participants (4)
-
David Reid -
James Y Knight -
Lenny G Arbage -
Nicola Larosa