twisted.web : limiting file size during upload (using python)
Hey I have a script to be able to upload logs to a twisted web server. I have the following requirements: 1) to be able to limit the size of the upload to 10MB. 2) to NOT store file in memory during the request. Does twisted.web provide these functionality? Any inputs on these would be welcome! Here is the *server part* os the script: try: oStream = open(filename, 'wb') oStream.write(request.args['fname'][0]) outputStream.close() except: # handle exception a test *client script:* <!DOCTYPE html> <html> <form action="https://serverUrl" enctype="multipart/form-data" method="post"> <div class="row"> <label for="fileUploadToServer">Select file</label><br /> fname : <input type="file" name="fname" /> <input type="submit" value="submit"> </div> </form> </html>
Pranav
I believe, assuming that this hasn't been worked on last time I looked at
the code, that this is not currently possible. The code is about half way
there.
The request processing function stores the request on the disk if it is
over a certain size. However, once the request has been uploaded, a
function from Python's cgi module is called to parse the data into
request.params. Not only does this lose some information about file uploads
(which can only be recovered by re-parsing the request), but it also loads
all data into memory.
There is also not currently a way to limit either the size of the request,
or the size of individual files in the request.
Assuming that you need a solution and don't want to work on implementing
this in Twisted, I would suggest using nginx as a reverse proxy in front of
Twisted, with the "Upload" module that can be found on the Addons page on
nginx's wiki. This module will place any uploaded files into files on the
disk, and include regular request parameters to tell your Twisted server
where these files are. It is also capable of putting limits in place on the
size of uploaded files.
Cameron
On 17 July 2013 01:58, Pranav Bhardwaj
Hey
I have a script to be able to upload logs to a twisted web server.
I have the following requirements:
1) to be able to limit the size of the upload to 10MB. 2) to NOT store file in memory during the request.
Does twisted.web provide these functionality?
Any inputs on these would be welcome!
Here is the *server part* os the script:
try: oStream = open(filename, 'wb') oStream.write(request.args['fname'][0]) outputStream.close()
except: # handle exception
a test *client script:*
<!DOCTYPE html> <html> <form action="https://serverUrl" enctype="multipart/form-data" method="post"> <div class="row"> <label for="fileUploadToServer">Select file</label><br /> fname : <input type="file" name="fname" /> <input type="submit" value="submit"> </div> </form> </html>
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
On Jul 16, 2013, at 6:55 PM, Shell
I believe, assuming that this hasn't been worked on last time I looked at the code, that this is not currently possible. The code is about half way there.
The request processing function stores the request on the disk if it is over a certain size. However, once the request has been uploaded, a function from Python's cgi module is called to parse the data into request.params. Not only does this lose some information about file uploads (which can only be recovered by re-parsing the request), but it also loads all data into memory.
The trick to doing this with present twisted.web is to override the 'handleContentChunk' method of Request, rather than trying to catch the streaming-upload behavior in your Resource class. Of course, the right thing to do is to help get https://tm.tl/#288 fixed :). -glyph
participants (3)
-
Glyph -
Pranav Bhardwaj -
Shell