Hallo,
I'm currently realizing a web server for my company which allows control of some of our embedded devices. After some research, I decided on python and twisted as the way to go using genshi as templating engine, which works very well. The problem I encountered is as follows: The user of the web panel should also be able to upgrade the firmware on our device. To make this possible, a file upload has to take place. The firmware file can have a size of up to 150MB, so it will not fit in the RAM of our device. It either won't fit on the NAND flash chip twice, so it should be written to NAND with its final name and not as a temporary file. To tell the server where to store the file upload, I could add a hidden input to the upload form indicating the type of the file being uploaded, so that the web server can save it to the right location. Where's the place to hook in twisted to implement such a behaviour? Do I really have to write a completely independent implementation of the HTTP handler or could I also reuse some parts of twisted.web? Does some code exist already which implements the wanted behaviour or could be modified to implement it?
Thanks for your reply, J. Pietron
On 02:45 pm, julian@whisper-net.de wrote:
Hallo,
I'm currently realizing a web server for my company which allows control of some of our embedded devices. After some research, I decided on python and twisted as the way to go using genshi as templating engine, which works very well. The problem I encountered is as follows: The user of the web panel should also be able to upgrade the firmware on our device. To make this possible, a file upload has to take place. The firmware file can have a size of up to 150MB, so it will not fit in the RAM of our device. It either won't fit on the NAND flash chip twice, so it should be written to NAND with its final name and not as a temporary file. To tell the server where to store the file upload, I could add a hidden input to the upload form indicating the type of the file being uploaded, so that the web server can save it to the right location. Where's the place to hook in twisted to implement such a behaviour? Do I really have to write a completely independent implementation of the HTTP handler or could I also reuse some parts of twisted.web? Does some code exist already which implements the wanted behaviour or could be modified to implement it?
You can override `Request.gotLength` to inspect the request and select an appropriate destination for the request body.
You can convince Site to use your custom request by setting it as the `requestFactory` attribute.
Jean-Paul
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Am 16.07.2010 17:33, schrieb exarkun@twistedmatrix.com:
You can override `Request.gotLength` to inspect the request and select an appropriate destination for the request body.
You can convince Site to use your custom request by setting it as the `requestFactory` attribute.
Jean-Paul
Hi,
thanks for your reply. In meantime I tried cherrypy, which does its job very good but has some flaws compared to my twisted applications when it comes to performance and resource usage. So I'm still interested in solving my problem by using twisted. The problem is as follows: The file which shall be uploaded is sent via a HTML form (encoded as multipart/form-data), and I'd like to use Python's FieldStorage to parse the request body because it handles the file matters very well. You can tell cherrypy, that for a certain request uri cherrypy should not parse the request body but instead pass it on unparsed to the handler method. Is there a simple way of configuring twisted.web in this way to? So that e.g. if request uri /do-upload-form is issued, twisted.web does not try to parse the request body on its own and generate the args dict to pass on to the render_POST method, but instead uses FieldStorage (or better: my own subclass of FieldStorage, which employs a NamedTempFile instead of an anonymous tempfile, which allows me to simply rename it after it has passed some sanity checks) for parsing the body?
Thanks for your reply, Julian
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
After some looks at twisted.web's http.py, I came to the conclusion, that for my solution to work the way of receiving client's request has to be modified quite much. I think twisted does the body parsing when all data has been received (i.e. is stored somewhere in memory). So it would be too late to hook in at this point, because the whole file would've been loaded to memory (or, rather, it wouldn't, because it simply wouldn't fit in). Instead, the way to go would be to parse the Content-type header of the client's request as early as possible to decide whether a) do the parsing in the traditional, twisted.web way (which means to wait until the request has been completely received and then parse it) b) in case of multipart/form-data redirect the following input to an instance of cgi.FieldStorage, which writes any files directly to disk instead of loading their contents to memory, or, alternatively, write an own FieldStorage-like class (but why should one re-implement already existing and well-tested classes?)
The only question that remains is that, while it would (imho) be quiet easy to watch out for an multipart/form-data header, where do I generate the input stream from that FieldStorage expects? In cherrypy, this is quiet simple, because it's not async as twisted is but uses a more traditional approach to sockets, so that an input stream is easily available. But as twisted is async and dataReceived() is called on received data, there is no input stream available to direct to FieldStorage, or am I wrong in this conclusion? If I'm not, is there any way to supply FieldStorage with an input stream without pre-reading the whole request body to memory?
Thanks, Julian
On 23 Jul, 08:34 pm, julian@whisper-net.de wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Am 16.07.2010 17:33, schrieb exarkun@twistedmatrix.com:
You can override `Request.gotLength` to inspect the request and select an appropriate destination for the request body.
You can convince Site to use your custom request by setting it as the `requestFactory` attribute.
Jean-Paul
Hi,
thanks for your reply. In meantime I tried cherrypy, which does its job very good but has some flaws compared to my twisted applications when it comes to performance and resource usage. So I'm still interested in solving my problem by using twisted. The problem is as follows: The file which shall be uploaded is sent via a HTML form (encoded as multipart/form-data), and I'd like to use Python's FieldStorage to parse the request body because it handles the file matters very well. You can tell cherrypy, that for a certain request uri cherrypy should not parse the request body but instead pass it on unparsed to the handler method. Is there a simple way of configuring twisted.web in this way to? So that e.g. if request uri /do-upload-form is issued, twisted.web does not try to parse the request body on its own and generate the args dict to pass on to the render_POST method, but instead uses FieldStorage (or better: my own subclass of FieldStorage, which employs a NamedTempFile instead of an anonymous tempfile, which allows me to simply rename it after it has passed some sanity checks) for parsing the body?
This will be easier once #288 is resolved. Until then, you just have to hack around with the Request to change the behavior at a very low level.
Jean-Paul
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Am 24.07.2010 05:50, schrieb exarkun@twistedmatrix.com:
On 23 Jul, 08:34 pm, julian@whisper-net.de wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Am 16.07.2010 17:33, schrieb exarkun@twistedmatrix.com:
You can override `Request.gotLength` to inspect the request and select an appropriate destination for the request body.
You can convince Site to use your custom request by setting it as the `requestFactory` attribute.
Jean-Paul
Hi,
thanks for your reply. In meantime I tried cherrypy, which does its job very good but has some flaws compared to my twisted applications when it comes to performance and resource usage. So I'm still interested in solving my problem by using twisted. The problem is as follows: The file which shall be uploaded is sent via a HTML form (encoded as multipart/form-data), and I'd like to use Python's FieldStorage to parse the request body because it handles the file matters very well. You can tell cherrypy, that for a certain request uri cherrypy should not parse the request body but instead pass it on unparsed to the handler method. Is there a simple way of configuring twisted.web in this way to? So that e.g. if request uri /do-upload-form is issued, twisted.web does not try to parse the request body on its own and generate the args dict to pass on to the render_POST method, but instead uses FieldStorage (or better: my own subclass of FieldStorage, which employs a NamedTempFile instead of an anonymous tempfile, which allows me to simply rename it after it has passed some sanity checks) for parsing the body?
This will be easier once #288 is resolved. Until then, you just have to hack around with the Request to change the behavior at a very low level.
Jean-Paul
If I've unterstood the source code of http.py, class Request right, the methods to override would be: 1. handleContentChunk, which is called after some data has been received by the transfer decoder. Something that behaves like cgi.FieldStorage, but in an async way, should be installed here / instead of the StringIO used for self.content 2. requestReceived, which has nothing to do anymore besides of calling self.process
The only thing that misses now is request's command (GET, POST, HEAD etc.) and request path. Is there a way to install these in the Request instance before allContentReceived is called other than overriding HTTPChannel's lineReceived?
Thanks, Julian
Python and twisted on an embedded device? Have you considered size of python as well as twisted, have you compiled those on the firmware, because as per my experience, it was difficult to install twisted on firmware, so I wrote a small c program which does the job. As per your question, twisted has a tcpserver, so you can have web interface using twisted.web and file upload using tcpserver, I have never done this but you can try.
Sent from my iPhone
On Jul 16, 2010, at 7:45 AM, Julian Pietron julian@whisper-net.de wrote:
Hallo,
I'm currently realizing a web server for my company which allows control of some of our embedded devices. After some research, I decided on python and twisted as the way to go using genshi as templating engine, which works very well. The problem I encountered is as follows: The user of the web panel should also be able to upgrade the firmware on our device. To make this possible, a file upload has to take place. The firmware file can have a size of up to 150MB, so it will not fit in the RAM of our device. It either won't fit on the NAND flash chip twice, so it should be written to NAND with its final name and not as a temporary file. To tell the server where to store the file upload, I could add a hidden input to the upload form indicating the type of the file being uploaded, so that the web server can save it to the right location. Where's the place to hook in twisted to implement such a behaviour? Do I really have to write a completely independent implementation of the HTTP handler or could I also reuse some parts of twisted.web? Does some code exist already which implements the wanted behaviour or could be modified to implement it?
Thanks for your reply, J. Pietron
Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web