Hallo, I'm currently realizing a web server for my company which allows control of some of our embedded devices. After some research, I decided on python and twisted as the way to go using genshi as templating engine, which works very well. The problem I encountered is as follows: The user of the web panel should also be able to upgrade the firmware on our device. To make this possible, a file upload has to take place. The firmware file can have a size of up to 150MB, so it will not fit in the RAM of our device. It either won't fit on the NAND flash chip twice, so it should be written to NAND with its final name and not as a temporary file. To tell the server where to store the file upload, I could add a hidden input to the upload form indicating the type of the file being uploaded, so that the web server can save it to the right location. Where's the place to hook in twisted to implement such a behaviour? Do I really have to write a completely independent implementation of the HTTP handler or could I also reuse some parts of twisted.web? Does some code exist already which implements the wanted behaviour or could be modified to implement it? Thanks for your reply, J. Pietron
On 02:45 pm, julian@whisper-net.de wrote:
Hallo,
I'm currently realizing a web server for my company which allows control of some of our embedded devices. After some research, I decided on python and twisted as the way to go using genshi as templating engine, which works very well. The problem I encountered is as follows: The user of the web panel should also be able to upgrade the firmware on our device. To make this possible, a file upload has to take place. The firmware file can have a size of up to 150MB, so it will not fit in the RAM of our device. It either won't fit on the NAND flash chip twice, so it should be written to NAND with its final name and not as a temporary file. To tell the server where to store the file upload, I could add a hidden input to the upload form indicating the type of the file being uploaded, so that the web server can save it to the right location. Where's the place to hook in twisted to implement such a behaviour? Do I really have to write a completely independent implementation of the HTTP handler or could I also reuse some parts of twisted.web? Does some code exist already which implements the wanted behaviour or could be modified to implement it?
You can override `Request.gotLength` to inspect the request and select an appropriate destination for the request body. You can convince Site to use your custom request by setting it as the `requestFactory` attribute. Jean-Paul
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 16.07.2010 17:33, schrieb exarkun@twistedmatrix.com:
You can override `Request.gotLength` to inspect the request and select an appropriate destination for the request body.
You can convince Site to use your custom request by setting it as the `requestFactory` attribute.
Jean-Paul
Hi, thanks for your reply. In meantime I tried cherrypy, which does its job very good but has some flaws compared to my twisted applications when it comes to performance and resource usage. So I'm still interested in solving my problem by using twisted. The problem is as follows: The file which shall be uploaded is sent via a HTML form (encoded as multipart/form-data), and I'd like to use Python's FieldStorage to parse the request body because it handles the file matters very well. You can tell cherrypy, that for a certain request uri cherrypy should not parse the request body but instead pass it on unparsed to the handler method. Is there a simple way of configuring twisted.web in this way to? So that e.g. if request uri /do-upload-form is issued, twisted.web does not try to parse the request body on its own and generate the args dict to pass on to the render_POST method, but instead uses FieldStorage (or better: my own subclass of FieldStorage, which employs a NamedTempFile instead of an anonymous tempfile, which allows me to simply rename it after it has passed some sanity checks) for parsing the body? Thanks for your reply, Julian -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.15 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJMSfzNAAoJENidYKvYQHlQNvwQAMdZLl6S2hXgGyMDYEezILEB xSQgaZ0ZcvWbuoAt4wwwYwzWHT778VaYfsnn5IJnpumluylu2KiMj1TDJpumOo7I WvSVpBVojrr1akXx8QLNOYCtFVLteRWOahKYG6P2OQlf5moJm/E4PfAKwVn4Ogba TgJ3joujIYyZy/WX5n+TBHWweT1dthr7a1nDsJJzMhTuqW7X3O9lw+ewC83n320L 3V4rO0rgBF84ltCjiiOK10lR5F2VjjkS975Y1N6CEYhtl0NORbWmKIummb5bpnjT uSqvwCb0WOXhzvPPHs+MDXDCxkW83TmFOHtCsFnWtS9FXmTUuMi1eNZX8PyshT9D 2t8lvdFFZFwvsgR0BATt+jI3nJzcz06Xo1OrYUr2G91UI8ITsFRcP4EdawKFAfap CInUvB7bZMDe19j+cRlGGRBEiLFaLMG6KUoaCFkHWVleMmgKPuEtq4Iz9QTNRNq1 HMZDFhmYIJGa8sSbijI6+UqrCqY70VcMQNScm0D3nUlLlXEoYM+myY2VC89NLWfx 8IN44CzLBI2KO9J8B2m/dk8rMRUvYRKP+Wll3OyOEdQvCh7pJh3auIn8nY3P14Az 5E3LvXxOJP4NCHIW8rImy/3Fpfl4dOANF7QuXdeXBp8XOnkG5qX69cqX6T2Cemtq 9qCEF/ddp3yoiNr4Kwng =EPhL -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 After some looks at twisted.web's http.py, I came to the conclusion, that for my solution to work the way of receiving client's request has to be modified quite much. I think twisted does the body parsing when all data has been received (i.e. is stored somewhere in memory). So it would be too late to hook in at this point, because the whole file would've been loaded to memory (or, rather, it wouldn't, because it simply wouldn't fit in). Instead, the way to go would be to parse the Content-type header of the client's request as early as possible to decide whether a) do the parsing in the traditional, twisted.web way (which means to wait until the request has been completely received and then parse it) b) in case of multipart/form-data redirect the following input to an instance of cgi.FieldStorage, which writes any files directly to disk instead of loading their contents to memory, or, alternatively, write an own FieldStorage-like class (but why should one re-implement already existing and well-tested classes?) The only question that remains is that, while it would (imho) be quiet easy to watch out for an multipart/form-data header, where do I generate the input stream from that FieldStorage expects? In cherrypy, this is quiet simple, because it's not async as twisted is but uses a more traditional approach to sockets, so that an input stream is easily available. But as twisted is async and dataReceived() is called on received data, there is no input stream available to direct to FieldStorage, or am I wrong in this conclusion? If I'm not, is there any way to supply FieldStorage with an input stream without pre-reading the whole request body to memory? Thanks, Julian -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.15 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJMSgsnAAoJENidYKvYQHlQhNgQAJk4u3VklMZeJkHvCn/H+mHv i6TDwkGAXEv5fhPxbSVNb+5zDQUS1w305Zx9/oQeJldViKzGN82p4x2D5doapzOI OD7Nm8wDo/giFByjLwXffxvpFKcz/fyKgS/eK8w6gj0MJQ9ZIqpwEXj9L3OvmFgx VUaVO3hDv4BRIxLCvG1dZYmdvqnY0W8/z1Bv472ZU48SoK+NhF+zNhhq3VPFDwOv J56HWrWR7DXjoBZvgxr6r39kmjpblkoknppAXhgEMMPmsSsKe3tG4QVuFUkvqjKL 4IajlSo+Lu9bqAwHhZBw91XfFJhWtflWxQQjgoCWAa3TeNj97WrDGI3y4MRCnDyt 36i4DRK1FMqmnUcxEXMW0C0pviU4caQASmQ+74RsgGOsgCFOtznZOEBk4RN+cLLk 6ONPHpD3q0mHhrzFcIQTgebocMyjnmK+I8AuPirwrVwF1plsKxozNTmHPPJKR1F3 hdWWk0z7ywOlkfpgBpO8sbCmp0bNbryCpCXLG8Qq3SqKbFW8nfwC/SYWr0P1nM6t VBTAxw31HfH0897xXK/wdnGIM7glBkypagcVp+YcyNJx/QN33zEA7M2Gn1h4sWpg o+tpTfbI201wrUWooM8U0srGa0fLB3ftjXAFfwOUwJVUVmP47hZLDisPaHrzdmZU dpTV/NcYhuqSnN+PgflF =z4LY -----END PGP SIGNATURE-----
On 23 Jul, 08:34 pm, julian@whisper-net.de wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Am 16.07.2010 17:33, schrieb exarkun@twistedmatrix.com:
You can override `Request.gotLength` to inspect the request and select an appropriate destination for the request body.
You can convince Site to use your custom request by setting it as the `requestFactory` attribute.
Jean-Paul
Hi,
thanks for your reply. In meantime I tried cherrypy, which does its job very good but has some flaws compared to my twisted applications when it comes to performance and resource usage. So I'm still interested in solving my problem by using twisted. The problem is as follows: The file which shall be uploaded is sent via a HTML form (encoded as multipart/form-data), and I'd like to use Python's FieldStorage to parse the request body because it handles the file matters very well. You can tell cherrypy, that for a certain request uri cherrypy should not parse the request body but instead pass it on unparsed to the handler method. Is there a simple way of configuring twisted.web in this way to? So that e.g. if request uri /do-upload-form is issued, twisted.web does not try to parse the request body on its own and generate the args dict to pass on to the render_POST method, but instead uses FieldStorage (or better: my own subclass of FieldStorage, which employs a NamedTempFile instead of an anonymous tempfile, which allows me to simply rename it after it has passed some sanity checks) for parsing the body?
This will be easier once #288 is resolved. Until then, you just have to hack around with the Request to change the behavior at a very low level. Jean-Paul
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Am 24.07.2010 05:50, schrieb exarkun@twistedmatrix.com:
On 23 Jul, 08:34 pm, julian@whisper-net.de wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Am 16.07.2010 17:33, schrieb exarkun@twistedmatrix.com:
You can override `Request.gotLength` to inspect the request and select an appropriate destination for the request body.
You can convince Site to use your custom request by setting it as the `requestFactory` attribute.
Jean-Paul
Hi,
thanks for your reply. In meantime I tried cherrypy, which does its job very good but has some flaws compared to my twisted applications when it comes to performance and resource usage. So I'm still interested in solving my problem by using twisted. The problem is as follows: The file which shall be uploaded is sent via a HTML form (encoded as multipart/form-data), and I'd like to use Python's FieldStorage to parse the request body because it handles the file matters very well. You can tell cherrypy, that for a certain request uri cherrypy should not parse the request body but instead pass it on unparsed to the handler method. Is there a simple way of configuring twisted.web in this way to? So that e.g. if request uri /do-upload-form is issued, twisted.web does not try to parse the request body on its own and generate the args dict to pass on to the render_POST method, but instead uses FieldStorage (or better: my own subclass of FieldStorage, which employs a NamedTempFile instead of an anonymous tempfile, which allows me to simply rename it after it has passed some sanity checks) for parsing the body?
This will be easier once #288 is resolved. Until then, you just have to hack around with the Request to change the behavior at a very low level.
Jean-Paul
If I've unterstood the source code of http.py, class Request right, the methods to override would be: 1. handleContentChunk, which is called after some data has been received by the transfer decoder. Something that behaves like cgi.FieldStorage, but in an async way, should be installed here / instead of the StringIO used for self.content 2. requestReceived, which has nothing to do anymore besides of calling self.process The only thing that misses now is request's command (GET, POST, HEAD etc.) and request path. Is there a way to install these in the Request instance before allContentReceived is called other than overriding HTTPChannel's lineReceived? Thanks, Julian -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.15 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQIcBAEBAgAGBQJMSt1YAAoJENidYKvYQHlQt5UP+waxLHEVCHQOOY7saeECwEEZ QsxZpsEyHKd8imcpfrzIgQaooj5OlcFLpAWkaaGFOhUDxq6RmXrLL60/awCTSiCI Xv1EsNJiwml5/uemhny4BCUq1aBcM0qcMg6BAZG8jwDy2nAVgyX3kall2a+17ybr aBnssceBHFJm2I7o/yV9ZkrzIPtOabZKqEAyBlE9pqYA1SHZfKhuUZw+BQMpdGdV HnE+58YRk39B1sTi5GQc5Qed3ynTxeE8tLTHgvi/99QMs+pGQP7kmFbq5KYch8ks IGVBYJCZXoSZ6aow6zkYSZ8/hmiI5bS3OAnzVSkcOz122UlCqhicSQ7aUUJXrKKs Er9R4ETNbCYAkZqw+8XofSgcawuP3XxDBfChPeDAC7G1bjnVEO7vEAeUyl+wQ9AO UXnQde25HAIz88ouXo50Au7Q0Wx/qpnC7Xnl2u6Bbykj0c1UNg1146ZyFCXx3L7f OT7kQgAIGm3cbCtVU1RLKegsXJD318xEEAHEUoAOjLzbHbuM7cLy6lP5udz/U4QF pcbpEahNesMKiD7qCGrFzp7nEHu2A272iSN3rbOq0YQILCMhe9uBFqwP1KPsPcMQ 2Or1/n0meu00kZqrFy2t7SsPg3c2754UwC9kLwWXrsP/XoP3HG+YeBFe54sowB2S lEmLCSwtNlEHH0dy4rmr =IJw9 -----END PGP SIGNATURE-----
Python and twisted on an embedded device? Have you considered size of python as well as twisted, have you compiled those on the firmware, because as per my experience, it was difficult to install twisted on firmware, so I wrote a small c program which does the job. As per your question, twisted has a tcpserver, so you can have web interface using twisted.web and file upload using tcpserver, I have never done this but you can try. Sent from my iPhone On Jul 16, 2010, at 7:45 AM, Julian Pietron <julian@whisper-net.de> wrote:
Hallo,
I'm currently realizing a web server for my company which allows control of some of our embedded devices. After some research, I decided on python and twisted as the way to go using genshi as templating engine, which works very well. The problem I encountered is as follows: The user of the web panel should also be able to upgrade the firmware on our device. To make this possible, a file upload has to take place. The firmware file can have a size of up to 150MB, so it will not fit in the RAM of our device. It either won't fit on the NAND flash chip twice, so it should be written to NAND with its final name and not as a temporary file. To tell the server where to store the file upload, I could add a hidden input to the upload form indicating the type of the file being uploaded, so that the web server can save it to the right location. Where's the place to hook in twisted to implement such a behaviour? Do I really have to write a completely independent implementation of the HTTP handler or could I also reuse some parts of twisted.web? Does some code exist already which implements the wanted behaviour or could be modified to implement it?
Thanks for your reply, J. Pietron
_______________________________________________ Twisted-web mailing list Twisted-web@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-web
participants (3)
-
Arun -
exarkun＠twistedmatrix.com -
Julian Pietron