-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

After some looks at twisted.web's http.py, I came to the conclusion, that for my solution to work the way of receiving client's request has to be modified quite much. I think twisted does the body parsing when all data has been received (i.e. is stored somewhere in memory). So it would be too late to hook in at this point, because the whole file would've been loaded to memory (or, rather, it wouldn't, because it simply wouldn't fit in). Instead, the way to go would be to parse the Content-type header of the client's request as early as possible to decide whether a) do the parsing in the traditional, twisted.web way (which means to wait until the request has been completely received and then parse it) b) in case of multipart/form-data redirect the following input to an instance of cgi.FieldStorage, which writes any files directly to disk instead of loading their contents to memory, or, alternatively, write an own FieldStorage-like class (but why should one re-implement already existing and well-tested classes?)

The only question that remains is that, while it would (imho) be quiet easy to watch out for an multipart/form-data header, where do I generate the input stream from that FieldStorage expects? In cherrypy, this is quiet simple, because it's not async as twisted is but uses a more traditional approach to sockets, so that an input stream is easily available. But as twisted is async and dataReceived() is called on received data, there is no input stream available to direct to FieldStorage, or am I wrong in this conclusion? If I'm not, is there any way to supply FieldStorage with an input stream without pre-reading the whole request body to memory?

Thanks, Julian

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Am 24.07.2010 05:50, schrieb exarkun@twistedmatrix.com:

On 23 Jul, 08:34 pm, julian@whisper-net.de wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Am 16.07.2010 17:33, schrieb exarkun@twistedmatrix.com:

...

You can override `Request.gotLength` to inspect the request and select an appropriate destination for the request body.

You can convince Site to use your custom request by setting it as the `requestFactory` attribute.

Jean-Paul

Hi,

thanks for your reply. In meantime I tried cherrypy, which does its job very good but has some flaws compared to my twisted applications when it comes to performance and resource usage. So I'm still interested in solving my problem by using twisted. The problem is as follows: The file which shall be uploaded is sent via a HTML form (encoded as multipart/form-data), and I'd like to use Python's FieldStorage to parse the request body because it handles the file matters very well. You can tell cherrypy, that for a certain request uri cherrypy should not parse the request body but instead pass it on unparsed to the handler method. Is there a simple way of configuring twisted.web in this way to? So that e.g. if request uri /do-upload-form is issued, twisted.web does not try to parse the request body on its own and generate the args dict to pass on to the render_POST method, but instead uses FieldStorage (or better: my own subclass of FieldStorage, which employs a NamedTempFile instead of an anonymous tempfile, which allows me to simply rename it after it has passed some sanity checks) for parsing the body?

This will be easier once #288 is resolved. Until then, you just have to hack around with the Request to change the behavior at a very low level.

Jean-Paul

If I've unterstood the source code of http.py, class Request right, the methods to override would be: 1. handleContentChunk, which is called after some data has been received by the transfer decoder. Something that behaves like cgi.FieldStorage, but in an async way, should be installed here / instead of the StringIO used for self.content 2. requestReceived, which has nothing to do anymore besides of calling self.process

The only thing that misses now is request's command (GET, POST, HEAD etc.) and request path. Is there a way to install these in the Request instance before allContentReceived is called other than overriding HTTPChannel's lineReceived?

Thanks, Julian