[Twisted-Python] Streaming HTTP
![](https://secure.gravatar.com/avatar/214c694acb154321379cbc58dc91528c.jpg?s=120&d=mm&r=g)
Folks, # Problem Statement Thanks for your feedback on my HTTP/2 questions. I’ve started work implementing a spike of a HTTP/2 protocol for twisted.web. I’m aiming to have something that works in at least some cases by the end of the day. As part of my dive into twisted.web, I noticed something that surprised me: it seems to have no support for ‘streaming’ request bodies. By this I mean that the Request.requestReceived() method is not actually called until the complete request body has been received. This is a somewhat unexpected limitation for Twisted: why should I have to wait until the entire body has been uploaded to start doing things with it? This problem is thrown into sharp relief with HTTP/2, which essentially always chunks the body, even if a content-length is provided. This means that it is now very easy to receive data in delimited chunks, which an implementation may want to have semantic meaning. However, the request is unable to access this data in this way. It also makes it impossible to use a HTTP/2 request/response pair as a long-running communication channel, as we cannot safely call requestReceived until the response is terminated (which also terminates the HTTP/2 stream). Adi pointed me at a related issue, #6928[0], which itself points at what appears to be an issue tracking exactly this request. That issue is issue #288[1], which is 12 years old(!). This has clearly been a pain point for quite some time. Issue #6928 has glyph suggesting that we come to the mailing list to discuss this, but the last time it was raised no responses were received[2]. I believe that with HTTP/2 on the horizon, this issue is more acute than it was before, and needs solving if Twisted is going to continue to remain relevant for the web. It should also allow people to build more performant web applications, as they should be able to handle how the data queues up in their apps. This does not immediately block my HTTP/2 work, so we can take some time and get this right. # Proposed Solution To help us move forward, I’m providing a proposal for how I’d solve this problem. This is not necessarily going to be the final approach, but is instead a straw-man we can use to form the basis of a discussion about what the correct fix should be. My proposal is to deprecate the current Request/Resource model. It currently functions and should continue to function, but as of this point we should consider it a bad way to do things, and we should push people to move to a fully asynchronous model. We should then move to an API that is much more like the one used by Go: specifically, that by default all requests/responses are streamed. Request objects (and, logically, any other object that handles requests/responses, such as Resource) should be extended to have a chunkReceived method that can be overridden by users. If a user chooses not to override that method, the default implementation would continue to do what is done now (save to a buffer). Once the request/response is complete (marked by receipt of a zero-length chunk, or a frame with END_STREAM set, or when the remaining content-length is 0), request/responseComplete would be called. For users that did not override chunkReceived can now safely access the content buffer: other users can do whatever they see fit. We’d also update requestReceived to ensure that it’s called when all the *headers* are received, rather than waiting for the body. A similar approach should be taken with sending data: we should assume that users want to chunk it if they do not provide a content-length. An extreme position to take (and I do) is that this should be sufficiently easy that most users actually *accidentally* end up chunking their data: that is, we do not provide special helpers to set content-length, instead just checking whether that’s a header users actually send, and if they don’t we chunk the data. This logic would make it much easier to work with HTTP/2 *and* with WebSockets, requiring substantially less special-case code to handle the WebSocket upgrade (when the headers are complete, we can spot the upgrade easily). What do people think of this approach? Cory [0]: https://twistedmatrix.com/trac/ticket/6928 [1]: https://twistedmatrix.com/trac/ticket/288 [2]: https://twistedmatrix.com/pipermail/twisted-python/2014-February/028069.html
![](https://secure.gravatar.com/avatar/fa611b50fabecfe29749b66ec52f46a0.jpg?s=120&d=mm&r=g)
What do people think of this approach?
It sounds fine to me. I think this issue in Twisted is currently the blocker (or at least *one of* the blockers) for Tahoe-LAFS ticket #113, #320, and #1032. It is also a blocker for the much more ambitious Tahoe-LAFS tickets #1288 and #1851. Regards, Zooko https://tahoe-lafs.org/trac/tahoe-lafs/ticket/113# command-line: do things in an incremental fashion and accept stdin as input https://tahoe-lafs.org/trac/tahoe-lafs/ticket/320# add streaming (on-line) upload to HTTP interface https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1032# Display active HTTP upload operations on the status page https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1288# support streaming uploads in uploader https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1851# new immutable file upload protocol: streaming, fewer round-trips, quota-respecting
![](https://secure.gravatar.com/avatar/869963bdf99b541c9f0bbfb04b0320f1.jpg?s=120&d=mm&r=g)
On Sat, 14 Nov 2015 at 17:45 Cory Benfield <cory@lukasa.co.uk> wrote:
As far as I know, this is exactly what is done in the existing HTTP/1.x implementation: if a Content-Length header field is not set before data is written to the request object (as it is called), chunked transfer encoding is automatically used for the response. This behaviour is required by HTTP/1.1.
![](https://secure.gravatar.com/avatar/214c694acb154321379cbc58dc91528c.jpg?s=120&d=mm&r=g)
On 15 Nov 2015, at 10:18, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
How does flow-control work with the Go API? How does user code processing chunks received unleash backpressure onto the sender?
One caveat: I am not an expert in what Go is doing here, more a casual user. However, my understanding is that Go code blocks will not handle chunks in parallel, or provide a new chunk until the previous one has been processed (essentially, the ‘chunk handling’ function is processed synchronously, once for each chunk), which means that it does not read more data from the socket. This exerts TCP level back pressure, and can be adjusted to exert HTTP/2 back-pressure if we’re sufficiently careful about it. Generally speaking in Twisted we could achieve this too, by careful use of Deferreds.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Nov 17, 2015, at 2:49 AM, Cory Benfield <cory@lukasa.co.uk> wrote:
Generally speaking in Twisted we could achieve this too, by careful use of Deferreds.
I have a bigger reply coming; however, I should note that this is basically what web2 did with its IStream interface. While this has a big advantage over the status quo (i.e. "not solving the problem") the _way_ it solved the problem ended up being both (A) error prone, and (B) slow. It's the sort of slowness that PyPy doesn't even help mitigate, because all the callbacks are heap-allocated, and... to be honest I don't even fully understand it, but I have experimentally verified that it's still pretty slow :-). This is more or less the reason that Tubes exists, although more on how we might proceed either with or without tube in another message... -glyph
![](https://secure.gravatar.com/avatar/214c694acb154321379cbc58dc91528c.jpg?s=120&d=mm&r=g)
On 18 Nov 2015, at 05:55, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Does HTTP2 have choke/unchoke notifications on individual streams? Ultimately this does resolve to TCP backpressure, though…
Not in such a binary form, no. The connection as a whole and each stream maintain independent flow control windows. This allows for pressure to be exerted on the sender to slow down, by allowing the flow control window to drop to zero. This means that there is some Twisted-level buffering, because we do have to get that data out of the socket and to queue at the application, but the amount of data to buffer is strictly bounded. Thus, if our application moves slowly, the remote side should be passively notified to slow down by the lack of window updates: we should only send those window updates once the application has actually taken some data from us. Cory
![](https://secure.gravatar.com/avatar/c4c5daddd52a9c703809ac68780cfe6a.jpg?s=120&d=mm&r=g)
Hi, sounds like a good idea. One topic: What should happen if the sender sends malformed data (e.g. content-length: 100 and sending 200 bytes body). I would usually want to know if the sender is broken instead of getting a responseComplete() that claims everything is fine. We simply subclassed twisted.web.server.Request and injected the needed parts so we could get the data we wanted in an incremental way and set some socket options (SO_RCVBUF) for multi-gigabyte file uploads. So we took 'gotLength()' as a signal that headers are fully received and also took 'write()', 'processingFailed()' to handle cleanup. Basically we replace the self.content stream object with our own version that does not buffer everything in memory. We tried to do the 100-Continue stuff too, but it simply didn't work with the Python stdlib httplib code (it just ignores those more or less). So we either just terminate the connection forcefully (reset) or read the full request and send it to /dev/null. Michael -- Michael Schlenker Senior Software Engineer CONTACT Software GmbH Tel.: +49 (421) 20153-80 Wiener Straße 1-3 Fax: +49 (421) 20153-41 28359 Bremen E-Mail: michael.schlenker@contact-software.com http://www.contact-software.com/ Registered office: Bremen, Germany Managing directors: Karl Heinz Zachries, Ralf Holtgrefe Court of register: Amtsgericht Bremen HRB 1321
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
This should definitely be an error. While this isn't totally irrelevant to the design of the new server API - the implementation should surely take this kind of error-checking into account, and the old one didn't so well - it's a bit of a distraction since it's not an essential difference. -glyph
![](https://secure.gravatar.com/avatar/c1cd4fcd6c951797d9567954cf2ccca0.jpg?s=120&d=mm&r=g)
Hi Cory,
I think it is worth noting that some of these issues were solved by twisted.web2, to the point that there were even adapters created that handled some of the differences between the existing web resource model and the new model in web2. Of course, web2 did not succeed in replacing web, due in part to concerns over the introduction of a new flow-control mechanism (IStream and company). Today though, we have https://github.com/twisted/tubes, which is being developed outside of twisted proper, but could perhaps be leveraged to overcome some of the previous concerns about introducing a mechanism different from IProducer/IConsumer. I mention this because I think there is work within the web2 codebase that might prove useful in solving some of the issues being discussed here, in a way that would benefit twisted, and its users. Thanks for reading, L. Daniel Burr
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Hooray!
As part of my dive into twisted.web, I noticed something that surprised me: it seems to have no support for ‘streaming’ request bodies. By this I mean that the Request.requestReceived() method is not actually called until the complete request body has been received. This is a somewhat unexpected limitation for Twisted: why should I have to wait until the entire body has been uploaded to start doing things with it?
This is exactly what <https://twistedmatrix.com/trac/ticket/288> is about (as you note).
I'm very glad to hear you say this, because: (A) we should absolutely make progress on this ticket now that there is some impetus to do so, but, (B) we should absolutely NOT let this new API block any purely protocol-level HTTP2 work that needs to proceed (I say "protocol level" as a contrast to "feature level" because we could totally implement a thing that speaks HTTP 2 against the current API, although it might be awkward to expose the advantageous parts of the protocol as API features until we do some cleanup)
We have considered it a bad way to do things for a long time. There have been several attempts to rewrite it (Nevow's request model, web2) but none of them have really been the comprehensive re-design we need.
We should then move to an API that is much more like the one used by Go: specifically, that by default all requests/responses are streamed. Request objects (and, logically, any other object that handles requests/responses, such as Resource) should be extended to have a chunkReceived method that can be overridden by users.
No. Let me elaborate :-). First of all, this is already the case (sort of). twisted.web.server.Request inherits from twisted.web.http.Request; as you can see in http.Request, there is already a method called handleContentChunk, which is called by HTTPChannel. By overriding this method, you can already handle request data streaming in from off the wire. This is one of the reasons that #288 is so interminable: using Twisted's public API, today, you can totally write an HTTP-based application that happily streams data from the wire. The only problem is that this API does not propagate up to Resource objects, because Resource objects can expect the (still, as of this writing, undocumented) "content" attribute to have been filled out in getChild. Mostly, they don't, actually! But it's impossible to tell if they might in the general case. You can (and many applications have) just broken the technical compatibility contract with Resource, and written a subclass of twisted.web.server.Site that has a custom requestFactory method that returns a 'streamed' resource. So, if we're already doing this, why "no"? Superclasses with overridable methods are a terrible mechanism for exposing extensibility. These are used extensively throughout Twisted, the older the API the more inheritance it uses. Newer code, you may notice, is generally written much more in a pattern of delegation to formal interfaces. So we have tried to learn our lesson here.
If a user chooses not to override that method, the default implementation would continue to do what is done now (save to a buffer). Once the request/response is complete (marked by receipt of a zero-length chunk, or a frame with END_STREAM set, or when the remaining content-length is 0),
For what it's worth, I hope that all of these will be exposed as the same event to applications, since the fact that these differ on the wire is entirely an implementation detail?
request/responseComplete would be called. For users that did not override chunkReceived can now safely access the content buffer: other users can do whatever they see fit. We’d also update requestReceived to ensure that it’s called when all the *headers* are received, rather than waiting for the body.
Again, this is very similar to what already happens, at the layer of the HTTP protocol. The question is, how do you indicate that you're delegating to a Resource object which may expect the .content attribute to already be populated during .getChild?
A similar approach should be taken with sending data: we should assume that users want to chunk it if they do not provide a content-length. An extreme position to take (and I do) is that this should be sufficiently easy that most users actually *accidentally* end up chunking their data: that is, we do not provide special helpers to set content-length, instead just checking whether that’s a header users actually send, and if they don’t we chunk the data.
request.write() already basically does this, I think? Here, at least, we have lots of opportunity to make the implementation do smarter things (better error checking regarding content-length, for example) without changing the interface at all.
This logic would make it much easier to work with HTTP/2 *and* with WebSockets, requiring substantially less special-case code to handle the WebSocket upgrade (when the headers are complete, we can spot the upgrade easily).
What do people think of this approach?
So I think you're roughly on the right track but there are probably some Twisted-level gaps to fill in. I've already gestured in the direction of Tubes (as have others) and it's something to think about. But before we get to that, let's talk about a much more basic deficiency in the API: although there's an "IRequest", and an "IResource", there's no such thing as an "IResponse". Instead, "IRequest" stands in for both the request and the response, because you write directly to a request (implicitly filling out its response as you do so). Luckily we have an existing interface that might point the way to a better solution, both for requests and responses: specifically, the client IResponse: https://twistedmatrix.com/documents/15.4.0/api/twisted.web.iweb.IResponse.ht.... This interface is actually pretty close to what we want for a server IResponse as well. Perhaps even identical. Its static data is all exposed as attributes which can be relatively simply inspected, and the way it delivers a streaming response is that it delivers its body to an IProtocol implementation (via .deliverBody(aProtocol)). This is not quite as graceful as having a .bodyFount() method that returns an IFount from the tubes package; however, the tubes package is still not exactly mature software, so we may not want to block on depending on it. Importantly though, this delivers all the events you need as a primitive for interfacing with such a high-level interface; it would definitely be better to add this sort of interface Real Soon Now, because then the tubes package could simply have a method, responseToFount (which it will need anyway to work with Agent) that calls deliverBody internally. This works as a primitive because you have all the hooks you need for flow-control. This protocol receives, to its 'makeConnection' method, an ITransport which can provide the IProducer https://twistedmatrix.com/documents/15.4.0/api/twisted.internet.interfaces.I... and IConsumer https://twistedmatrix.com/documents/15.4.0/api/twisted.internet.interfaces.I... interfaces for flow-control. It receives dataReceived to tell it a chunk has arrived and connectionLost to tell it the stream has terminated. Unfortunately the client IRequest https://twistedmatrix.com/documents/15.4.0/api/twisted.web.iweb.IClientReque... isn't quite as useful (although its relative minimalism should be an inspiration to anyone designing a next-generation IRequest more than the current IRequest's sprawling kitchen-sink aesthetic). However, IResponse.deliverBody could be applied to IGoodRequest as well. If we have a very similar-to-IResponse shaped IRequest object, say with 'method', 'uri' and 'headers', and then a 'deliverBody' that delivers the request body in much the same way, we could get a gracefully structured streaming request with works with a lot of existing code within Twisted. Then the question is: what to do with IResource? Right now the flow of processing a request is, roughly: -> wait for full request to arrive -> have HTTPChannel fill out IRequest object -> look at request.site.resource for the root *-> call getChildWithDefault repeatedly, mutating "cursor" state on the IRequest as you move (specifically: "prepath" and "postpath" attributes) -> eventually reach the leaf Resource, or one with 'isLeaf' set on it, and delegate producing the response to that resource *-> call resource.render(request) -> examine the return value; if it's bytes, deliver them and close the connection; NOT_DONE_YET, just leave the connection open, Instead, I think a good flow would be: -> receive method/headers from request -> recurse down from request.site.resource, calling something like nevow's or web2's locateChild, but not modifying 'request' at each stage; instead, pass a "cursor" object - perhaps indeed just a twisted.python.url.URL - indicating where we are in the resource traversal hierarchy. the reason the request.prepath and request.postpath attributes exist is mainly for Resource objects to be able to orient themselves within a resource tree and generate links. also, it probably bears some explanation; the signature of the current "get the next resource" call is resource.getChildWithDefault(onePathSegment, request) -> resource. This is somewhat limiting as it requires you to consume only an individual path segment at a time, which can be highly awkward for implementing sites that have a URL structure that is, for example, /YYYY/MM/DD/HH/MM/index.html. Instead, locateChild took the entire remaining path, and returned a 2-tuple of a resource, and the _still_ remaining path. So for the above, you could do: def locateChild(self, request, path): y, m, d = path[:3] return ymdresource(y, m, d), path[3:] that 2-tuple instructs the traversal machinery, "keep going". One alternative that we toyed with for this was to make consuming the path destructive, since that made it a lot easier to tell what resource you were "looking at": def locateChild(self, request, path): y, m, d = path.consume(3) return ymdresource(y, m, d) Either of these approaches also let you implement 'isLeaf' attribute without special support from the framework; you simply return leaf(), () or path.consume(path.length) -> finally, call .responseForRequest(request) -> IResponse on the final Resource and deliver the IResponse to the network. The way compatibility could be achieved here is to write a wrapper that would implement .responseForRequest to first collect the entire body, then synthesize a gross old-style-IRequest-like object out of the combination of that body and the other information about the resource, then call .getChildWithDefault on it a few times, then call the old-style .render_GET, et. al. The IResponse returned from this compatibility .responseForRequest would wrap up calls like request.write and turn them into write() calls. This is long and increasingly rambly, so I should probably stop now, send it, and get your feedback. Does the strategy I'm proposing make sense? I'm sure I'm leaving a ton out so feel free to ask for clarification. Hopefully I didn't leave too many incomplete sentences in the middle. -glyph
![](https://secure.gravatar.com/avatar/214c694acb154321379cbc58dc91528c.jpg?s=120&d=mm&r=g)
On 18 Nov 2015, at 12:18, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Sorry about the delay in responding to this, but I wanted to make sure I knew at least a bit about what I was talking about before I responded!
So, I think in general this is interesting. One of the big difficulties I’m having right now is that I’m trying to combine this “streaming HTTP” work with the implementation of HTTP/2, which means that I need to keep the HTTP/2 work in mind whenever I talk about this *and* update the HTTP/2 design in response to decisions we make here. This means I’ve got quite a lot of balls in the air right now, and I am confident I’ll drop quite a few. One thing I’m deliberately not doing here is considering Tubes, in part because I’m extremely concerned about backward compatibility, and want the HTTP/2 work to function in the same environment. Unfortunately, this means this conversation is blending into the HTTP/2 one, so I’m going to hijack this thread and bring in some concrete discussion of what I’m working on with the HTTP/2 stuff. I was having a conversation about the HTTP/2 architecture on #twisted-dev yesterday, which has led towards my current working approach for HTTP/2, which will be to have two underlying objects. We’ll have H2Connection, which implements IProtocol, and H2Stream, which implements ITransport. These two objects will be *extremely* tightly coupled: H2Stream cannot meaningfully run over an arbitrary transport mechanism, and knows a great deal about how H2Connections work. The reason we need to take this approach is because IConsumer doesn’t allow for us to have correlators, so even if we only had H2Connection it wouldn’t be able to identify a given producer with the stream it holds. By extension, IConsumer cannot consume multiple producers at once. For this reason, we need an interface between H2Connection and H2Stream that is similar to ITransport and IConsumer, but more featureful. Basically, H2Stream is a thin shim between a producer and H2Connection that adds a stream ID to a few function calls.
Just let me clarify how this is expected to work. Somewhere we have a t.w.s.Site, which builds some kind of HTTP protocol (currently HTTPChannel, in future some object that can transparently swap between HTTPChannel and H2Connection) when connections are received. These two protocols each build an IGoodRequest, which is very similar to IRequest but has a deliverBody method. The consumer of this (whether IResource or some other thing). These objects, if they want to consume a stream, register a protocol via deliverBody. At this point, H2Connection (via H2Stream) provides itself as the transport to that protocol, and calls deliverBody when chunks of data are received. When the object receiving the request is ready to send a response, it calls…something (sendResponse?) and provides an object implementing a server IResponse. The code in the H2Stream/H2Connection sends the headers, then calls deliverBody on the IResponse, passing H2Connection (again via H2Stream) as the protocol that gets called. In this world, H2Stream actually would need to implement IProtocol as well as ITransport. Is my understand of that correct? If so, I think this design can work: essentially, H2Stream becomes the weird intermediary layer that appears as both a transport and a protocol to the request/response layer. Underneath the covers it mostly delegates to H2Connection, which implements a slightly weirdo version of IConsumer (and in fact IProducer) that can only be consumed by H2Stream.
[snip long discussion of how to write locateChild] Agreed that these proposed approaches would work well. I have no concrete feedback on them, they seem good to me.
-> finally, call .responseForRequest(request) -> IResponse on the final Resource and deliver the IResponse to the network.
The way compatibility could be achieved here is to write a wrapper that would implement .responseForRequest to first collect the entire body, then synthesize a gross old-style-IRequest-like object out of the combination of that body and the other information about the resource, then call .getChildWithDefault on it a few times, then call the old-style .render_GET, et. al. The IResponse returned from this compatibility .responseForRequest would wrap up calls like request.write and turn them into write() calls.
This seems super-gross but vaguely do-able, and we’ll need to write it in order to get the new H2Connection/H2Stream objects working with the old paradigm anyway. All of this approach sounds reasonable modulo some careful thinking about how exactly we tie this in with the old paradigm. I’m particularly concerned about H2Channel, which I suspect many applications may know a great deal about. Changing its interface is likely to be slightly tricky, but we’ll see how it goes. Cory
![](https://secure.gravatar.com/avatar/214c694acb154321379cbc58dc91528c.jpg?s=120&d=mm&r=g)
For those interested in how this is progressing, there’s a draft patch available showing some of the proposed direction of this work, at #7460. I’d like as much review as possible, so please weigh in: https://twistedmatrix.com/trac/ticket/7460 Cory
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Clearly this is a challenging topic that requires lots of thought on the part of each interlocutor, and may require long rounds of consideration before each reply. No need to apologize.
Hijack away. I think we should be primarily concerned with getting HTTP/2 integrated for the moment. The reason this raises so many concerns related to the streaming stuff is that the internal implementation of HTTP/2 ought to be more amenable to pulling apart to fit into an actually good interface to the HTTP protocol. I think that twisted._threads points in a promising direction for this sort of work: let's make the old, crappy HTTP APIs work as-is, but with a new, private implementation that is better-factored but not fully documented. We have the old interface as a proof-of-concept, so the new stuff needs to at least be good enough to be an internal implementation detail for that; we don't have to commit to a new public API to land it, and hopefully with some minor edits we can just make it public as the "good" interface (and then backport HTTP/1.1 over it, since we will probably be dealing with legacy HTTP/1.1 clients and servers until we're all dead).
I was having a conversation about the HTTP/2 architecture on #twisted-dev yesterday, which has led towards my current working approach for HTTP/2, which will be to have two underlying objects. We’ll have H2Connection, which implements IProtocol, and H2Stream, which implements ITransport. These two objects will be *extremely* tightly coupled: H2Stream cannot meaningfully run over an arbitrary transport mechanism, and knows a great deal about how H2Connections work.
This seems good, except for the "extreme" tight coupling. IProtocol and ITransport aren't that tightly coupled. Why do H2Stream and H2Connection need to be?
The reason we need to take this approach is because IConsumer doesn’t allow for us to have correlators, so even if we only had H2Connection it wouldn’t be able to identify a given producer with the stream it holds. By extension, IConsumer cannot consume multiple producers at once. For this reason, we need an interface between H2Connection and H2Stream that is similar to ITransport and IConsumer, but more featureful. Basically, H2Stream is a thin shim between a producer and H2Connection that adds a stream ID to a few function calls.
This is basically a good pattern. It exposes a hard-to-screw-up interface to the next layer up, because you can't forget to include a (mandatory) stream ID. I've implemented several multiplexing things that work more or less like this.
Another option could also be having a t.w.s.NewSite (with that name hopefully obviously being a straw man) so that Site can simply be deprecated in favor of the new thing. Making Site itself be able to accommodate the new stuff would be nice but is definitely not mandatory.
These two protocols each build an IGoodRequest, which is very similar to IRequest but has a deliverBody method. The consumer of this (whether IResource or some other thing). These objects, if they want to consume a stream, register a protocol via deliverBody. At this point, H2Connection (via H2Stream) provides itself as the transport to that protocol, and calls deliverBody when chunks of data are received.
This sounds great. One thing to maybe watch out for: what if nobody calls deliverBody? This can sometimes be a little annoying in client code, to debug why a channel is never closed. Having a nice error in this case would be a cherry on top.
When the object receiving the request is ready to send a response, it calls…something (sendResponse?) and provides an object implementing a server IResponse. The code in the H2Stream/H2Connection sends the headers, then calls deliverBody on the IResponse, passing H2Connection (again via H2Stream) as the protocol that gets called. In this world, H2Stream actually would need to implement IProtocol as well as ITransport.
A minor bit of critique here: the Single Responsibility Principle <https://en.wikipedia.org/wiki/Single_responsibility_principle> dictates that we ought not to have H2Stream literally implement both IProtocol and ITransport; rather, we should have an _H2StreamProtocol and an _H2StreamTransport, since the thing talking to the IProtocol implementation really ought to be wholly distinct from the thing talking to the ITransport implementation, and this kind of duality makes it very easy for users - especially programmers new to Twisted - to get confused. As Nathaniel Manista and Augie Fackler put it in The Talk <https://www.youtube.com/watch?v=3MNVP9-hglc>, we want to express ourselves "structurally", if you only want application code to talk to the transport implementation and it's an error to talk to the protocol implementation, pass only the transport implementation.
Is my understand of that correct? If so, I think this design can work: essentially, H2Stream becomes the weird intermediary layer that appears as both a transport and a protocol to the request/response layer. Underneath the covers it mostly delegates to H2Connection, which implements a slightly weirdo version of IConsumer (and in fact IProducer) that can only be consumed by H2Stream.
I don't quite get why it needs to be slightly weirdo (hopefully IPushProducer is sufficient?) but yes, this all sounds right to me.
"super-gross but vaguely do-able" is what we're shooting for in the compatibility layer :).
All of this approach sounds reasonable modulo some careful thinking about how exactly we tie this in with the old paradigm. I’m particularly concerned about H2Channel, which I suspect many applications may know a great deal about. Changing its interface is likely to be slightly tricky, but we’ll see how it goes.
It might be useful to think about a parent interface, IHTTPChannel with all the least-common-denominator stuff on it, and sub-interfaces IHTTP1_1Channel and IHTTP2_0Channel which each derive from that and provide additional version-specific stuff. I don't have enough protocol-specific knowledge to hand in short-term memory to comment on what that functionality might be though. -glyph
![](https://secure.gravatar.com/avatar/e81edff3af564b86f4c9d780a8023299.jpg?s=120&d=mm&r=g)
When the object receiving the request is ready to send a response, it calls…something (sendResponse?) and provides an object implementing a server IResponse. The code in the H2Stream/H2Connection sends the headers, then calls deliverBody on the IResponse, passing H2Connection (again via H2Stream) as the protocol that gets called. In this world, H2Stream actually would need to implement IProtocol as well as ITransport.
Probably the reponse is provided via a deferred returned from the function that received the request. Following the design of twisted.web.client.Request, the request would take something like an IBodyProducer that the stream would have write to it (Probably indirectly: see twisted.web._newclient.Request._writeToChuncked/_writeToContentLength). Tom
![](https://secure.gravatar.com/avatar/e81edff3af564b86f4c9d780a8023299.jpg?s=120&d=mm&r=g)
After having written the following comments, I realized that my thoughts are only about the high-level interface of Site/Resource. I think those are the interfaces most users care about, so what it makes most sense to think deeply about having a painless transiftion for. I suspect that if people are using lower-level interfaces, they are probably willing to make more invasive changes in order to be able to take advantage of HTTP/2; in particular, since they likely have resaon to use HTTP/2 specific features.
It is probably possible to implement something like you suggest, without having to change the model too much. As I understand it, the big impediment to properly handling streaming requests is `.content` (and some related convenience things like `.args`), and the fact that both `.getChild` and `.render` are called after `.content` is populated. It is probably possible to address those issues without changing the shape of those interfaces (even if we change the names). I know #288 had at least two suggestions on how to do that. One was to have a marker interface, to indicate that a given Resource wants new behavior, where `.content` isn't populated, and the other was to have new methods that have new behavior, which default to slurping up everything and calling the old functions. On the other hand, there might be other stuff that wants cleaning up, that having a break would be better at addressing; replacing `NOT_DONE_YET` with deferreds comes to mind.
We should then move to an API that is much more like the one used by Go: specifically, that by default all requests/responses are streamed. Request objects (and, logically, any other object that handles requests/responses, such as Resource) should be extended to have a chunkReceived method that can be overridden by users. If a user chooses not to override that method, the default implementation would continue to do what is done now (save to a buffer). Once the request/response is complete (marked by receipt of a zero-length chunk, or a frame with END_STREAM set, or when the remaining content-length is 0), request/responseComplete would be called. For users that did not override chunkReceived can now safely access the content buffer: other users can do whatever they see fit. We’d also update requestReceived to ensure that it’s called when all the *headers* are received, rather than waiting for the body.
I haven't thought about this deeply, but my first thought, is that it would be reasonable to mirror how the client handles streaming responses. `Agent.request` returns `Response` as soon as the headers have been received. To get the body of the response, you call `Response.deliverBody` which takes an `IProtocol` that will receive the body. There is also a helper `readBody` that wraps that and returns a deferred that fires with body, once it has been received (and treq also has ``collect`` that wraps that and calls a function with the bits of the data).
A similar approach should be taken with sending data: we should assume that users want to chunk it if they do not provide a content-length. An extreme position to take (and I do) is that this should be sufficiently easy that most users actually *accidentally* end up chunking their data: that is, we do not provide special helpers to set content-length, instead just checking whether that’s a header users actually send, and if they don’t we chunk the data.
Regarding sending data, this is already what we do (at least as long as the client is speaking HTTP/1.1). Tom
![](https://secure.gravatar.com/avatar/fa611b50fabecfe29749b66ec52f46a0.jpg?s=120&d=mm&r=g)
What do people think of this approach?
It sounds fine to me. I think this issue in Twisted is currently the blocker (or at least *one of* the blockers) for Tahoe-LAFS ticket #113, #320, and #1032. It is also a blocker for the much more ambitious Tahoe-LAFS tickets #1288 and #1851. Regards, Zooko https://tahoe-lafs.org/trac/tahoe-lafs/ticket/113# command-line: do things in an incremental fashion and accept stdin as input https://tahoe-lafs.org/trac/tahoe-lafs/ticket/320# add streaming (on-line) upload to HTTP interface https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1032# Display active HTTP upload operations on the status page https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1288# support streaming uploads in uploader https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1851# new immutable file upload protocol: streaming, fewer round-trips, quota-respecting
![](https://secure.gravatar.com/avatar/869963bdf99b541c9f0bbfb04b0320f1.jpg?s=120&d=mm&r=g)
On Sat, 14 Nov 2015 at 17:45 Cory Benfield <cory@lukasa.co.uk> wrote:
As far as I know, this is exactly what is done in the existing HTTP/1.x implementation: if a Content-Length header field is not set before data is written to the request object (as it is called), chunked transfer encoding is automatically used for the response. This behaviour is required by HTTP/1.1.
![](https://secure.gravatar.com/avatar/214c694acb154321379cbc58dc91528c.jpg?s=120&d=mm&r=g)
On 15 Nov 2015, at 10:18, Tobias Oberstein <tobias.oberstein@tavendo.de> wrote:
How does flow-control work with the Go API? How does user code processing chunks received unleash backpressure onto the sender?
One caveat: I am not an expert in what Go is doing here, more a casual user. However, my understanding is that Go code blocks will not handle chunks in parallel, or provide a new chunk until the previous one has been processed (essentially, the ‘chunk handling’ function is processed synchronously, once for each chunk), which means that it does not read more data from the socket. This exerts TCP level back pressure, and can be adjusted to exert HTTP/2 back-pressure if we’re sufficiently careful about it. Generally speaking in Twisted we could achieve this too, by careful use of Deferreds.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Nov 17, 2015, at 2:49 AM, Cory Benfield <cory@lukasa.co.uk> wrote:
Generally speaking in Twisted we could achieve this too, by careful use of Deferreds.
I have a bigger reply coming; however, I should note that this is basically what web2 did with its IStream interface. While this has a big advantage over the status quo (i.e. "not solving the problem") the _way_ it solved the problem ended up being both (A) error prone, and (B) slow. It's the sort of slowness that PyPy doesn't even help mitigate, because all the callbacks are heap-allocated, and... to be honest I don't even fully understand it, but I have experimentally verified that it's still pretty slow :-). This is more or less the reason that Tubes exists, although more on how we might proceed either with or without tube in another message... -glyph
![](https://secure.gravatar.com/avatar/214c694acb154321379cbc58dc91528c.jpg?s=120&d=mm&r=g)
On 18 Nov 2015, at 05:55, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Does HTTP2 have choke/unchoke notifications on individual streams? Ultimately this does resolve to TCP backpressure, though…
Not in such a binary form, no. The connection as a whole and each stream maintain independent flow control windows. This allows for pressure to be exerted on the sender to slow down, by allowing the flow control window to drop to zero. This means that there is some Twisted-level buffering, because we do have to get that data out of the socket and to queue at the application, but the amount of data to buffer is strictly bounded. Thus, if our application moves slowly, the remote side should be passively notified to slow down by the lack of window updates: we should only send those window updates once the application has actually taken some data from us. Cory
![](https://secure.gravatar.com/avatar/c4c5daddd52a9c703809ac68780cfe6a.jpg?s=120&d=mm&r=g)
Hi, sounds like a good idea. One topic: What should happen if the sender sends malformed data (e.g. content-length: 100 and sending 200 bytes body). I would usually want to know if the sender is broken instead of getting a responseComplete() that claims everything is fine. We simply subclassed twisted.web.server.Request and injected the needed parts so we could get the data we wanted in an incremental way and set some socket options (SO_RCVBUF) for multi-gigabyte file uploads. So we took 'gotLength()' as a signal that headers are fully received and also took 'write()', 'processingFailed()' to handle cleanup. Basically we replace the self.content stream object with our own version that does not buffer everything in memory. We tried to do the 100-Continue stuff too, but it simply didn't work with the Python stdlib httplib code (it just ignores those more or less). So we either just terminate the connection forcefully (reset) or read the full request and send it to /dev/null. Michael -- Michael Schlenker Senior Software Engineer CONTACT Software GmbH Tel.: +49 (421) 20153-80 Wiener Straße 1-3 Fax: +49 (421) 20153-41 28359 Bremen E-Mail: michael.schlenker@contact-software.com http://www.contact-software.com/ Registered office: Bremen, Germany Managing directors: Karl Heinz Zachries, Ralf Holtgrefe Court of register: Amtsgericht Bremen HRB 1321
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
This should definitely be an error. While this isn't totally irrelevant to the design of the new server API - the implementation should surely take this kind of error-checking into account, and the old one didn't so well - it's a bit of a distraction since it's not an essential difference. -glyph
![](https://secure.gravatar.com/avatar/c1cd4fcd6c951797d9567954cf2ccca0.jpg?s=120&d=mm&r=g)
Hi Cory,
I think it is worth noting that some of these issues were solved by twisted.web2, to the point that there were even adapters created that handled some of the differences between the existing web resource model and the new model in web2. Of course, web2 did not succeed in replacing web, due in part to concerns over the introduction of a new flow-control mechanism (IStream and company). Today though, we have https://github.com/twisted/tubes, which is being developed outside of twisted proper, but could perhaps be leveraged to overcome some of the previous concerns about introducing a mechanism different from IProducer/IConsumer. I mention this because I think there is work within the web2 codebase that might prove useful in solving some of the issues being discussed here, in a way that would benefit twisted, and its users. Thanks for reading, L. Daniel Burr
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Hooray!
As part of my dive into twisted.web, I noticed something that surprised me: it seems to have no support for ‘streaming’ request bodies. By this I mean that the Request.requestReceived() method is not actually called until the complete request body has been received. This is a somewhat unexpected limitation for Twisted: why should I have to wait until the entire body has been uploaded to start doing things with it?
This is exactly what <https://twistedmatrix.com/trac/ticket/288> is about (as you note).
I'm very glad to hear you say this, because: (A) we should absolutely make progress on this ticket now that there is some impetus to do so, but, (B) we should absolutely NOT let this new API block any purely protocol-level HTTP2 work that needs to proceed (I say "protocol level" as a contrast to "feature level" because we could totally implement a thing that speaks HTTP 2 against the current API, although it might be awkward to expose the advantageous parts of the protocol as API features until we do some cleanup)
We have considered it a bad way to do things for a long time. There have been several attempts to rewrite it (Nevow's request model, web2) but none of them have really been the comprehensive re-design we need.
We should then move to an API that is much more like the one used by Go: specifically, that by default all requests/responses are streamed. Request objects (and, logically, any other object that handles requests/responses, such as Resource) should be extended to have a chunkReceived method that can be overridden by users.
No. Let me elaborate :-). First of all, this is already the case (sort of). twisted.web.server.Request inherits from twisted.web.http.Request; as you can see in http.Request, there is already a method called handleContentChunk, which is called by HTTPChannel. By overriding this method, you can already handle request data streaming in from off the wire. This is one of the reasons that #288 is so interminable: using Twisted's public API, today, you can totally write an HTTP-based application that happily streams data from the wire. The only problem is that this API does not propagate up to Resource objects, because Resource objects can expect the (still, as of this writing, undocumented) "content" attribute to have been filled out in getChild. Mostly, they don't, actually! But it's impossible to tell if they might in the general case. You can (and many applications have) just broken the technical compatibility contract with Resource, and written a subclass of twisted.web.server.Site that has a custom requestFactory method that returns a 'streamed' resource. So, if we're already doing this, why "no"? Superclasses with overridable methods are a terrible mechanism for exposing extensibility. These are used extensively throughout Twisted, the older the API the more inheritance it uses. Newer code, you may notice, is generally written much more in a pattern of delegation to formal interfaces. So we have tried to learn our lesson here.
If a user chooses not to override that method, the default implementation would continue to do what is done now (save to a buffer). Once the request/response is complete (marked by receipt of a zero-length chunk, or a frame with END_STREAM set, or when the remaining content-length is 0),
For what it's worth, I hope that all of these will be exposed as the same event to applications, since the fact that these differ on the wire is entirely an implementation detail?
request/responseComplete would be called. For users that did not override chunkReceived can now safely access the content buffer: other users can do whatever they see fit. We’d also update requestReceived to ensure that it’s called when all the *headers* are received, rather than waiting for the body.
Again, this is very similar to what already happens, at the layer of the HTTP protocol. The question is, how do you indicate that you're delegating to a Resource object which may expect the .content attribute to already be populated during .getChild?
A similar approach should be taken with sending data: we should assume that users want to chunk it if they do not provide a content-length. An extreme position to take (and I do) is that this should be sufficiently easy that most users actually *accidentally* end up chunking their data: that is, we do not provide special helpers to set content-length, instead just checking whether that’s a header users actually send, and if they don’t we chunk the data.
request.write() already basically does this, I think? Here, at least, we have lots of opportunity to make the implementation do smarter things (better error checking regarding content-length, for example) without changing the interface at all.
This logic would make it much easier to work with HTTP/2 *and* with WebSockets, requiring substantially less special-case code to handle the WebSocket upgrade (when the headers are complete, we can spot the upgrade easily).
What do people think of this approach?
So I think you're roughly on the right track but there are probably some Twisted-level gaps to fill in. I've already gestured in the direction of Tubes (as have others) and it's something to think about. But before we get to that, let's talk about a much more basic deficiency in the API: although there's an "IRequest", and an "IResource", there's no such thing as an "IResponse". Instead, "IRequest" stands in for both the request and the response, because you write directly to a request (implicitly filling out its response as you do so). Luckily we have an existing interface that might point the way to a better solution, both for requests and responses: specifically, the client IResponse: https://twistedmatrix.com/documents/15.4.0/api/twisted.web.iweb.IResponse.ht.... This interface is actually pretty close to what we want for a server IResponse as well. Perhaps even identical. Its static data is all exposed as attributes which can be relatively simply inspected, and the way it delivers a streaming response is that it delivers its body to an IProtocol implementation (via .deliverBody(aProtocol)). This is not quite as graceful as having a .bodyFount() method that returns an IFount from the tubes package; however, the tubes package is still not exactly mature software, so we may not want to block on depending on it. Importantly though, this delivers all the events you need as a primitive for interfacing with such a high-level interface; it would definitely be better to add this sort of interface Real Soon Now, because then the tubes package could simply have a method, responseToFount (which it will need anyway to work with Agent) that calls deliverBody internally. This works as a primitive because you have all the hooks you need for flow-control. This protocol receives, to its 'makeConnection' method, an ITransport which can provide the IProducer https://twistedmatrix.com/documents/15.4.0/api/twisted.internet.interfaces.I... and IConsumer https://twistedmatrix.com/documents/15.4.0/api/twisted.internet.interfaces.I... interfaces for flow-control. It receives dataReceived to tell it a chunk has arrived and connectionLost to tell it the stream has terminated. Unfortunately the client IRequest https://twistedmatrix.com/documents/15.4.0/api/twisted.web.iweb.IClientReque... isn't quite as useful (although its relative minimalism should be an inspiration to anyone designing a next-generation IRequest more than the current IRequest's sprawling kitchen-sink aesthetic). However, IResponse.deliverBody could be applied to IGoodRequest as well. If we have a very similar-to-IResponse shaped IRequest object, say with 'method', 'uri' and 'headers', and then a 'deliverBody' that delivers the request body in much the same way, we could get a gracefully structured streaming request with works with a lot of existing code within Twisted. Then the question is: what to do with IResource? Right now the flow of processing a request is, roughly: -> wait for full request to arrive -> have HTTPChannel fill out IRequest object -> look at request.site.resource for the root *-> call getChildWithDefault repeatedly, mutating "cursor" state on the IRequest as you move (specifically: "prepath" and "postpath" attributes) -> eventually reach the leaf Resource, or one with 'isLeaf' set on it, and delegate producing the response to that resource *-> call resource.render(request) -> examine the return value; if it's bytes, deliver them and close the connection; NOT_DONE_YET, just leave the connection open, Instead, I think a good flow would be: -> receive method/headers from request -> recurse down from request.site.resource, calling something like nevow's or web2's locateChild, but not modifying 'request' at each stage; instead, pass a "cursor" object - perhaps indeed just a twisted.python.url.URL - indicating where we are in the resource traversal hierarchy. the reason the request.prepath and request.postpath attributes exist is mainly for Resource objects to be able to orient themselves within a resource tree and generate links. also, it probably bears some explanation; the signature of the current "get the next resource" call is resource.getChildWithDefault(onePathSegment, request) -> resource. This is somewhat limiting as it requires you to consume only an individual path segment at a time, which can be highly awkward for implementing sites that have a URL structure that is, for example, /YYYY/MM/DD/HH/MM/index.html. Instead, locateChild took the entire remaining path, and returned a 2-tuple of a resource, and the _still_ remaining path. So for the above, you could do: def locateChild(self, request, path): y, m, d = path[:3] return ymdresource(y, m, d), path[3:] that 2-tuple instructs the traversal machinery, "keep going". One alternative that we toyed with for this was to make consuming the path destructive, since that made it a lot easier to tell what resource you were "looking at": def locateChild(self, request, path): y, m, d = path.consume(3) return ymdresource(y, m, d) Either of these approaches also let you implement 'isLeaf' attribute without special support from the framework; you simply return leaf(), () or path.consume(path.length) -> finally, call .responseForRequest(request) -> IResponse on the final Resource and deliver the IResponse to the network. The way compatibility could be achieved here is to write a wrapper that would implement .responseForRequest to first collect the entire body, then synthesize a gross old-style-IRequest-like object out of the combination of that body and the other information about the resource, then call .getChildWithDefault on it a few times, then call the old-style .render_GET, et. al. The IResponse returned from this compatibility .responseForRequest would wrap up calls like request.write and turn them into write() calls. This is long and increasingly rambly, so I should probably stop now, send it, and get your feedback. Does the strategy I'm proposing make sense? I'm sure I'm leaving a ton out so feel free to ask for clarification. Hopefully I didn't leave too many incomplete sentences in the middle. -glyph
![](https://secure.gravatar.com/avatar/214c694acb154321379cbc58dc91528c.jpg?s=120&d=mm&r=g)
On 18 Nov 2015, at 12:18, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Sorry about the delay in responding to this, but I wanted to make sure I knew at least a bit about what I was talking about before I responded!
So, I think in general this is interesting. One of the big difficulties I’m having right now is that I’m trying to combine this “streaming HTTP” work with the implementation of HTTP/2, which means that I need to keep the HTTP/2 work in mind whenever I talk about this *and* update the HTTP/2 design in response to decisions we make here. This means I’ve got quite a lot of balls in the air right now, and I am confident I’ll drop quite a few. One thing I’m deliberately not doing here is considering Tubes, in part because I’m extremely concerned about backward compatibility, and want the HTTP/2 work to function in the same environment. Unfortunately, this means this conversation is blending into the HTTP/2 one, so I’m going to hijack this thread and bring in some concrete discussion of what I’m working on with the HTTP/2 stuff. I was having a conversation about the HTTP/2 architecture on #twisted-dev yesterday, which has led towards my current working approach for HTTP/2, which will be to have two underlying objects. We’ll have H2Connection, which implements IProtocol, and H2Stream, which implements ITransport. These two objects will be *extremely* tightly coupled: H2Stream cannot meaningfully run over an arbitrary transport mechanism, and knows a great deal about how H2Connections work. The reason we need to take this approach is because IConsumer doesn’t allow for us to have correlators, so even if we only had H2Connection it wouldn’t be able to identify a given producer with the stream it holds. By extension, IConsumer cannot consume multiple producers at once. For this reason, we need an interface between H2Connection and H2Stream that is similar to ITransport and IConsumer, but more featureful. Basically, H2Stream is a thin shim between a producer and H2Connection that adds a stream ID to a few function calls.
Just let me clarify how this is expected to work. Somewhere we have a t.w.s.Site, which builds some kind of HTTP protocol (currently HTTPChannel, in future some object that can transparently swap between HTTPChannel and H2Connection) when connections are received. These two protocols each build an IGoodRequest, which is very similar to IRequest but has a deliverBody method. The consumer of this (whether IResource or some other thing). These objects, if they want to consume a stream, register a protocol via deliverBody. At this point, H2Connection (via H2Stream) provides itself as the transport to that protocol, and calls deliverBody when chunks of data are received. When the object receiving the request is ready to send a response, it calls…something (sendResponse?) and provides an object implementing a server IResponse. The code in the H2Stream/H2Connection sends the headers, then calls deliverBody on the IResponse, passing H2Connection (again via H2Stream) as the protocol that gets called. In this world, H2Stream actually would need to implement IProtocol as well as ITransport. Is my understand of that correct? If so, I think this design can work: essentially, H2Stream becomes the weird intermediary layer that appears as both a transport and a protocol to the request/response layer. Underneath the covers it mostly delegates to H2Connection, which implements a slightly weirdo version of IConsumer (and in fact IProducer) that can only be consumed by H2Stream.
[snip long discussion of how to write locateChild] Agreed that these proposed approaches would work well. I have no concrete feedback on them, they seem good to me.
-> finally, call .responseForRequest(request) -> IResponse on the final Resource and deliver the IResponse to the network.
The way compatibility could be achieved here is to write a wrapper that would implement .responseForRequest to first collect the entire body, then synthesize a gross old-style-IRequest-like object out of the combination of that body and the other information about the resource, then call .getChildWithDefault on it a few times, then call the old-style .render_GET, et. al. The IResponse returned from this compatibility .responseForRequest would wrap up calls like request.write and turn them into write() calls.
This seems super-gross but vaguely do-able, and we’ll need to write it in order to get the new H2Connection/H2Stream objects working with the old paradigm anyway. All of this approach sounds reasonable modulo some careful thinking about how exactly we tie this in with the old paradigm. I’m particularly concerned about H2Channel, which I suspect many applications may know a great deal about. Changing its interface is likely to be slightly tricky, but we’ll see how it goes. Cory
![](https://secure.gravatar.com/avatar/214c694acb154321379cbc58dc91528c.jpg?s=120&d=mm&r=g)
For those interested in how this is progressing, there’s a draft patch available showing some of the proposed direction of this work, at #7460. I’d like as much review as possible, so please weigh in: https://twistedmatrix.com/trac/ticket/7460 Cory
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Clearly this is a challenging topic that requires lots of thought on the part of each interlocutor, and may require long rounds of consideration before each reply. No need to apologize.
Hijack away. I think we should be primarily concerned with getting HTTP/2 integrated for the moment. The reason this raises so many concerns related to the streaming stuff is that the internal implementation of HTTP/2 ought to be more amenable to pulling apart to fit into an actually good interface to the HTTP protocol. I think that twisted._threads points in a promising direction for this sort of work: let's make the old, crappy HTTP APIs work as-is, but with a new, private implementation that is better-factored but not fully documented. We have the old interface as a proof-of-concept, so the new stuff needs to at least be good enough to be an internal implementation detail for that; we don't have to commit to a new public API to land it, and hopefully with some minor edits we can just make it public as the "good" interface (and then backport HTTP/1.1 over it, since we will probably be dealing with legacy HTTP/1.1 clients and servers until we're all dead).
I was having a conversation about the HTTP/2 architecture on #twisted-dev yesterday, which has led towards my current working approach for HTTP/2, which will be to have two underlying objects. We’ll have H2Connection, which implements IProtocol, and H2Stream, which implements ITransport. These two objects will be *extremely* tightly coupled: H2Stream cannot meaningfully run over an arbitrary transport mechanism, and knows a great deal about how H2Connections work.
This seems good, except for the "extreme" tight coupling. IProtocol and ITransport aren't that tightly coupled. Why do H2Stream and H2Connection need to be?
The reason we need to take this approach is because IConsumer doesn’t allow for us to have correlators, so even if we only had H2Connection it wouldn’t be able to identify a given producer with the stream it holds. By extension, IConsumer cannot consume multiple producers at once. For this reason, we need an interface between H2Connection and H2Stream that is similar to ITransport and IConsumer, but more featureful. Basically, H2Stream is a thin shim between a producer and H2Connection that adds a stream ID to a few function calls.
This is basically a good pattern. It exposes a hard-to-screw-up interface to the next layer up, because you can't forget to include a (mandatory) stream ID. I've implemented several multiplexing things that work more or less like this.
Another option could also be having a t.w.s.NewSite (with that name hopefully obviously being a straw man) so that Site can simply be deprecated in favor of the new thing. Making Site itself be able to accommodate the new stuff would be nice but is definitely not mandatory.
These two protocols each build an IGoodRequest, which is very similar to IRequest but has a deliverBody method. The consumer of this (whether IResource or some other thing). These objects, if they want to consume a stream, register a protocol via deliverBody. At this point, H2Connection (via H2Stream) provides itself as the transport to that protocol, and calls deliverBody when chunks of data are received.
This sounds great. One thing to maybe watch out for: what if nobody calls deliverBody? This can sometimes be a little annoying in client code, to debug why a channel is never closed. Having a nice error in this case would be a cherry on top.
When the object receiving the request is ready to send a response, it calls…something (sendResponse?) and provides an object implementing a server IResponse. The code in the H2Stream/H2Connection sends the headers, then calls deliverBody on the IResponse, passing H2Connection (again via H2Stream) as the protocol that gets called. In this world, H2Stream actually would need to implement IProtocol as well as ITransport.
A minor bit of critique here: the Single Responsibility Principle <https://en.wikipedia.org/wiki/Single_responsibility_principle> dictates that we ought not to have H2Stream literally implement both IProtocol and ITransport; rather, we should have an _H2StreamProtocol and an _H2StreamTransport, since the thing talking to the IProtocol implementation really ought to be wholly distinct from the thing talking to the ITransport implementation, and this kind of duality makes it very easy for users - especially programmers new to Twisted - to get confused. As Nathaniel Manista and Augie Fackler put it in The Talk <https://www.youtube.com/watch?v=3MNVP9-hglc>, we want to express ourselves "structurally", if you only want application code to talk to the transport implementation and it's an error to talk to the protocol implementation, pass only the transport implementation.
Is my understand of that correct? If so, I think this design can work: essentially, H2Stream becomes the weird intermediary layer that appears as both a transport and a protocol to the request/response layer. Underneath the covers it mostly delegates to H2Connection, which implements a slightly weirdo version of IConsumer (and in fact IProducer) that can only be consumed by H2Stream.
I don't quite get why it needs to be slightly weirdo (hopefully IPushProducer is sufficient?) but yes, this all sounds right to me.
"super-gross but vaguely do-able" is what we're shooting for in the compatibility layer :).
All of this approach sounds reasonable modulo some careful thinking about how exactly we tie this in with the old paradigm. I’m particularly concerned about H2Channel, which I suspect many applications may know a great deal about. Changing its interface is likely to be slightly tricky, but we’ll see how it goes.
It might be useful to think about a parent interface, IHTTPChannel with all the least-common-denominator stuff on it, and sub-interfaces IHTTP1_1Channel and IHTTP2_0Channel which each derive from that and provide additional version-specific stuff. I don't have enough protocol-specific knowledge to hand in short-term memory to comment on what that functionality might be though. -glyph
![](https://secure.gravatar.com/avatar/e81edff3af564b86f4c9d780a8023299.jpg?s=120&d=mm&r=g)
When the object receiving the request is ready to send a response, it calls…something (sendResponse?) and provides an object implementing a server IResponse. The code in the H2Stream/H2Connection sends the headers, then calls deliverBody on the IResponse, passing H2Connection (again via H2Stream) as the protocol that gets called. In this world, H2Stream actually would need to implement IProtocol as well as ITransport.
Probably the reponse is provided via a deferred returned from the function that received the request. Following the design of twisted.web.client.Request, the request would take something like an IBodyProducer that the stream would have write to it (Probably indirectly: see twisted.web._newclient.Request._writeToChuncked/_writeToContentLength). Tom
![](https://secure.gravatar.com/avatar/e81edff3af564b86f4c9d780a8023299.jpg?s=120&d=mm&r=g)
After having written the following comments, I realized that my thoughts are only about the high-level interface of Site/Resource. I think those are the interfaces most users care about, so what it makes most sense to think deeply about having a painless transiftion for. I suspect that if people are using lower-level interfaces, they are probably willing to make more invasive changes in order to be able to take advantage of HTTP/2; in particular, since they likely have resaon to use HTTP/2 specific features.
It is probably possible to implement something like you suggest, without having to change the model too much. As I understand it, the big impediment to properly handling streaming requests is `.content` (and some related convenience things like `.args`), and the fact that both `.getChild` and `.render` are called after `.content` is populated. It is probably possible to address those issues without changing the shape of those interfaces (even if we change the names). I know #288 had at least two suggestions on how to do that. One was to have a marker interface, to indicate that a given Resource wants new behavior, where `.content` isn't populated, and the other was to have new methods that have new behavior, which default to slurping up everything and calling the old functions. On the other hand, there might be other stuff that wants cleaning up, that having a break would be better at addressing; replacing `NOT_DONE_YET` with deferreds comes to mind.
We should then move to an API that is much more like the one used by Go: specifically, that by default all requests/responses are streamed. Request objects (and, logically, any other object that handles requests/responses, such as Resource) should be extended to have a chunkReceived method that can be overridden by users. If a user chooses not to override that method, the default implementation would continue to do what is done now (save to a buffer). Once the request/response is complete (marked by receipt of a zero-length chunk, or a frame with END_STREAM set, or when the remaining content-length is 0), request/responseComplete would be called. For users that did not override chunkReceived can now safely access the content buffer: other users can do whatever they see fit. We’d also update requestReceived to ensure that it’s called when all the *headers* are received, rather than waiting for the body.
I haven't thought about this deeply, but my first thought, is that it would be reasonable to mirror how the client handles streaming responses. `Agent.request` returns `Response` as soon as the headers have been received. To get the body of the response, you call `Response.deliverBody` which takes an `IProtocol` that will receive the body. There is also a helper `readBody` that wraps that and returns a deferred that fires with body, once it has been received (and treq also has ``collect`` that wraps that and calls a function with the bits of the data).
A similar approach should be taken with sending data: we should assume that users want to chunk it if they do not provide a content-length. An extreme position to take (and I do) is that this should be sufficiently easy that most users actually *accidentally* end up chunking their data: that is, we do not provide special helpers to set content-length, instead just checking whether that’s a header users actually send, and if they don’t we chunk the data.
Regarding sending data, this is already what we do (at least as long as the client is speaking HTTP/1.1). Tom
participants (8)
-
Cory Benfield
-
Glyph Lefkowitz
-
Louis D. Burr
-
Michael Schlenker
-
Tobias Oberstein
-
Tom Prince
-
Tristan Seligmann
-
Zooko Wilcox-O'Hearn