What do people think of this approach?

It sounds fine to me. I think this issue in Twisted is currently the blocker (or at least *one of* the blockers) for Tahoe-LAFS ticket #113, #320, and #1032. It is also a blocker for the much more ambitious Tahoe-LAFS tickets #1288 and #1851. Regards, Zooko https://tahoe-lafs.org/trac/tahoe-lafs/ticket/113# command-line: do things in an incremental fashion and accept stdin as input https://tahoe-lafs.org/trac/tahoe-lafs/ticket/320# add streaming (on-line) upload to HTTP interface https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1032# Display active HTTP upload operations on the status page https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1288# support streaming uploads in uploader https://tahoe-lafs.org/trac/tahoe-lafs/ticket/1851# new immutable file upload protocol: streaming, fewer round-trips, quota-respecting

We should then move to an API that is much more like the one used by Go: specifically, that by default all requests/responses are streamed. Request objects (and, logically, any other object that handles requests/responses, such as Resource) should be extended to have a chunkReceived method that can be overridden by users.

How does flow-control work with the Go API? How does user code processing chunks received unleash backpressure onto the sender? Cheers, /Tobias

Hi Cory,

On Nov 13, 2015, at 6:36 AM, Cory Benfield wrote:

Folks,

# Problem Statement

Thanks for your feedback on my HTTP/2 questions. I’ve started work implementing a spike of a HTTP/2 protocol for twisted.web. I’m aiming to have something that works in at least some cases by the end of the day.

As part of my dive into twisted.web, I noticed something that surprised me: it seems to have no support for ‘streaming’ request bodies. By this I mean that the Request.requestReceived() method is not actually called until the complete request body has been received. This is a somewhat unexpected limitation for Twisted: why should I have to wait until the entire body has been uploaded to start doing things with it?

This problem is thrown into sharp relief with HTTP/2, which essentially always chunks the body, even if a content-length is provided. This means that it is now very easy to receive data in delimited chunks, which an implementation may want to have semantic meaning. However, the request is unable to access this data in this way. It also makes it impossible to use a HTTP/2 request/response pair as a long-running communication channel, as we cannot safely call requestReceived until the response is terminated (which also terminates the HTTP/2 stream).

Adi pointed me at a related issue, #6928[0], which itself points at what appears to be an issue tracking exactly this request. That issue is issue #288[1], which is 12 years old(!). This has clearly been a pain point for quite some time.

I think it is worth noting that some of these issues were solved by twisted.web2, to the point that there were even adapters created that handled some of the differences between the existing web resource model and the new model in web2. Of course, web2 did not succeed in replacing web, due in part to concerns over the introduction of a new flow-control mechanism (IStream and company). Today though, we have https://github.com/twisted/tubes, which is being developed outside of twisted proper, but could perhaps be leveraged to overcome some of the previous concerns about introducing a mechanism different from IProducer/IConsumer. I mention this because I think there is work within the web2 codebase that might prove useful in solving some of the issues being discussed here, in a way that would benefit twisted, and its users. Thanks for reading, L. Daniel Burr

On 15 Nov 2015, at 10:18, Tobias Oberstein wrote:

How does flow-control work with the Go API? How does user code processing chunks received unleash backpressure onto the sender?

One caveat: I am not an expert in what Go is doing here, more a casual user. However, my understanding is that Go code blocks will not handle chunks in parallel, or provide a new chunk until the previous one has been processed (essentially, the ‘chunk handling’ function is processed synchronously, once for each chunk), which means that it does not read more data from the socket. This exerts TCP level back pressure, and can be adjusted to exert HTTP/2 back-pressure if we’re sufficiently careful about it. Generally speaking in Twisted we could achieve this too, by careful use of Deferreds.

Am 17.11.2015 um 11:49 schrieb Cory Benfield:

...

On 15 Nov 2015, at 10:18, Tobias Oberstein wrote:

How does flow-control work with the Go API? How does user code processing chunks received unleash backpressure onto the sender?

One caveat: I am not an expert in what Go is doing here, more a casual user. However, my understanding is that Go code blocks will not handle chunks in parallel, or provide a new chunk until the previous one has been processed (essentially, the ‘chunk handling’ function is processed synchronously, once for each chunk), which means that it does not read more data from the socket. This exerts TCP level back pressure, and can be adjusted to exert HTTP/2 back-pressure if we’re sufficiently careful about it.

Thing is, HTTP2 multiplexes multiple streams over 1 TCP, and each stream should be flow-controlled individually - and hence can't rely on TCP backpressure for that. /Tobias

On 18 Nov 2015, at 05:55, Glyph Lefkowitz wrote:

Does HTTP2 have choke/unchoke notifications on individual streams? Ultimately this does resolve to TCP backpressure, though…

Not in such a binary form, no. The connection as a whole and each stream maintain independent flow control windows. This allows for pressure to be exerted on the sender to slow down, by allowing the flow control window to drop to zero. This means that there is some Twisted-level buffering, because we do have to get that data out of the socket and to queue at the application, but the amount of data to buffer is strictly bounded. Thus, if our application moves slowly, the remote side should be passively notified to slow down by the lack of window updates: we should only send those window updates once the application has actually taken some data from us. Cory

On Nov 18, 2015, at 2:28 AM, Cory Benfield wrote:

...

On 18 Nov 2015, at 05:55, Glyph Lefkowitz wrote:

Does HTTP2 have choke/unchoke notifications on individual streams? Ultimately this does resolve to TCP backpressure, though…

Not in such a binary form, no. The connection as a whole and each stream maintain independent flow control windows. This allows for pressure to be exerted on the sender to slow down, by allowing the flow control window to drop to zero. This means that there is some Twisted-level buffering, because we do have to get that data out of the socket and to queue at the application, but the amount of data to buffer is strictly bounded. Thus, if our application moves slowly, the remote side should be passively notified to slow down by the lack of window updates: we should only send those window updates once the application has actually taken some data from us.

So the receiver sends explicit "unchoke" messages to increase the window, and the "choke" message is implicit? -glyph

On 18 Nov 2015, at 12:18, Glyph Lefkowitz wrote:

Sorry about the delay in responding to this, but I wanted to make sure I knew at least a bit about what I was talking about before I responded!

...

What do people think of this approach?

So I think you're roughly on the right track but there are probably some Twisted-level gaps to fill in.

I've already gestured in the direction of Tubes (as have others) and it's something to think about. But before we get to that, let's talk about a much more basic deficiency in the API: although there's an "IRequest", and an "IResource", there's no such thing as an "IResponse". Instead, "IRequest" stands in for both the request and the response, because you write directly to a request (implicitly filling out its response as you do so).

So, I think in general this is interesting. One of the big difficulties I’m having right now is that I’m trying to combine this “streaming HTTP” work with the implementation of HTTP/2, which means that I need to keep the HTTP/2 work in mind whenever I talk about this *and* update the HTTP/2 design in response to decisions we make here. This means I’ve got quite a lot of balls in the air right now, and I am confident I’ll drop quite a few. One thing I’m deliberately not doing here is considering Tubes, in part because I’m extremely concerned about backward compatibility, and want the HTTP/2 work to function in the same environment. Unfortunately, this means this conversation is blending into the HTTP/2 one, so I’m going to hijack this thread and bring in some concrete discussion of what I’m working on with the HTTP/2 stuff. I was having a conversation about the HTTP/2 architecture on #twisted-dev yesterday, which has led towards my current working approach for HTTP/2, which will be to have two underlying objects. We’ll have H2Connection, which implements IProtocol, and H2Stream, which implements ITransport. These two objects will be *extremely* tightly coupled: H2Stream cannot meaningfully run over an arbitrary transport mechanism, and knows a great deal about how H2Connections work. The reason we need to take this approach is because IConsumer doesn’t allow for us to have correlators, so even if we only had H2Connection it wouldn’t be able to identify a given producer with the stream it holds. By extension, IConsumer cannot consume multiple producers at once. For this reason, we need an interface between H2Connection and H2Stream that is similar to ITransport and IConsumer, but more featureful. Basically, H2Stream is a thin shim between a producer and H2Connection that adds a stream ID to a few function calls.

Luckily we have an existing interface that might point the way to a better solution, both for requests and responses: specifically, the client IResponse: https://twistedmatrix.com/documents/15.4.0/api/twisted.web.iweb.IResponse.ht....

This interface is actually pretty close to what we want for a server IResponse as well. Perhaps even identical. Its static data is all exposed as attributes which can be relatively simply inspected, and the way it delivers a streaming response is that it delivers its body to an IProtocol implementation (via .deliverBody(aProtocol)). This is not quite as graceful as having a .bodyFount() method that returns an IFount from the tubes package; however, the tubes package is still not exactly mature software, so we may not want to block on depending on it. Importantly though, this delivers all the events you need as a primitive for interfacing with such a high-level interface; it would definitely be better to add this sort of interface Real Soon Now, because then the tubes package could simply have a method, responseToFount (which it will need anyway to work with Agent) that calls deliverBody internally.

This works as a primitive because you have all the hooks you need for flow-control. This protocol receives, to its 'makeConnection' method, an ITransport which can provide the IProducer https://twistedmatrix.com/documents/15.4.0/api/twisted.internet.interfaces.I... and IConsumer https://twistedmatrix.com/documents/15.4.0/api/twisted.internet.interfaces.I... interfaces for flow-control. It receives dataReceived to tell it a chunk has arrived and connectionLost to tell it the stream has terminated.

Just let me clarify how this is expected to work. Somewhere we have a t.w.s.Site, which builds some kind of HTTP protocol (currently HTTPChannel, in future some object that can transparently swap between HTTPChannel and H2Connection) when connections are received. These two protocols each build an IGoodRequest, which is very similar to IRequest but has a deliverBody method. The consumer of this (whether IResource or some other thing). These objects, if they want to consume a stream, register a protocol via deliverBody. At this point, H2Connection (via H2Stream) provides itself as the transport to that protocol, and calls deliverBody when chunks of data are received. When the object receiving the request is ready to send a response, it calls…something (sendResponse?) and provides an object implementing a server IResponse. The code in the H2Stream/H2Connection sends the headers, then calls deliverBody on the IResponse, passing H2Connection (again via H2Stream) as the protocol that gets called. In this world, H2Stream actually would need to implement IProtocol as well as ITransport. Is my understand of that correct? If so, I think this design can work: essentially, H2Stream becomes the weird intermediary layer that appears as both a transport and a protocol to the request/response layer. Underneath the covers it mostly delegates to H2Connection, which implements a slightly weirdo version of IConsumer (and in fact IProducer) that can only be consumed by H2Stream.

Unfortunately the client IRequest https://twistedmatrix.com/documents/15.4.0/api/twisted.web.iweb.IClientReque... isn't quite as useful (although its relative minimalism should be an inspiration to anyone designing a next-generation IRequest more than the current IRequest's sprawling kitchen-sink aesthetic). However, IResponse.deliverBody could be applied to IGoodRequest as well. If we have a very similar-to-IResponse shaped IRequest object, say with 'method', 'uri' and 'headers', and then a 'deliverBody' that delivers the request body in much the same way, we could get a gracefully structured streaming request with works with a lot of existing code within Twisted.

Then the question is: what to do with IResource?

Right now the flow of processing a request is, roughly:

-> wait for full request to arrive -> have HTTPChannel fill out IRequest object -> look at request.site.resource for the root *-> call getChildWithDefault repeatedly, mutating "cursor" state on the IRequest as you move (specifically: "prepath" and "postpath" attributes) -> eventually reach the leaf Resource, or one with 'isLeaf' set on it, and delegate producing the response to that resource *-> call resource.render(request) -> examine the return value; if it's bytes, deliver them and close the connection; NOT_DONE_YET, just leave the connection open,

Instead, I think a good flow would be:

[snip long discussion of how to write locateChild] Agreed that these proposed approaches would work well. I have no concrete feedback on them, they seem good to me.

-> finally, call .responseForRequest(request) -> IResponse on the final Resource and deliver the IResponse to the network.

The way compatibility could be achieved here is to write a wrapper that would implement .responseForRequest to first collect the entire body, then synthesize a gross old-style-IRequest-like object out of the combination of that body and the other information about the resource, then call .getChildWithDefault on it a few times, then call the old-style .render_GET, et. al. The IResponse returned from this compatibility .responseForRequest would wrap up calls like request.write and turn them into write() calls.

This seems super-gross but vaguely do-able, and we’ll need to write it in order to get the new H2Connection/H2Stream objects working with the old paradigm anyway. All of this approach sounds reasonable modulo some careful thinking about how exactly we tie this in with the old paradigm. I’m particularly concerned about H2Channel, which I suspect many applications may know a great deal about. Changing its interface is likely to be slightly tricky, but we’ll see how it goes. Cory

On Nov 19, 2015, at 3:50 AM, Cory Benfield wrote:

...

On 18 Nov 2015, at 12:18, Glyph Lefkowitz wrote:

Sorry about the delay in responding to this, but I wanted to make sure I knew at least a bit about what I was talking about before I responded!

Clearly this is a challenging topic that requires lots of thought on the part of each interlocutor, and may require long rounds of consideration before each reply. No need to apologize.

...

...

What do people think of this approach?

So I think you're roughly on the right track but there are probably some Twisted-level gaps to fill in.

I've already gestured in the direction of Tubes (as have others) and it's something to think about. But before we get to that, let's talk about a much more basic deficiency in the API: although there's an "IRequest", and an "IResource", there's no such thing as an "IResponse". Instead, "IRequest" stands in for both the request and the response, because you write directly to a request (implicitly filling out its response as you do so).

So, I think in general this is interesting. One of the big difficulties I’m having right now is that I’m trying to combine this “streaming HTTP” work with the implementation of HTTP/2, which means that I need to keep the HTTP/2 work in mind whenever I talk about this *and* update the HTTP/2 design in response to decisions we make here. This means I’ve got quite a lot of balls in the air right now, and I am confident I’ll drop quite a few. One thing I’m deliberately not doing here is considering Tubes, in part because I’m extremely concerned about backward compatibility, and want the HTTP/2 work to function in the same environment.

Unfortunately, this means this conversation is blending into the HTTP/2 one, so I’m going to hijack this thread and bring in some concrete discussion of what I’m working on with the HTTP/2 stuff.

Hijack away. I think we should be primarily concerned with getting HTTP/2 integrated for the moment. The reason this raises so many concerns related to the streaming stuff is that the internal implementation of HTTP/2 ought to be more amenable to pulling apart to fit into an actually good interface to the HTTP protocol. I think that twisted._threads points in a promising direction for this sort of work: let's make the old, crappy HTTP APIs work as-is, but with a new, private implementation that is better-factored but not fully documented. We have the old interface as a proof-of-concept, so the new stuff needs to at least be good enough to be an internal implementation detail for that; we don't have to commit to a new public API to land it, and hopefully with some minor edits we can just make it public as the "good" interface (and then backport HTTP/1.1 over it, since we will probably be dealing with legacy HTTP/1.1 clients and servers until we're all dead).

I was having a conversation about the HTTP/2 architecture on #twisted-dev yesterday, which has led towards my current working approach for HTTP/2, which will be to have two underlying objects. We’ll have H2Connection, which implements IProtocol, and H2Stream, which implements ITransport. These two objects will be *extremely* tightly coupled: H2Stream cannot meaningfully run over an arbitrary transport mechanism, and knows a great deal about how H2Connections work.

This seems good, except for the "extreme" tight coupling. IProtocol and ITransport aren't that tightly coupled. Why do H2Stream and H2Connection need to be?

The reason we need to take this approach is because IConsumer doesn’t allow for us to have correlators, so even if we only had H2Connection it wouldn’t be able to identify a given producer with the stream it holds. By extension, IConsumer cannot consume multiple producers at once. For this reason, we need an interface between H2Connection and H2Stream that is similar to ITransport and IConsumer, but more featureful. Basically, H2Stream is a thin shim between a producer and H2Connection that adds a stream ID to a few function calls.

This is basically a good pattern. It exposes a hard-to-screw-up interface to the next layer up, because you can't forget to include a (mandatory) stream ID. I've implemented several multiplexing things that work more or less like this.

...

Luckily we have an existing interface that might point the way to a better solution, both for requests and responses: specifically, the client IResponse: https://twistedmatrix.com/documents/15.4.0/api/twisted.web.iweb.IResponse.ht....

This interface is actually pretty close to what we want for a server IResponse as well. Perhaps even identical. Its static data is all exposed as attributes which can be relatively simply inspected, and the way it delivers a streaming response is that it delivers its body to an IProtocol implementation (via .deliverBody(aProtocol)). This is not quite as graceful as having a .bodyFount() method that returns an IFount from the tubes package; however, the tubes package is still not exactly mature software, so we may not want to block on depending on it. Importantly though, this delivers all the events you need as a primitive for interfacing with such a high-level interface; it would definitely be better to add this sort of interface Real Soon Now, because then the tubes package could simply have a method, responseToFount (which it will need anyway to work with Agent) that calls deliverBody internally.

This works as a primitive because you have all the hooks you need for flow-control. This protocol receives, to its 'makeConnection' method, an ITransport which can provide the IProducer https://twistedmatrix.com/documents/15.4.0/api/twisted.internet.interfaces.I... and IConsumer https://twistedmatrix.com/documents/15.4.0/api/twisted.internet.interfaces.I... interfaces for flow-control. It receives dataReceived to tell it a chunk has arrived and connectionLost to tell it the stream has terminated.

Just let me clarify how this is expected to work. Somewhere we have a t.w.s.Site, which builds some kind of HTTP protocol (currently HTTPChannel, in future some object that can transparently swap between HTTPChannel and H2Connection) when connections are received.

Another option could also be having a t.w.s.NewSite (with that name hopefully obviously being a straw man) so that Site can simply be deprecated in favor of the new thing. Making Site itself be able to accommodate the new stuff would be nice but is definitely not mandatory.

These two protocols each build an IGoodRequest, which is very similar to IRequest but has a deliverBody method. The consumer of this (whether IResource or some other thing). These objects, if they want to consume a stream, register a protocol via deliverBody. At this point, H2Connection (via H2Stream) provides itself as the transport to that protocol, and calls deliverBody when chunks of data are received.

This sounds great. One thing to maybe watch out for: what if nobody calls deliverBody? This can sometimes be a little annoying in client code, to debug why a channel is never closed. Having a nice error in this case would be a cherry on top.

When the object receiving the request is ready to send a response, it calls…something (sendResponse?) and provides an object implementing a server IResponse. The code in the H2Stream/H2Connection sends the headers, then calls deliverBody on the IResponse, passing H2Connection (again via H2Stream) as the protocol that gets called. In this world, H2Stream actually would need to implement IProtocol as well as ITransport.

A minor bit of critique here: the Single Responsibility Principle https://en.wikipedia.org/wiki/Single_responsibility_principle dictates that we ought not to have H2Stream literally implement both IProtocol and ITransport; rather, we should have an _H2StreamProtocol and an _H2StreamTransport, since the thing talking to the IProtocol implementation really ought to be wholly distinct from the thing talking to the ITransport implementation, and this kind of duality makes it very easy for users - especially programmers new to Twisted - to get confused. As Nathaniel Manista and Augie Fackler put it in The Talk https://www.youtube.com/watch?v=3MNVP9-hglc, we want to express ourselves "structurally", if you only want application code to talk to the transport implementation and it's an error to talk to the protocol implementation, pass only the transport implementation.

Is my understand of that correct? If so, I think this design can work: essentially, H2Stream becomes the weird intermediary layer that appears as both a transport and a protocol to the request/response layer. Underneath the covers it mostly delegates to H2Connection, which implements a slightly weirdo version of IConsumer (and in fact IProducer) that can only be consumed by H2Stream.

I don't quite get why it needs to be slightly weirdo (hopefully IPushProducer is sufficient?) but yes, this all sounds right to me.

...

Unfortunately the client IRequest https://twistedmatrix.com/documents/15.4.0/api/twisted.web.iweb.IClientReque... isn't quite as useful (although its relative minimalism should be an inspiration to anyone designing a next-generation IRequest more than the current IRequest's sprawling kitchen-sink aesthetic). However, IResponse.deliverBody could be applied to IGoodRequest as well. If we have a very similar-to-IResponse shaped IRequest object, say with 'method', 'uri' and 'headers', and then a 'deliverBody' that delivers the request body in much the same way, we could get a gracefully structured streaming request with works with a lot of existing code within Twisted.

Then the question is: what to do with IResource?

Right now the flow of processing a request is, roughly:

-> wait for full request to arrive -> have HTTPChannel fill out IRequest object -> look at request.site.resource for the root *-> call getChildWithDefault repeatedly, mutating "cursor" state on the IRequest as you move (specifically: "prepath" and "postpath" attributes) -> eventually reach the leaf Resource, or one with 'isLeaf' set on it, and delegate producing the response to that resource *-> call resource.render(request) -> examine the return value; if it's bytes, deliver them and close the connection; NOT_DONE_YET, just leave the connection open,

Instead, I think a good flow would be:

[snip long discussion of how to write locateChild]

Agreed that these proposed approaches would work well. I have no concrete feedback on them, they seem good to me.

...

-> finally, call .responseForRequest(request) -> IResponse on the final Resource and deliver the IResponse to the network.

The way compatibility could be achieved here is to write a wrapper that would implement .responseForRequest to first collect the entire body, then synthesize a gross old-style-IRequest-like object out of the combination of that body and the other information about the resource, then call .getChildWithDefault on it a few times, then call the old-style .render_GET, et. al. The IResponse returned from this compatibility .responseForRequest would wrap up calls like request.write and turn them into write() calls.

This seems super-gross but vaguely do-able, and we’ll need to write it in order to get the new H2Connection/H2Stream objects working with the old paradigm anyway.

"super-gross but vaguely do-able" is what we're shooting for in the compatibility layer :).

All of this approach sounds reasonable modulo some careful thinking about how exactly we tie this in with the old paradigm. I’m particularly concerned about H2Channel, which I suspect many applications may know a great deal about. Changing its interface is likely to be slightly tricky, but we’ll see how it goes.

It might be useful to think about a parent interface, IHTTPChannel with all the least-common-denominator stuff on it, and sub-interfaces IHTTP1_1Channel and IHTTP2_0Channel which each derive from that and provide additional version-specific stuff. I don't have enough protocol-specific knowledge to hand in short-term memory to comment on what that functionality might be though. -glyph

After having written the following comments, I realized that my thoughts are only about the high-level interface of Site/Resource. I think those are the interfaces most users care about, so what it makes most sense to think deeply about having a painless transiftion for. I suspect that if people are using lower-level interfaces, they are probably willing to make more invasive changes in order to be able to take advantage of HTTP/2; in particular, since they likely have resaon to use HTTP/2 specific features.

My proposal is to deprecate the current Request/Resource model. It currently functions and should continue to function, but as of this point we should consider it a bad way to do things, and we should push people to move to a fully asynchronous model.

It is probably possible to implement something like you suggest, without having to change the model too much. As I understand it, the big impediment to properly handling streaming requests is `.content` (and some related convenience things like `.args`), and the fact that both `.getChild` and `.render` are called after `.content` is populated. It is probably possible to address those issues without changing the shape of those interfaces (even if we change the names). I know #288 had at least two suggestions on how to do that. One was to have a marker interface, to indicate that a given Resource wants new behavior, where `.content` isn't populated, and the other was to have new methods that have new behavior, which default to slurping up everything and calling the old functions. On the other hand, there might be other stuff that wants cleaning up, that having a break would be better at addressing; replacing `NOT_DONE_YET` with deferreds comes to mind.

We should then move to an API that is much more like the one used by Go: specifically, that by default all requests/responses are streamed. Request objects (and, logically, any other object that handles requests/responses, such as Resource) should be extended to have a chunkReceived method that can be overridden by users. If a user chooses not to override that method, the default implementation would continue to do what is done now (save to a buffer). Once the request/response is complete (marked by receipt of a zero-length chunk, or a frame with END_STREAM set, or when the remaining content-length is 0), request/responseComplete would be called. For users that did not override chunkReceived can now safely access the content buffer: other users can do whatever they see fit. We’d also update requestReceived to ensure that it’s called when all the *headers* are received, rather than waiting for the body.

I haven't thought about this deeply, but my first thought, is that it would be reasonable to mirror how the client handles streaming responses. `Agent.request` returns `Response` as soon as the headers have been received. To get the body of the response, you call `Response.deliverBody` which takes an `IProtocol` that will receive the body. There is also a helper `readBody` that wraps that and returns a deferred that fires with body, once it has been received (and treq also has ``collect`` that wraps that and calls a function with the bits of the data).

A similar approach should be taken with sending data: we should assume that users want to chunk it if they do not provide a content-length. An extreme position to take (and I do) is that this should be sufficiently easy that most users actually *accidentally* end up chunking their data: that is, we do not provide special helpers to set content-length, instead just checking whether that’s a header users actually send, and if they don’t we chunk the data.

Regarding sending data, this is already what we do (at least as long as the client is speaking HTTP/1.1). Tom