[Twisted-Python] web client FileBodyProducer - transfer encoding
![](https://secure.gravatar.com/avatar/a07b6d294d96567cea15ae7ff55df0be.jpg?s=120&d=mm&r=g)
hey all - quick question. Trying to understand FileBodyProducer as it pertains to POST. json_body = FileBodyProducer(StringIO(json.dumps({'key': 'value'}))) agent.request("POST", ~*uri*, Headers({'User-Agent': ['AkamaiTest']}), json_body) Does the agent chunk the POST body? Is this controllable? Any insight appreciated!
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
You probably don't want to try to control this. I believe that proxies are within their rights to mess around with chunk boundaries and re-buffer things, so you don't have any strong guarantees that chunk sizes will be preserved. Why is it that you want to control chunking in the first place? -glyph
![](https://secure.gravatar.com/avatar/a07b6d294d96567cea15ae7ff55df0be.jpg?s=120&d=mm&r=g)
Dealing with older apache, ruby, passenger setup. Hoping to mimic behavior and show OPS that chunked encoding isn't working correctly. With python requests (http://docs.python-requests.org/en/master/) all POST calls work correctly, but twisted requests are failing. I'm thinking (and hoping) it's failing because twisted is chunking the post body. When I send the requests to a twisted endpoint like below the requests are successful and the response is identical. from twisted.web import server, resource from twisted.internet import reactor class Simple(resource.Resource): isLeaf = True def render_POST(self, request): print request.content.getvalue() return request.content.getvalue() site = server.Site(Simple()) reactor.listenTCP(8080, site) reactor.run() On Mon, Jan 30, 2017 at 2:39 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On Mon, Jan 30, 2017 at 7:44 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Well... It *is* sort of tuneable, as you implied earlier. If you pass an IBodyProducer with a non-None length, Agent will send a Content-Length header - not use chunked Transfer-Encoding. FileBodyProducer doesn't know how to determine the length of a StringIO, so you get chunked with this example. If you write the JSON to a regular file and FileBodyProducer(open(the file)) you'll get a Content-Length request. You could also write a new (trivial) IBodyProducer that does know how to compute the length of a StringIO. The documentation doesn't exactly spell this out - but the only reason `length` is part of the interface is to be able to generate the Content-Length header. What would the ticket be? Expanded documentation to make this behavior into an explicit guarantee of the interface? A new toggle somewhere to force Agent.request into one mode or the other (regardless of the performance consequences)? And that stuff you said about proxies before was true ... So even if you can control this in Agent, you're still not *guaranteed* the server will see what you send. Jean-Paul
![](https://secure.gravatar.com/avatar/a07b6d294d96567cea15ae7ff55df0be.jpg?s=120&d=mm&r=g)
Thanks - that's exactly what I was looking for. *But note* when using FileBodyProducer(StringIO(json.dumps(~blah))) -- I still see a content-length in the request header as it's received by twisted. Am I correct to assume the request is agnostic...meaning it's shaped the same for twisted as it is for apache. Headers({'host': ['localhost:8080'], 'connection': ['close'], 'content-length': ['671'], 'user-agent': ['PassengerTest']}) from twisted.web import server, resource from twisted.internet import reactor class Simple(resource.Resource): isLeaf = True def render_POST(self, request): print request.requestHeaders return request.content.getvalue() site = server.Site(Simple()) reactor.listenTCP(8080, site) reactor.run() On Mon, Jan 30, 2017 at 4:59 PM, Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Not quite true: FileBodyProducer does know how to compute the length of a StringIO, via seek():
Rather than "regardless" of the consequences, perhaps just a maximum body size. If we made it an explicit guarantee in Agent, perhaps the interface change would not be in '.request' (which, as a formal interface used both inside and outside of Twisted, is fairly fixed), but rather a new `NonChunkedBodyProducer` concrete class that would implement IBodyProducer in terms of another IBodyProducer which either does or doesn't have a `length`. (Or a function that does same, always returning its argument if `length` is already set...)
And that stuff you said about proxies before was true ... So even if you can control this in Agent, you're still not guaranteed the server will see what you send.
Thanks - that's exactly what I was looking for.
But note when using FileBodyProducer(StringIO(json.dumps(~blah))) -- I still see a content-length in the request header as it's received by twisted. Am I correct to assume the request is agnostic...meaning it's shaped the same for twisted as it is for apache.
It is shaped the same. The reason you're seeing the error is due to the issue I pointed out above.
Headers({'host': ['localhost:8080'], 'connection': ['close'], 'content-length': ['671'], 'user-agent': ['PassengerTest']})
-glyph
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
You probably don't want to try to control this. I believe that proxies are within their rights to mess around with chunk boundaries and re-buffer things, so you don't have any strong guarantees that chunk sizes will be preserved. Why is it that you want to control chunking in the first place? -glyph
![](https://secure.gravatar.com/avatar/a07b6d294d96567cea15ae7ff55df0be.jpg?s=120&d=mm&r=g)
Dealing with older apache, ruby, passenger setup. Hoping to mimic behavior and show OPS that chunked encoding isn't working correctly. With python requests (http://docs.python-requests.org/en/master/) all POST calls work correctly, but twisted requests are failing. I'm thinking (and hoping) it's failing because twisted is chunking the post body. When I send the requests to a twisted endpoint like below the requests are successful and the response is identical. from twisted.web import server, resource from twisted.internet import reactor class Simple(resource.Resource): isLeaf = True def render_POST(self, request): print request.content.getvalue() return request.content.getvalue() site = server.Site(Simple()) reactor.listenTCP(8080, site) reactor.run() On Mon, Jan 30, 2017 at 2:39 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/607cfd4a5b41fe6c886c978128b9c03e.jpg?s=120&d=mm&r=g)
On Mon, Jan 30, 2017 at 7:44 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
Well... It *is* sort of tuneable, as you implied earlier. If you pass an IBodyProducer with a non-None length, Agent will send a Content-Length header - not use chunked Transfer-Encoding. FileBodyProducer doesn't know how to determine the length of a StringIO, so you get chunked with this example. If you write the JSON to a regular file and FileBodyProducer(open(the file)) you'll get a Content-Length request. You could also write a new (trivial) IBodyProducer that does know how to compute the length of a StringIO. The documentation doesn't exactly spell this out - but the only reason `length` is part of the interface is to be able to generate the Content-Length header. What would the ticket be? Expanded documentation to make this behavior into an explicit guarantee of the interface? A new toggle somewhere to force Agent.request into one mode or the other (regardless of the performance consequences)? And that stuff you said about proxies before was true ... So even if you can control this in Agent, you're still not *guaranteed* the server will see what you send. Jean-Paul
![](https://secure.gravatar.com/avatar/a07b6d294d96567cea15ae7ff55df0be.jpg?s=120&d=mm&r=g)
Thanks - that's exactly what I was looking for. *But note* when using FileBodyProducer(StringIO(json.dumps(~blah))) -- I still see a content-length in the request header as it's received by twisted. Am I correct to assume the request is agnostic...meaning it's shaped the same for twisted as it is for apache. Headers({'host': ['localhost:8080'], 'connection': ['close'], 'content-length': ['671'], 'user-agent': ['PassengerTest']}) from twisted.web import server, resource from twisted.internet import reactor class Simple(resource.Resource): isLeaf = True def render_POST(self, request): print request.requestHeaders return request.content.getvalue() site = server.Site(Simple()) reactor.listenTCP(8080, site) reactor.run() On Mon, Jan 30, 2017 at 4:59 PM, Jean-Paul Calderone < exarkun@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Not quite true: FileBodyProducer does know how to compute the length of a StringIO, via seek():
Rather than "regardless" of the consequences, perhaps just a maximum body size. If we made it an explicit guarantee in Agent, perhaps the interface change would not be in '.request' (which, as a formal interface used both inside and outside of Twisted, is fairly fixed), but rather a new `NonChunkedBodyProducer` concrete class that would implement IBodyProducer in terms of another IBodyProducer which either does or doesn't have a `length`. (Or a function that does same, always returning its argument if `length` is already set...)
And that stuff you said about proxies before was true ... So even if you can control this in Agent, you're still not guaranteed the server will see what you send.
Thanks - that's exactly what I was looking for.
But note when using FileBodyProducer(StringIO(json.dumps(~blah))) -- I still see a content-length in the request header as it's received by twisted. Am I correct to assume the request is agnostic...meaning it's shaped the same for twisted as it is for apache.
It is shaped the same. The reason you're seeing the error is due to the issue I pointed out above.
Headers({'host': ['localhost:8080'], 'connection': ['close'], 'content-length': ['671'], 'user-agent': ['PassengerTest']})
-glyph
participants (3)
-
Glyph Lefkowitz
-
Jean-Paul Calderone
-
Kevin Mcintyre