[Twisted-Python] treq // Agent and Content-Length

I’m making a small proxy server in Twisted which is supposed to essentially "mask" the original destination of an URL. Specifically it does the following: 1. Receives an HTTP request on /<hmac>/<url encoded url> 2. Verifies that the hmac in the URL matches the url decoded url 3. Makes an HTTP request to the url decoded url 4. Verify the response 4a. Verify that the *on the wire* size is not greater than a set number of bytes, prior to downloading the body. 4b. Verify that the content-type is acceptable prior to downloading the body. 5. Download the body of the response without modifying it (No Gzip decode) 6. Return the unmodified downloaded response as the resposne to the HTTP request in 1, as well as copying over several Response Headers. This should be able to pass through ``Accept-Encoding`` from the request in #1 to the request made in #3. I've had a problem getting 4a to happen, no matter what I've tried if the Content-Encoding is set to gzip it appears that the length of the response is unknown. Additionally the "connection headers" are split out from the normal headers, so I cannot seem to locate the original ``Content-Length`` header to simply check that. I got frustrated and implemented it using requests and that has worked OK. It however uses deferToThread and I would like to get back to using the built in tooling. So my question is, how can I make Agent/treq do this? The requests version of the code is here: https://gist.github.com/dstufft/7c8aac898de0ad359675 The treq versions of the code is here: https://gist.github.com/dstufft/0dffb12201d235f35bc7 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
participants (1)
-
Donald Stufft