Hi all, I've been using my HTTP/1.1 httplib for quite a while now. It seems quite stable in everything that I've tried. I need to port over the recent changes for SSL stuff, but it is ready to go. The library contains a deprecated, backwards-compat class (HTTP), and its replacement: HTTPConnection. Responses are now handled through HTTPResponse class. Doc for the new class will also be needed, but (of course) the old doc still applies. Any comments on the module and/or its inclusion into 1.6? Cheers, -g -- Greg Stein, http://www.lyra.org/
Doesn't Jeremy have one too? I would suggest putting the new classes in a new module, e.g. httplib2.py, and leaving the old httplib unchanged. BTW, where can we look at this code? --Guido van Rossum (home page: http://www.python.org/~guido/)
On Wed, 16 Feb 2000, Guido van Rossum wrote:
Yes, we talked about it at IPC7. As I recall, Jeremy wasn't happy with the result... it was request/response based with a lot of helper classes. My httplib is connection-oriented (like the current httplib), and I think Jeremy had said that was good (empirically, after his prototype). But that's just let my memory... Jeremy?
I would suggest putting the new classes in a new module, e.g. httplib2.py, and leaving the old httplib unchanged.
Why? The compatibility class exports the same attributes, the same methods, the same constructor, etc. Creating a new module is needless duplication and confusing for users ("which one do I use?" "how is that one different?" "do I have to switch modules to use HTTP/1.1?") A single module that reuses code is going to be much more maintainable than two, possibly diverging, modules. I also feel having one module will help to gently prod people towards using HTTP/1.1 and the new HTTPConnection class (which I think is *subjectively* a nice goal).
BTW, where can we look at this code?
Oops. Sorry about that... As usual, it is with the rest of my Python code at http://www.lyra.org/greg/python/ (the davlib is there, too) Cheers, -g -- Greg Stein, http://www.lyra.org/
"GS" == Greg Stein <gstein@lyra.org> writes:
Doesn't Jeremy have one too?
GS> Yes, we talked about it at IPC7. As I recall, Jeremy wasn't GS> happy with the result... it was request/response based with a GS> lot of helper classes. My httplib is connection-oriented (like GS> the current httplib), and I think Jeremy had said that was good GS> (empirically, after his prototype). GS> But that's just let my memory... Jeremy? Sounds right to me. I haven't worked on http code in years. I seem to recall liking the basic architecture, but not the implementation. That said, I have no objections to including your modifications: It preserves the current interface and improves its support for http/1.1. I still think the httplib interface could be improved, but I don't have any time to work on it. The one thing I don't see how to do with the new code is pipeline requests. Is that possible? It's not a big deal if pipelining isn't support, since the current implementation doesn't support it at all. Jeremy
On Thu, 17 Feb 2000, Jeremy Hylton wrote:
Pipelining is very important... it is once of the big advances in HTTP/1.1 (well, the advance is really the "chunked" transfer encoding which *allows* pipelining to occur more often). To do pipelining, just keep feeding requests into the thing: ------------------------- conn = httplib.HTTPConnection('www.python.org') conn.request('GET', '/') code, msg, response = conn.getreply() print response.read() conn.request('GET', '/sigs/') code, msg, response = conn.getreply() print response.read() ------------------------- Note that the code requires you to consume the response before issuing another request (it will raise an error otherwise). The connection object will connect automatically connect to the host. It will keep the connection open per HTTP/1.1 guidelines (or close it when told to). If an "broken pipe" occurs during the sending of the request (the other end closed the socket), then it will close, reconnect, and re-issue the request (just once). The broken pipe typically occurs if you wait too long between requests. The client can force a connect or a close thru the connection object API. In addition to the simple request() method above, there is the standard putrequest, putheader, endheader style. Cheers, -g -- Greg Stein, http://www.lyra.org/
Greg Stein writes:
You should be able to pipeline requests even if responses don't use chunking; if I parse the first HTML page while still receiving, I should be able to request non-cached images or other dependencies before the first HTML is completely received. This allows the server to locate the resources while still waiting for me finish pulling the first response off the network. It improves the possibility of scheduling resource location at the server (possibly important if there's a dynamic backend) as well as avoiding establishing new TCP connections. (Yes, some of this can be done with keep-alive stuff, but I don't know that httplib supports that at all.) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives
On Thu, 17 Feb 2000, Fred L. Drake, Jr. wrote:
Not in all cases. You can pipeline only if the server sends a Content-Length: header or uses chunking. There are a few other conditions (look at the httplib.HTTPResponse constructor in my httplib), but the Content-Length/chunking is the major point.
This would be nice, but my httplib currently enforces consumption before a new request. I do not recall the specific reasons why, but I imagine it would be possible to change it. [ today: you could certainly use a second connection object to improve client response, with the corresponding impact on the server ]
(Yes, some of this can be done with keep-alive stuff, but I don't know that httplib supports that at all.)
My httplib deals with the keep-alive header, when applicable (you are supposed to use Connection: in HTTP/1.1; it only looks at keep-alive if you aren't using 1.1). Cheers, -g -- Greg Stein, http://www.lyra.org/
This sounds good enough for me! I endorse it for inclusion as long as their is accompanying documentation <0.25 wink>. Jeremy
On Thu, 17 Feb 2000, Jeremy Hylton wrote:
This sounds good enough for me! I endorse it for inclusion as long as their is accompanying documentation <0.25 wink>.
Nowadays, I consider it a prerequisite, along with an updated test suite. In the past, I've submitted stuff and the doc stuff has hung over my head for a long while... I'd rather get it done up front :-) Cheers, -g -- Greg Stein, http://www.lyra.org/
Doesn't Jeremy have one too? I would suggest putting the new classes in a new module, e.g. httplib2.py, and leaving the old httplib unchanged. BTW, where can we look at this code? --Guido van Rossum (home page: http://www.python.org/~guido/)
On Wed, 16 Feb 2000, Guido van Rossum wrote:
Yes, we talked about it at IPC7. As I recall, Jeremy wasn't happy with the result... it was request/response based with a lot of helper classes. My httplib is connection-oriented (like the current httplib), and I think Jeremy had said that was good (empirically, after his prototype). But that's just let my memory... Jeremy?
I would suggest putting the new classes in a new module, e.g. httplib2.py, and leaving the old httplib unchanged.
Why? The compatibility class exports the same attributes, the same methods, the same constructor, etc. Creating a new module is needless duplication and confusing for users ("which one do I use?" "how is that one different?" "do I have to switch modules to use HTTP/1.1?") A single module that reuses code is going to be much more maintainable than two, possibly diverging, modules. I also feel having one module will help to gently prod people towards using HTTP/1.1 and the new HTTPConnection class (which I think is *subjectively* a nice goal).
BTW, where can we look at this code?
Oops. Sorry about that... As usual, it is with the rest of my Python code at http://www.lyra.org/greg/python/ (the davlib is there, too) Cheers, -g -- Greg Stein, http://www.lyra.org/
"GS" == Greg Stein <gstein@lyra.org> writes:
Doesn't Jeremy have one too?
GS> Yes, we talked about it at IPC7. As I recall, Jeremy wasn't GS> happy with the result... it was request/response based with a GS> lot of helper classes. My httplib is connection-oriented (like GS> the current httplib), and I think Jeremy had said that was good GS> (empirically, after his prototype). GS> But that's just let my memory... Jeremy? Sounds right to me. I haven't worked on http code in years. I seem to recall liking the basic architecture, but not the implementation. That said, I have no objections to including your modifications: It preserves the current interface and improves its support for http/1.1. I still think the httplib interface could be improved, but I don't have any time to work on it. The one thing I don't see how to do with the new code is pipeline requests. Is that possible? It's not a big deal if pipelining isn't support, since the current implementation doesn't support it at all. Jeremy
On Thu, 17 Feb 2000, Jeremy Hylton wrote:
Pipelining is very important... it is once of the big advances in HTTP/1.1 (well, the advance is really the "chunked" transfer encoding which *allows* pipelining to occur more often). To do pipelining, just keep feeding requests into the thing: ------------------------- conn = httplib.HTTPConnection('www.python.org') conn.request('GET', '/') code, msg, response = conn.getreply() print response.read() conn.request('GET', '/sigs/') code, msg, response = conn.getreply() print response.read() ------------------------- Note that the code requires you to consume the response before issuing another request (it will raise an error otherwise). The connection object will connect automatically connect to the host. It will keep the connection open per HTTP/1.1 guidelines (or close it when told to). If an "broken pipe" occurs during the sending of the request (the other end closed the socket), then it will close, reconnect, and re-issue the request (just once). The broken pipe typically occurs if you wait too long between requests. The client can force a connect or a close thru the connection object API. In addition to the simple request() method above, there is the standard putrequest, putheader, endheader style. Cheers, -g -- Greg Stein, http://www.lyra.org/
Greg Stein writes:
You should be able to pipeline requests even if responses don't use chunking; if I parse the first HTML page while still receiving, I should be able to request non-cached images or other dependencies before the first HTML is completely received. This allows the server to locate the resources while still waiting for me finish pulling the first response off the network. It improves the possibility of scheduling resource location at the server (possibly important if there's a dynamic backend) as well as avoiding establishing new TCP connections. (Yes, some of this can be done with keep-alive stuff, but I don't know that httplib supports that at all.) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> Corporation for National Research Initiatives
On Thu, 17 Feb 2000, Fred L. Drake, Jr. wrote:
Not in all cases. You can pipeline only if the server sends a Content-Length: header or uses chunking. There are a few other conditions (look at the httplib.HTTPResponse constructor in my httplib), but the Content-Length/chunking is the major point.
This would be nice, but my httplib currently enforces consumption before a new request. I do not recall the specific reasons why, but I imagine it would be possible to change it. [ today: you could certainly use a second connection object to improve client response, with the corresponding impact on the server ]
(Yes, some of this can be done with keep-alive stuff, but I don't know that httplib supports that at all.)
My httplib deals with the keep-alive header, when applicable (you are supposed to use Connection: in HTTP/1.1; it only looks at keep-alive if you aren't using 1.1). Cheers, -g -- Greg Stein, http://www.lyra.org/
This sounds good enough for me! I endorse it for inclusion as long as their is accompanying documentation <0.25 wink>. Jeremy
On Thu, 17 Feb 2000, Jeremy Hylton wrote:
This sounds good enough for me! I endorse it for inclusion as long as their is accompanying documentation <0.25 wink>.
Nowadays, I consider it a prerequisite, along with an updated test suite. In the past, I've submitted stuff and the doc stuff has hung over my head for a long while... I'd rather get it done up front :-) Cheers, -g -- Greg Stein, http://www.lyra.org/
participants (4)
-
Fred L. Drake, Jr. -
Greg Stein -
Guido van Rossum -
Jeremy Hylton