"Guido van Rossum" wrote:

I recently needed to access an HTTP URL with a timeout. I ended up monkey-patching httplib.HTTPConnection so that the connect() method has an optional second paramer, timeout, defaulting to None; if not None, a call to settimeout() is added right after successful creation of the socket.

Does anybody else think this is a good idea? (Personally I think this should've been done years ago. :-) Shall I check it into the head?

I've done similar things to urllib in the past (also handling timeouts and insufficient data transfer rates). Seems like a reasonable change to me. It also makes me think that I should switch to httplib (since all the urls I ever fetch are http). - Josiah

Guido> I recently needed to access an HTTP URL with a timeout. I ended Guido> up monkey-patching httplib.HTTPConnection so that the connect() Guido> method has an optional second paramer, timeout, defaulting to Guido> None; if not None, a call to settimeout() is added right after Guido> successful creation of the socket. Guido> Does anybody else think this is a good idea? In principle it's probably a fine idea. We should consider if it's possible to develop a uniform approach to timeouts for all the libraries that use sockets though. Otherwise you run the risk of doing it in different ways for different libraries and having to make a (small, but annoying) semantic leap when going from, say, httplib to smtpllib or ftplib. Skip

On 2/9/07, Fred L. Drake, Jr. wrote:

On Friday 09 February 2007 08:52, skip@pobox.com wrote:

...

In principle it's probably a fine idea. We should consider if it's possible to develop a uniform approach to timeouts for all the libraries that use sockets though. Otherwise you run the risk of doing it in different ways for different libraries and having to make a (small, but annoying) semantic leap when going from, say, httplib to smtpllib or ftplib.

Agreed. In the meanwhile, there's socket.setdefaulttimeout(), which has proved quite useful.

Didn't work for me, since my ap is a multi-threaded webserver and I only want one specific type of request to time out. I'm not going to change ftplib.py and all the others. I do think it's relevant to decide whether the timeout should be passed to the constructor or to the connect() method. I think it may be better to pass the timeout to the constructor.

-Fred

-- Fred L. Drake, Jr. <fdrake at acm.org>

-- --Guido van Rossum (home page: http://www.python.org/~guido/)

You're not helping, Skip. Can you spend a minute looking at urllib2 and seeing how the two variants of my proposal (pass to constructor vs. pass to connect()) would impact on it? --Guido On 2/9/07, skip@pobox.com wrote:

Guido> Didn't work for me, since my ap is a multi-threaded webserver and Guido> I only want one specific type of request to time out.

Understood.

Guido> I'm not going to change ftplib.py and all the others.

Also understood. This has, as far as I know, been the response of everybody who has encountered this problem before. Most people are only interested in a single protocol (or module) and therefore aren't motivated to change/fix the other protocols which could benefit from timeouts. After all, who really wants to update gopherlib? So there is sits.

Guido> I do think it's relevant to decide whether the timeout should be Guido> passed to the constructor or to the connect() method. I think it Guido> may be better to pass the timeout to the constructor.

Another thing to consider is how this change might percolate up to urllib/urllib2. (I advocate leaving urllib alone, but urllib2 should probably grow a timeout feature.)

Skip

-- --Guido van Rossum (home page: http://www.python.org/~guido/)

Guido> I'm not going to change ftplib.py and all the others. >> Also understood. This has, as far as I know, been the response of >> everybody who has encountered this problem before. Martin> You should read your SF bug list more frequently, then. You are Martin> currently assigned bug #723312 (and have been since 2003-04-17), Martin> which has a patch to change ftplib. Assigned by the author, because I apparently offered him some advice I no longer recall giving? After much chiding here I'll take a look. I don't know if feature requests for Roundup are still being accepted, but I hope one of its features is that it can remind people periodically of the tickets they own. My primary goal in life is not to close Python bugs and patches, so I hope people will understand if I don't think on a Thursday evening, "Gee, I think I'll drop by SourceForge and see what work I have to do tonight". Skip

>> I don't know if feature requests for Roundup are still being >> accepted, but I hope one of its features is that it can remind people >> periodically of the tickets they own. My primary goal in life is not >> to close Python bugs and patches, so I hope people will understand if >> I don't think on a Thursday evening, "Gee, I think I'll drop by >> SourceForge and see what work I have to do tonight". Martin> Maybe it would be better to remove you from the list of possible Martin> assignees, then? I'll take a look. Skip

On 2/10/07, skip@pobox.com wrote:

Guido, I looked at urllib2 and quickly gave up. I have no idea how that code works (where is a lower level library's connection object instantiated, for example?). I presume with timeouts in the lower level libraries someone who knows how urllib2 works will be able to graft timeouts onto it without too much work.

Thanks for looking anyway! I had a quick look myself; I think it relies on the implicit call to connect() in the HTTPConnection class's request() method. This answers my question: the timeout should be passed to the HTTPConnection constructor which stores it in an instance variable just like host and port, and then connect() will get it from there. -- --Guido van Rossum (home page: http://www.python.org/~guido/)

On 2/10/07, skip@pobox.com wrote:

Guido> I'm not going to change ftplib.py and all the others.

>> Also understood. This has, as far as I know, been the response of >> everybody who has encountered this problem before.

Martin> You should read your SF bug list more frequently, then. You are Martin> currently assigned bug #723312 (and have been since 2003-04-17), Martin> which has a patch to change ftplib.

Assigned by the author, because I apparently offered him some advice I no longer recall giving? After much chiding here I'll take a look.

I don't know if feature requests for Roundup are still being accepted, but I hope one of its features is that it can remind people periodically of the tickets they own.

Good idea. Kind of like a weekly "here is your Python todo list in terms of issues" email. I think it should be doable. As for taking new ideas, the goal right now is to get the transition done. The biggest thing holding it up now is documentation, especially howto docs. Once we are transitioned we are expecting people to want to change things or add features. But if you want to get the ball rolling on this for possible inclusion afterwards, email tracker-discuss about your idea.

My primary goal in life is not to close Python bugs and patches, so I hope people will understand if I don't think on a Thursday evening, "Gee, I think I'll drop by SourceForge and see what work I have to do tonight".

I understand. =)