[Web-SIG] Threading and client-side support

[John J Lee]

On Sun, 26 Oct 2003, Ian Bicking wrote:

> On Saturday, October 25, 2003, at 07:18 PM, John J Lee wrote:
[...]
> > What is the current state of thread-safety in the Python standard
> > library client-side web code (ie. httplib, urllib, urllib2)?
>
> As far as I know they are threadsafe.

I suppose I should ask on python-dev if there's a policy / tradition here.

[...]
> urlopen_lock = threading.Lock()
> def urlopen(url, data=None):
[...]

OK, thanks, that's basically as my vague understanding had it, but I had
the impression that there were all kinds of flavours of thread-safety,
guaranteeing various subtly different things?  I guess I've got some
reading to do...

Some thinking out loud in case anybody cares to help clear up my current
confusion:

Hmm, urllib2 doesn't do what your example does, but I suppose
OpenerDirectors don't currently have any state that could get lost in a
race condition in that particular case.  That would change with cookie
handling.

Am I going to have a hard time spotting all the places where I need locks?
I can't see any other place where I'd need locks other than in CookieJar.
I suppose I need to lock all access to all CookieJar methods, so that
neither reading or writing state can happen whenever CookieJar state is
changing?  I suppose I'd also need to just label the .cookies attribute as
non-threadsafe (or get rid of it, or add a __getattr__ to allow locking it
-- yuck).  Can I justify saying that some of this is the application's
problem?  For example, perhaps the .filename and attribute of CookieJar
could mess things up if altered by one thread while another thread was
reading it in order to open a file?  Is it the application's own stupid
fault if it fails to lock access to that attribute in cases where that
might happen, or is it CookieJar's problem?


John