urllib.urlopen + https = all threads locked?

[Gerhard Häring]

On Wed, Oct 24, 2001 at 02:51:08PM -0700, Jeff Johnson wrote:
> I have a multi-threaded application that makes calls
> urllib.urlopen("https://yadayada.com").  Until the call completes (or
> maybe the following read) all other threads just freeze.  They
> continue working after the call completes so it took me a while to
> realize there was a problem.
> 
> Someone suggested:
> 
> "Sounds like the C code that does the SSL is not _releasing_ 
> the global thread lock."

Yes, this is true.

> I'm using python SSL that comes with Python 2.1.1 on FreeBSD 4.3.
> 
> I searched usenet and everywhere else I could think of but only found
> one usenet post about broken ThreadingMixin know to be broken in
> 2.1.1c but I'm not sure if this is related.
> 
> Any suggestions?

Could you perhaps provide a test case for publicly available SSL
servers? That would be helpful for fixing the problem.

In the meantime, this is a quick (untested) patch against current CVS
Python:

*** /mnt/gargamel/src/python.orig/dist/src/Modules/socketmodule.c	Fri Oct 19 19:11:51 2001
--- socketmodule.c	Thu Oct 25 01:19:24 2001
***************
*** 2689,2695 ****
--- 2689,2697 ----
  	if (!PyArg_ParseTuple(args, "s#:write", &data, &len))
  		return NULL;
  
+ 	Py_BEGIN_ALLOW_THREADS
  	len = SSL_write(self->ssl, data, len);
+ 	Py_END_ALLOW_THREADS
  	if (len > 0)
  		return PyInt_FromLong(len);
  	else
***************
*** 2714,2721 ****
  	if (!(buf = PyString_FromStringAndSize((char *) 0, len)))
  		return NULL;
  
  	count = SSL_read(self->ssl, PyString_AsString(buf), len);
!  	if (count <= 0) {
  		Py_DECREF(buf);
  		return PySSL_SetError(self->ssl, count);
  	}
--- 2716,2725 ----
  	if (!(buf = PyString_FromStringAndSize((char *) 0, len)))
  		return NULL;
  
+ 	Py_BEGIN_ALLOW_THREADS
  	count = SSL_read(self->ssl, PyString_AsString(buf), len);
!  	Py_END_ALLOW_THREADS
! 	if (count <= 0) {
  		Py_DECREF(buf);
  		return PySSL_SetError(self->ssl, count);
  	}

If the patch doesn't apply cleanly, just search for SSL_read and
SSL_write and wrap these lines with Py_BEGIN_ALLOW_THREADS resp.
Py_END_ALLOW_THREADS.

> Is this a known problem?

Unless it's reported as a bug with the Sourceforge bugtracker, it's not
a known problem <0.7 wink>.

Gerhard
-- 
mail:   gerhard <at> bigfoot <dot> de       registered Linux user #64239
web:    http://www.cs.fhm.edu/~ifw00065/    OpenPGP public key id 86AB43C0
public key fingerprint: DEC1 1D02 5743 1159 CD20  A4B6 7B22 6575 86AB 43C0
reduce(lambda x,y:x+y,map(lambda x:chr(ord(x)^42),tuple('zS^BED\nX_FOY\x0b')))