[Python-bugs-list] [Bug #133200] cPickle does not use Py_BEGIN_ALLOW_THREADS.

[noreply@sourceforge.net]

Bug #133200, was updated on 2001-Feb-20 00:12
Here is a current snapshot of the bug.

Project: Python
Category: Threads
Status: Open
Resolution: None
Bug Group: None
Priority: 5
Submitted by: ianbanks
Assigned to : dcjim
Summary: cPickle does not use Py_BEGIN_ALLOW_THREADS.

Details: This bug refers to:

python/dist/src/Modules/cPickle.c
Revision 2.54 (In SourceForge)

The use of fread (line 506) and fwrite (line 410) are not wrapped by
Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS.

This causes certain uses of cPickle in threaded programs to deadlock, where
pickle does not.


Follow-Ups:

Date: 2001-Feb-20 19:33
By: ianbanks

Comment:
 1:
 2: import socket, thread, cPickle
 3: 
 4: def Consumer(socketname):
 5:     print "Consumer: (Client) Starting"
 6:     client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
 7:     client.connect(socketname)
 8:     file = client.makefile("rb+")
 9: 
10:     print "Consumer: Loading File"
11:     print "Consumer: Loaded " + cPickle.load(file)
12:     print "Consumer: Done"
13: 
14: def Producer(socketname):
15:     # Create the Server socket.
16:     print "Producer: (Server) Starting"
17:     server = socket.socket(socket.AF_UNIX,  socket.SOCK_STREAM)
18:     server.bind(socketname)
19:     server.listen(5)
20: 
21:     # Accept a connection and create a file  object.
22:     newsocket, peer = server.accept()
23:     print "Producer: Connection Accepted"
24:     newfile = newsocket.makefile("rb+")
25: 
26:     # Dump a pickled object.
27:     print "Producer: Dumping File"
28:     cPickle.dump("Testing", newfile, 1)
29:     print "Producer: Done"
30: 
31: socketname = "/tmp/testsocket"
32: thread.start_new_thread(Producer,  (socketname,))
33: thread.start_new_thread(Consumer, (socketname,))
34: while 1:
35:     # Busy wait.
36:     pass

I think this locks up because of this sequence:

Both threads become runnable, and the Producer runs from line 15 to just
after line 24. The context switches to the Consumer and runs from line 5 to
line 11, blocking on the read() (and the underlying fread in cThreads). The
Producer tries to aquire the global lock, the Consumer waits on the "data
available" condition.

The lock-up doesn't always occur. On some systems it's easy to reproduce,
others seems to hide it.

Is it poor practice to assume simple I/O in threads won't block the entire
process? That would seem to imply that things like ThreadingMixIn derived
servers were technically subject to denial of service, and that looped-back
connections were unsafe.

It occurs on:

 o Linux 2.2.17-RAID / glibc 2.2.1 / Intel
 o Linux 2.2.18 (SMP) / glibc 2.1.3 / Intel
 o Linux 2.2.17pre4-RAI / glibc 2.1.3 / Intel

-------------------------------------------------------

Date: 2001-Feb-20 13:17
By: tim_one

Comment:
Releasing the global lock across I/O operations is never done to prevent
deadlocks, it's to let the OS overlap I/O with computation (or other I/O)
in some other thread.

OTOH, if the stream is connected to some network resource, it's possible
for the I/O to hang for external reasons, and without releasing the global
lock then the whole app hangs.  While not technically a deadlock, it
probably sure *looks* like one <wink>.

Jim, if you don't object, I'd be happy to put in global lock fiddling here
where appropriate (just reassign the bug report to me in that case).

-------------------------------------------------------

Date: 2001-Feb-20 11:54
By: fdrake

Comment:
Can you provide an example that actually deadlocks?  Please provide
platform information as well.

cPckle certainly could be more thread-friendly in the way that you suggest,
but it should not actually deadlock either.
-------------------------------------------------------

For detailed info, follow this link:
http://sourceforge.net/bugs/?func=detailbug&bug_id=133200&group_id=5470