[Python-Dev] Extension modules, Threading, and the GIL

[James Henstridge]

This is a multi-part message in MIME format.
--------------090900070709010305040001
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

I originally sent this as private mail to David, but he suggested I
repost it here as well.

The Python GIL has been causing trouble getting threading to work
correctly in my Python bindings for GTK 2.x.  Both Python and GTK have
an idea about how threading should work, and meshing the two together
isn't quite successful.  It would be great if Python made this kind of
thing easier.

James.

-- 
Email: james@daa.com.au
WWW:   http://www.daa.com.au/~james/




--------------090900070709010305040001
Content-Type: message/rfc822;
 name="your threading/extensions posts on python-dev.eml"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="your threading/extensions posts on python-dev.eml"

Message-ID: <3E251157.3020809@daa.com.au>
Date: Wed, 15 Jan 2003 15:44:23 +0800
From: James Henstridge <james@daa.com.au>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.2.1) Gecko/20021130
X-Accept-Language: en-au, en
MIME-Version: 1.0
To:  dave@boost-consulting.com
Subject: your threading/extensions posts on python-dev
Content-Type: text/plain; charset=us-ascii; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

I am the author/maintainer of the Python bindings for GTK+ and GNOME, so 
have run into a number of these problems as well.  If changing Python 
made threading more reliable in PyGTK, that would be great.

Neither Python or GTK+ are free-threaded, which makes juggling the locks 
quite difficult (the thread support code in current 1.99.x releases of 
PyGTK doesn't really work).

I don't know how different pygtk's requirements are to your Qt code are, 
so I will give a quick description of the relevant bits of pygtk.

1. GTK uses an event loop, so the program will spend most of its time in
    a call to gtk.main().  We obviously want to allow other threads to
    run during this call (=> release python GIL).

2. GTK has a signal system to handle notifications.  Arbitrary numbers
    of callbacks can be attached to signals.  PyGTK has a generic marshal
    function to handle all signal callbacks, which uses introspection to
    handle the different signatures.

    Some signals will be emitted in response to method calls (ie. GIL is
    held), while others are emitted in the event loop (GIL not held).
    Some signals can be emitted in both conditions, so it isn't really
    possible for the signal handler marshal code to know if the GIL is
    being held or not.

3. Add to this mix GTK+'s global lock which must be held when
    executing gtk functions.  This lock is held during signal emissions,
    but must be manually acquired if a thread other than the one running
    the main loop wants to execute gtk functions.  The functions used to
    deallocate gtk objects count here, which means that the GTK lock must
    be held while finalising the Python wrappers.

Having a recursive GIL in Python would help a lot with signal emission 
issue (to handle cases where the signal emission occurs while the Python 
GIL is not held, and while it is held).

I am sure that it would be possible to clear up a lot of these issues by 
dropping the Python lock during every gtk API call, though this would 
require a fair bit of work to the code generator and testing (there are 
a few thousand gtk APIs).

Sorry for a long winded email, but if you are going to work on Python 
threading, it would be good if the changes could help solve at least 
some of these problems.

James Henstridge.

-- 
Email: james@daa.com.au              | Linux.conf.au   http://linux.conf.au/
WWW:   http://www.daa.com.au/~james/ | Jan 22-25   Perth, Western 
Australia.




--------------090900070709010305040001--