[ python-Feature Requests-1432694 ] Implement preemptive threads in Python

[SourceForge.net]

Feature Requests item #1432694, was opened at 2006-02-16 18:11
Message generated for change (Comment added) made by ncoghlan
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1432694&group_id=5470

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Threads
Group: None
Status: Open
Resolution: Invalid
Priority: 5
Submitted By: Andrey Petrov (darkprokoba)
Assigned to: Michael Hudson (mwh)
Summary: Implement preemptive threads in Python

Initial Comment:
Cooperative threads have their uses but I find the lack
of real native pre-emptive threads a major issue in a
21-st century language.
We should consider the inertia of programmers from
C++/Java/.NET background. They are used to and
experienced in solving certain kind of problems with
pre-emptive threads, and it's just too plain
inconvenient to step back and try to cope with
cooperative threads.

The idea behind Python is that programming should not
be a pain, right?

This applies especially to GUI programs, where threads
are used to perform time-consuming work in order to
avoid the GUI from freezing (was I surprised the first
time I tried this in Python!)
Just look at some of the modern Python GUI
applications, found in some recent Linux distributions.
99% of Fedora's configuration tools suffer extreme
usability issues caused by their single-threadedness. I
hate using yumex because the GUI is basically frozen
for the half minute it takes to reload all repos I have
configured on my machine!!! I know there might be a way
to alleviate this particual class of problems by more
careful cooperative thread programing... but I believe
we need a more straightforward solution (i.e.
preemptive threads) and not workarounds.

I apologize if this issue has already been raised before.

Cheers,
darkprokoba

----------------------------------------------------------------------

>Comment By: Nick Coghlan (ncoghlan)
Date: 2006-04-07 21:28

Message:
Logged In: YES 
user_id=1038590

One addition to Josiah's comments is that the previous
attempts at free-threading failed to gain an appreciable
performance benefit, as the gains from removing the global
lock were lost in the subsequent need to add explicit
synchronisation everywhere. Free threading is not a panacea.

If Queue.Queue is used to pass data to worker threads, idle
threads will not be consuming any GIL time. If there is OS
level context-switch thrashing going on, then its time to
look at using thread pools instead of a thread per task.

Now, if you were to rephrase this request as "I want to be
able to make the GUI thread higher priority than the worker
threads so I can have a gazillion threads without hurting
responsiveness of the main event loop so badly", that would
be an entirely different story, and not a question I have
seen asked recently (or at all, in fact).

(Josiah's comments about any such thing being a couple of
years away still hold, however - the multi-process support
libraries are here and now, and scale a lot better than
threads ever will, no matter what language you use)

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2006-02-22 18:35

Message:
Logged In: YES 
user_id=341410

How many worker threads do you expect?  If you work the
sys.setcheckinterval() just right, you may be able to
prioritize your threads...

Any C library which does not acquire and release the GIL
when it is supposed to is broken.  Claiming that Python is
broken in relation to C libraries being broken is misplaced
blame.

Python uses a GIL because it makes interaction with Python
objects easier across threads.  There have been previous
failed efforts to make Python scalable across multiple
processors; search for "python free threading" on google to
see all of the relevant information about rewriting the
Python interpreter with multiple processors in mind.

Alternatively, you can always use one of the half-dozen
parallel processing libraries for Python to distribute your
work onto different processors, who all synchronize up with
the main GUI process.  Some are quite intuitive (check out
Kamaelia), and will work today (any effort to get a 'free
threading' idea into Python core will have to wait until at
least Python 2.6, which is at least 2 years away).

----------------------------------------------------------------------

Comment By: Andrey Petrov (darkprokoba)
Date: 2006-02-22 17:55

Message:
Logged In: YES 
user_id=950037

> thread context switches happen at a regular
> rate
thanks for your response, but with enough worker threads
it's still easy to starve the GUI thread.
and the other problems of this model remain:
1) a stupid native module could forget to release the global
interpreter lock and block on I/O, freezing the entire
interpreter (i.e. GIL is a pain for native module writers)
2) poor CPU utilization on SMP machines (which are quite
mainstream now)

Python's model just doesn't cut it for me (I'm well aware
now of how it works) so my question boils down to:

Do the python authors plan to make the interpreter reentrant
and if so when can we expect to see this in a production
release?

----------------------------------------------------------------------

Comment By: Josiah Carlson (josiahcarlson)
Date: 2006-02-22 15:44

Message:
Logged In: YES 
user_id=341410

If GUI applications are written properly, they don't need to
hang when they are doing "real work".  The applications you
are using which seem to hang are doing so not because Python
"doesn't have preemptive threads", they are doing so because
the writers of those GUI applications have no idea how to
either 1. write their programs asynchronously via generators
or callbacks or 2. use threads to do the actual work which
post GUI update events.  Talk to the writers of those
applications to get them fixed.

Note that regardless of Python's lack of preemptive
multithreading, thread context switches happen at a regular
rate, discoverable via sys.getcheckinterval(), and setable
via sys.setcheckinterval().

----------------------------------------------------------------------

Comment By: Andrey Petrov (darkprokoba)
Date: 2006-02-17 19:52

Message:
Logged In: YES 
user_id=950037

OK, let me try again :-)
The problem is in the global interpreter lock:
http://docs.python.org/api/threads.html
this basically says that you can have as many native threads
as you like but only one of them could have the global
interpreter lock and could therefore be executing python
code at a time. The only use of python's multiple threads
then is so they could release the global lock and block on
i/o operations while one lucky thread has the lock and
executes python code and accesses python objects happily.
So we have multiple native threads (e.g. pthreads) but they
pass arround the global lock in a cooperative manner. This
is not preemptive threading imho.
It's a severely limited model and has the following
potential problem: a thread may enter a time-consuming i/o
operation but forget to release the global interpreter lock
- leading to a freeze in the entire interpreter (all threads
are waiting for the global lock, while the thread that has
it waits on the i/o operation)
Are there any plans for alleviating this problem? A thread
should not voluntarily release the lock so that the rest of
the threads get a chance to execute python code. It should
be possible to preempt any thread at any time without its
consent in order to have a true preemptive threading model.
Or please correct me If I'm wrong.

----------------------------------------------------------------------

Comment By: Michael Hudson (mwh)
Date: 2006-02-17 00:14

Message:
Logged In: YES 
user_id=6656

We _have_ preemptive threads in Python ("import threading").  Suggest you 
report a real problem.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1432694&group_id=5470