[Python-Dev] Making python C-API thread safe (try 2)

[Jeff Epler]

Harri,
I don't understand how your suggested changes, even once carried out,
will let me use threads to increase the speed of a CPU-intensive Python
program.  For instance, consider the following code:

solutions = []

def is_solution(l):
    # CPU-intensive code here.  Let's say it runs for 1 second per call

def consider_solution(l):
    if is_solution(l):
        solutions.append(l)

def problem_space(l):
    # A generator of items in the problem space
    # pretty fast!
    ...
    yield l

def all_solutions():
    for l in problem_space:
        consider_solution(l)


I could thread it, so that N threads each run is_solution on a different
candidate:
    def all_solutions():
        queue = worker_tasks(consider_solution, N)
        for l in problem_space:
            queue.add(l)
        queue.shutdown()

But with your proposed changes, it sounds like each thread becomes an
island, with no access to common objects (like the list "solutions" or
the queue connecting the main thread with the worker threads).
If threading truly worked, then I'd be able to run efficiently on n*1
CPUs, where n is the ratio of the speed of one iteration of is_solution
compared to one iteration of problem_space.

On the other hand, I can make the above work quickly today by using
processes and pipes.  I can do this only because I've identified the
parts that need to be shared (the queue of candidate solutions, and the
list of confirmed solutions).  I think that's the same level of effort
required under the "thread is an island" approach you're suggesting, but
the processes&pipes code will likely be easier to write.

Jeff