[BangPypers] Ideas for Python concurrency...

[vikas ruhil]

Python GIL problem is solved you can look at here multiprocessing module was
developed, which enables true parallel processing with Python
on a multiprocore machine, with an interface very close to that of the
threading module.Moreover, one can run a program across machines! In other
words, the multiprocessing module allowsto run several threads not only on
the different cores of one machine, but on many machines at once, in
cooperation in the same manner that threads cooperate on one machine. By the
way, this idea is similarto something I did for Perl at IIT BOMBAY 15 dayz
ago (PerlDSM: A Distributed Shared Memory System for Perl.Proceedings of
PDPTA 2002, 63-68). We will not cover the cross-machine case here

1#dice probability finder
2# based on Python multiprocessing class

3 # usage: python DiceProb.py n k nreps nthreads
4 # where we wish to find the probability of getting a total of k dots
5 # when we roll n dice; we’ll use nreps total repetitions of the
6 # simulation, dividing those repetitions among nthreads threads
7
8 import sys
9 import random
10 from multiprocessing import Process, Lock, Value
11
12 class glbls: # globals, other than shared
13 n = int(sys.argv[1])
14 k = int(sys.argv[2])
15 nreps = int(sys.argv[3])
16 nthreads = int(sys.argv[4])
17 thrdlist = [] # list of all instances of this class
18
19 def worker(id,tot,totlock):
20 mynreps = glbls.nreps/glbls.nthreads
21 r = random.Random() # set up random number generator
22 count = 0 # number of times get total of k
23 for i in range(mynreps):
24 if rolldice(r) == glbls.k:
25 count += 1
26 totlock.acquire()
27 tot.value += count
28 totlock.release()
29 # check for load balance
30 print ’thread’, id, ’exiting; total was’, count
31
32 def rolldice(r):
33 ndots = 0
34 for roll in range(glbls.n):
35 dots = r.randint(1,6)
36 ndots += dots
37 return ndots
38
39 def main():
40 tot = Value(’i’,0)
70 CHAPTER 3. THE PYTHON THREADS AND MULTIPROCESSING MODULES
41 totlock = Lock()
42 for i in range(glbls.nthreads):
43 pr = Process(target=worker, args=(i,tot,totlock))
44 glbls.thrdlist.append(pr)
45 pr.start()
46 for thrd in glbls.thrdlist: thrd.join()
47 # adjust for truncation, in case nthreads doesn’t divide nreps evenly
48 actualnreps = glbls.nreps/glbls.nthreads * glbls.nthreads
49 print ’the probability is’,float(tot.value)/actualnreps
50
51 if __name__ == ’__main__’:
52 main()
how does it work? The general structure looks similar to that of the Python
threading module, using
Process() to create a create a thread, start() to get it running, Lock() to
create a lock, acquire() and release()
to lock and unlock a lock, and so on
 now look for multiprocessing

1 #!/usr/bin/env python
2
3 # prime number counter, based on Python multiprocessing class
4
5 # usage: python PrimeThreading.py n nthreads
6 # where we wish the count of the number of primes from 2 to n, and to
7 # use nthreads to do the work
8
9 # uses Sieve of Erathosthenes: write out all numbers from 2 to n, then
10 # cross out all the multiples of 2, then of 3, then of 5, etc., up to
11 # sqrt(n); what’s left at the end are the primes
12
13 import sys
14 import math
15 from multiprocessing import Process, Lock, Array, Value
16
17 class glbls: # globals, other than shared
18 n = int(sys.argv[1])
19 nthreads = int(sys.argv[2])
20 thrdlist = [] # list of all instances of this class
21
22 def prmfinder(id,prm,nxtk,nxtklock):
23 lim = math.sqrt(glbls.n)
24 nk = 0 # count of k’s done by this thread, to assess load balance
25 while 1:
26 # find next value to cross out with
27 nxtklock.acquire()
28 k = nxtk.value
29 nxtk.value = nxtk.value + 1
30 nxtklock.release()
31 if k > lim: break
32 nk += 1 # increment workload data
33 if prm[k]: # now cross out
34 r = glbls.n / k
35 for i in range(2,r+1):
36 prm[i*k] = 0
37 print ’thread’, id, ’exiting; processed’, nk, ’values of k’
38
39 def main():
40 prime = Array(’i’,(glbls.n+1) * [1]) # 1 means prime, until find
otherwise
41 nextk = Value(’i’,2) # next value to try crossing out with
42 nextklock = Lock()
43 for i in range(glbls.nthreads):
44 pf = Process(target=prmfinder, args=(i,prime,nextk,nextklock))
45 glbls.thrdlist.append(pf)
46 pf.start()
47 for thrd in glbls.thrdlist: thrd.join()
48 print ’there are’, reduce(lambda x,y: x+y, prime) - 2, ’primes’
49
50 if __name__ == ’__main__’:
51 main()


look here i use array ()  now may i know you got it or you can  use the Pool
class to create a set of threads, rather than doing so “by hand” in a loop
as above.You can start them with various initial values for the threads
using Pool.map(), which works similarly toPython’s ordinary map() function


On Tue, Feb 8, 2011 at 2:33 PM, Noufal Ibrahim <noufal at gmail.com> wrote:

> On Tue, Feb 08 2011, Baishampayan Ghose wrote:
>
> >> Mutliprocessing means, data copying, talking to each other through
> PIPES,
> >> also it has its issues with running on Windows (all function calls
> should be
> >> pickelable)
> >>
> >> Threads seems pretty stable on most platforms where Python runs.
> >
> > Threads won't help you much because of the Python GIL.
>
> [...]
>
> He's suggesting an alternate implementation which might work around the
> serialisation enforced by the GIL for CPU bound threads.
>
> --
> _______________________________________________
> BangPypers mailing list
> BangPypers at python.org
> http://mail.python.org/mailman/listinfo/bangpypers
>