[BangPypers] Ideas for Python concurrency...

[vikas ruhil]

also when you perform  you can face any problem you can look at debugger You
can obtain it from http://code.google.com/p/
pydbgr/<http://code.google.com/p/pydbgr/>


On Tue, Feb 8, 2011 at 2:44 PM, vikas ruhil <vikasruhil06 at gmail.com> wrote:

> Python GIL problem is solved you can look at here multiprocessing module
> was developed, which enables true parallel processing with Python
> on a multiprocore machine, with an interface very close to that of the
> threading module.Moreover, one can run a program across machines! In other
> words, the multiprocessing module allowsto run several threads not only on
> the different cores of one machine, but on many machines at once, in
> cooperation in the same manner that threads cooperate on one machine. By
> the way, this idea is similarto something I did for Perl at IIT BOMBAY 15
> dayz ago (PerlDSM: A Distributed Shared Memory System for Perl.Proceedings
> of PDPTA 2002, 63-68). We will not cover the cross-machine case here
>
> 1#dice probability finder
> 2# based on Python multiprocessing class
>
> 3 # usage: python DiceProb.py n k nreps nthreads
> 4 # where we wish to find the probability of getting a total of k dots
> 5 # when we roll n dice; we’ll use nreps total repetitions of the
> 6 # simulation, dividing those repetitions among nthreads threads
> 7
> 8 import sys
> 9 import random
> 10 from multiprocessing import Process, Lock, Value
> 11
> 12 class glbls: # globals, other than shared
> 13 n = int(sys.argv[1])
> 14 k = int(sys.argv[2])
> 15 nreps = int(sys.argv[3])
> 16 nthreads = int(sys.argv[4])
> 17 thrdlist = [] # list of all instances of this class
> 18
> 19 def worker(id,tot,totlock):
> 20 mynreps = glbls.nreps/glbls.nthreads
> 21 r = random.Random() # set up random number generator
> 22 count = 0 # number of times get total of k
> 23 for i in range(mynreps):
> 24 if rolldice(r) == glbls.k:
> 25 count += 1
> 26 totlock.acquire()
> 27 tot.value += count
> 28 totlock.release()
> 29 # check for load balance
> 30 print ’thread’, id, ’exiting; total was’, count
> 31
> 32 def rolldice(r):
> 33 ndots = 0
> 34 for roll in range(glbls.n):
> 35 dots = r.randint(1,6)
> 36 ndots += dots
> 37 return ndots
> 38
> 39 def main():
> 40 tot = Value(’i’,0)
> 70 CHAPTER 3. THE PYTHON THREADS AND MULTIPROCESSING MODULES
> 41 totlock = Lock()
> 42 for i in range(glbls.nthreads):
> 43 pr = Process(target=worker, args=(i,tot,totlock))
> 44 glbls.thrdlist.append(pr)
> 45 pr.start()
> 46 for thrd in glbls.thrdlist: thrd.join()
> 47 # adjust for truncation, in case nthreads doesn’t divide nreps evenly
> 48 actualnreps = glbls.nreps/glbls.nthreads * glbls.nthreads
> 49 print ’the probability is’,float(tot.value)/actualnreps
> 50
> 51 if __name__ == ’__main__’:
> 52 main()
> how does it work? The general structure looks similar to that of the Python
> threading module, using
> Process() to create a create a thread, start() to get it running, Lock() to
> create a lock, acquire() and release()
> to lock and unlock a lock, and so on
>  now look for multiprocessing
>
> 1 #!/usr/bin/env python
> 2
> 3 # prime number counter, based on Python multiprocessing class
> 4
> 5 # usage: python PrimeThreading.py n nthreads
> 6 # where we wish the count of the number of primes from 2 to n, and to
> 7 # use nthreads to do the work
> 8
> 9 # uses Sieve of Erathosthenes: write out all numbers from 2 to n, then
> 10 # cross out all the multiples of 2, then of 3, then of 5, etc., up to
> 11 # sqrt(n); what’s left at the end are the primes
> 12
> 13 import sys
> 14 import math
> 15 from multiprocessing import Process, Lock, Array, Value
> 16
> 17 class glbls: # globals, other than shared
> 18 n = int(sys.argv[1])
> 19 nthreads = int(sys.argv[2])
> 20 thrdlist = [] # list of all instances of this class
> 21
> 22 def prmfinder(id,prm,nxtk,nxtklock):
> 23 lim = math.sqrt(glbls.n)
> 24 nk = 0 # count of k’s done by this thread, to assess load balance
> 25 while 1:
> 26 # find next value to cross out with
> 27 nxtklock.acquire()
> 28 k = nxtk.value
> 29 nxtk.value = nxtk.value + 1
> 30 nxtklock.release()
> 31 if k > lim: break
> 32 nk += 1 # increment workload data
> 33 if prm[k]: # now cross out
> 34 r = glbls.n / k
> 35 for i in range(2,r+1):
> 36 prm[i*k] = 0
> 37 print ’thread’, id, ’exiting; processed’, nk, ’values of k’
> 38
> 39 def main():
> 40 prime = Array(’i’,(glbls.n+1) * [1]) # 1 means prime, until find
> otherwise
> 41 nextk = Value(’i’,2) # next value to try crossing out with
> 42 nextklock = Lock()
> 43 for i in range(glbls.nthreads):
> 44 pf = Process(target=prmfinder, args=(i,prime,nextk,nextklock))
> 45 glbls.thrdlist.append(pf)
> 46 pf.start()
> 47 for thrd in glbls.thrdlist: thrd.join()
> 48 print ’there are’, reduce(lambda x,y: x+y, prime) - 2, ’primes’
> 49
> 50 if __name__ == ’__main__’:
> 51 main()
>
>
> look here i use array ()  now may i know you got it or you can  use the
> Pool class to create a set of threads, rather than doing so “by hand” in a
> loop as above.You can start them with various initial values for the threads
> using Pool.map(), which works similarly toPython’s ordinary map() function
>
>
>
> On Tue, Feb 8, 2011 at 2:33 PM, Noufal Ibrahim <noufal at gmail.com> wrote:
>
>> On Tue, Feb 08 2011, Baishampayan Ghose wrote:
>>
>> >> Mutliprocessing means, data copying, talking to each other through
>> PIPES,
>> >> also it has its issues with running on Windows (all function calls
>> should be
>> >> pickelable)
>> >>
>> >> Threads seems pretty stable on most platforms where Python runs.
>> >
>> > Threads won't help you much because of the Python GIL.
>>
>> [...]
>>
>> He's suggesting an alternate implementation which might work around the
>> serialisation enforced by the GIL for CPU bound threads.
>>
>> --
>> _______________________________________________
>> BangPypers mailing list
>> BangPypers at python.org
>> http://mail.python.org/mailman/listinfo/bangpypers
>>
>
>