[Chicago] help with file locking
Dan McGee
dpmcgee at gmail.com
Wed Feb 29 19:28:55 CET 2012
On Wed, Feb 29, 2012 at 11:50 AM, Massimo Di Pierro
<mdipierro at cs.depaul.edu> wrote:
> Hello everybody. I am stuck with a problem:
>
> ===== begin test.py =====
> #!/usr/bin/env python
>
>
> # -*- coding: utf-8 -*-
>
>
> import fcntl
> import multiprocessing
> import unittest
>
> N=10000
>
> def lock(fp, mode):
> fcntl.flock(fp.fileno(), mode)
>
> def unlock(fp):
> fcntl.flock(fp.fileno(), fcntl.LOCK_UN)
You can pass fp directly to these, they handle file-like objects.
>
> def read_write(args):
> (filename, iterations) = args
> for i in range(0, iterations):
> fp = open(filename,'r')
> lock(fp,fcntl.LOCK_SH) # shared lock
> content = fp.read()
> unlock(fp) # unlock
> fp.close()
> if len(content)!=N:
> return False
> fp = open(filename,'w')
You're violating the rules of opening files. From the docs:
'w' for writing (truncating the file if it already exists)
Simple as that.
Instead, you need something like this for your write section:
fp = open(filename, 'a')
lock(fp, fcntl.LOCK_EX, pn) # exclusive lock
fp.seek(0)
fp.truncate()
fp.write(content)
fp.flush()
unlock(fp, pn) # unlock
fp.close()
> lock(fp,fcntl.LOCK_EX) # exclusive lock
> fp.write(content)
You need a call to `fp.flush()` here for sure- Python is buffering
your writes and isn't guaranteed to do anything until you call
fp.close(), after your unlock.
> unlock(fp) # unlock
> fp.close()
> return True
>
> class TestParallelLocks(unittest.TestCase):
>
> def setUp(self):
> self.filename = 'test.txt'
> contents = 'x'*N
> fp = open(self.filename,'w')
> fp.write(contents)
> fp.close()
>
> def tearDown(self):
> try:
> os.remove(self.filename)
> except:
> pass
>
> def test_reads_and_writes(self):
> readwriters = 10
> pool = multiprocessing.Pool(processes = readwriters)
> results = pool.map(read_write, [[self.filename, 10]] * readwriters)
> for result in results:
> self.assertTrue(result)
>
> if __name__ == '__main__':
> unittest.main()
> ====== end test.py =====
>
> When I run it, it will often fail the test (on Leopard, Lion, Ubuntu, Python
> 2.5, 2.6, 2.7).
>
> Can you reproduce the problem? Do you see anything wrong with the code?
>
> Massimo
>
>
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>
More information about the Chicago
mailing list