Thread Locking issue - Can't allocate lock (sem_init fail)

[Bryan Olson]

jamskip at googlemail.com wrote:
[...]
 > The program is multithreaded to speed up the processing...there are
 > input and output Queues.

It's not the major point here, but are you aware of Python's GIL?

 > Now, each domain entry is an class object containing various bits of
 > info. Each domain class also has its own lock, so that only one thread
 > can modify each domain at a time.
 >
 > I'm load-testing the program with a sample of 1 million email
 > addresses and when i hit about the 500,000 mark i get a locking
 > error...

Does that correspond to creating 500,000 locks? For sure?

 >     sem_init: No space left on device
[...]
 >     error: can't allocate lock

What happens if you try to allocate a million locks, as in:

     from threading import Lock
     locks = []
     for i in range(10**6):
         try:
             locks.append(Lock())
             locks[-1].acquire()
         except:
             print "Failed after %d locks." % i
             raise

I tried the above in WindowsXP and Ubuntu 8.04 (Python 2.6 and 2.52 
respectively), and it ran on both without the exception.

 > Googling for this sort of error doesn't yield any results, and i can't
 > find any information about limits to the number of locks you can have
 > in Python. The 'No space left on device' message indicates a memory
 > issue, however i doubt this since its running on a linux server with 4
 > cores and 16GB ram. It seems more like an internal Python limit has
 > been hit (sem_init - semaphore initialisation?). Does anyone know more
 > about threading internals and any internal limits?

Python treading is implemented on top of some thread facility the target 
O.S. provides. Creating a Python lock may or may not use up some limited 
kernel resource. Given my experiments, limited as they were I do not 
think this is a Python limit. If could be an issue with your platform, 
or a defect in the code you inherited.

 > For the timebeing, I've implemented locking at a higher level which
 > will reduce performance but keep the number of lock objects to a
 > minumum.

That's probably a reasonable strategy whether or not you can create a 
million locks.


-- 
--Bryan