I have a question about managing resources in a threadsafe way across xdist -n.
My group is using py.test as a high-level driver for testing an mpi-based numerical code. Many of our system-level tests wrap a system call to mpirun then postprocess results. I have a decorator for the tests that hints at the number of processors needed (usually something like 1,2,8).
I would like to launch as much as I can at once given the available processors. For instance, if 16 processors are available there is no reason I couldn't be doing a 12 and a 4 processor test. I was thinking of using xdist with some modest number of processors representing the maximum number of concurrent tests. The xdist test processors would launch mpi jobs when enough processors become available to satisfy the np hint for that test. This would be managed by having the tests "check out" cores and sleep if they aren't available yet.
This design requires a threadsafe method to query, acquire and lock the count of available mpi cores. I could use some sort of lock or semaphore from threading, but I thought it would be good to run this by the xdist cognoscenti and find out if there might be a preferred way of doing this given how xdist itself distributes its work or manages threads.
Thanks much,
Eli
