[Numpy-discussion] locking np.random.Generator in a cython nogil context?

[Evgeni Burovski]

On Mon, Dec 14, 2020 at 4:46 PM Kevin Sheppard
<kevin.k.sheppard at gmail.com> wrote:
>
> You need to reacquire the gil, then you can get the lock and rerelease the gil.
>
> I think this works (on phone, so untested)
>
> with gil:
>     with nogil, lock:
>         ...

Thanks Kevin.
This surely works, but feels seriously weird. Is this the only / the
recommended way?
I can of course adjust the buffer size to amortize the time for the
GIL manipulation, but this really starts looking like a code smell.
My original motivation was to have inner simulation loops python-free.
Most likely, the issue is that I'm not using the Generator correctly?

Evgeni


> On Mon, Dec 14, 2020, 13:37 Evgeni Burovski <evgeny.burovskiy at gmail.com> wrote:
>>
>> Hi,
>>
>> What would be the correct way of locking the bit generator of
>> np.random.Generator in cython's nogil context?
>> (This is threading 101, surely, so please forgive my ignorance).
>>
>> The docs for extending np.random.Generator in cython
>> (https://numpy.org/doc/stable/reference/random/extending.html#cython)
>> recommend the following idiom for generating uniform variates, where
>> the GIL is released and a Generator-specific lock is held:
>>
>> x = PCG64()
>> rng = <bitgen_t *> PyCapsule_GetPointer(x.capsule, capsule_name)
>> with nogil, x.lock:
>>     rng.next_double(rng.state)
>>
>> What is the correct way of locking it when already in the nogil
>> section (so that x.lock is not accessible)?
>>
>> The use case is a long-running MC process which generates random
>> variates in a tight loop (so the loop is all nogil). In case it
>> matters, I probably won't be using python threads, but may use
>> multiprocessing.
>>
>> Basically,
>>
>>     cdef double uniform(self) nogil:
>>         if self.idx >= self.buf.shape[0]:
>>             self._fill()
>>         cdef double value = self.buf[self.idx]
>>         self.idx += 1
>>         return value
>>
>>     cdef void _fill(self) nogil:
>>         self.idx = 0
>>         # HERE: Lock ?
>>         for i in range(self.buf.shape[0]):
>>             self.buf[i] = self.rng.next_double(self.rng.state)
>>
>>
>> Thanks,
>> Evgeni
>>
>>
>> P.S. The full cdef class, for completeness:
>>
>> cdef class RndmWrapper():
>>     cdef:
>>         double[::1] buf
>>         Py_ssize_t idx
>>         bitgen_t *rng
>>         object py_gen  # keep the garbage collector away
>>
>>    def __init__(self, seed=(1234, 0), buf_size=4096, bitgen_kind=None):
>>         if bitgen_kind is None:
>>             bitgen_kind = PCG64
>>
>>         # cf Numpy-discussion list, K.~Sheppard, R.~Kern, June 29,
>> 2020 and below
>>         # https://mail.python.org/pipermail/numpy-discussion/2020-June/080794.html
>>         entropy, num = seed
>>         seed_seq = SeedSequence(entropy, spawn_key=(num,))
>>         py_gen = bitgen_kind(seed_seq)
>>
>>         # store the python object to avoid it being garbage collected
>>         self.py_gen = py_gen
>>
>>         capsule = py_gen.capsule
>>         self.rng = <bitgen_t *>PyCapsule_GetPointer(capsule, capsule_name)
>>         if not PyCapsule_IsValid(capsule, capsule_name):
>>             raise ValueError("Invalid pointer to anon_func_state")
>>
>>         self.buf = np.empty(buf_size, dtype='float64')
>>         self._fill()
>>
>>     @cython.boundscheck(False)
>>     @cython.wraparound(False)
>>     cdef void _fill(self) nogil:
>>         self.idx = 0
>>         for i in range(self.buf.shape[0]):
>>             self.buf[i] = self.rng.next_double(self.rng.state)
>>
>>     @cython.boundscheck(False)
>>     @cython.wraparound(False)
>>     cdef double uniform(self) nogil:
>>         if self.idx >= self.buf.shape[0]:
>>             self._fill()
>>         cdef double value = self.buf[self.idx]
>>         self.idx += 1
>>         return value
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at python.org
>> https://mail.python.org/mailman/listinfo/numpy-discussion
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion