[Numpy-discussion] locking np.random.Generator in a cython nogil context?

[Evgeni Burovski]

Hi,

What would be the correct way of locking the bit generator of
np.random.Generator in cython's nogil context?
(This is threading 101, surely, so please forgive my ignorance).

The docs for extending np.random.Generator in cython
(https://numpy.org/doc/stable/reference/random/extending.html#cython)
recommend the following idiom for generating uniform variates, where
the GIL is released and a Generator-specific lock is held:

x = PCG64()
rng = <bitgen_t *> PyCapsule_GetPointer(x.capsule, capsule_name)
with nogil, x.lock:
    rng.next_double(rng.state)

What is the correct way of locking it when already in the nogil
section (so that x.lock is not accessible)?

The use case is a long-running MC process which generates random
variates in a tight loop (so the loop is all nogil). In case it
matters, I probably won't be using python threads, but may use
multiprocessing.

Basically,

    cdef double uniform(self) nogil:
        if self.idx >= self.buf.shape[0]:
            self._fill()
        cdef double value = self.buf[self.idx]
        self.idx += 1
        return value

    cdef void _fill(self) nogil:
        self.idx = 0
        # HERE: Lock ?
        for i in range(self.buf.shape[0]):
            self.buf[i] = self.rng.next_double(self.rng.state)


Thanks,
Evgeni


P.S. The full cdef class, for completeness:

cdef class RndmWrapper():
    cdef:
        double[::1] buf
        Py_ssize_t idx
        bitgen_t *rng
        object py_gen  # keep the garbage collector away

   def __init__(self, seed=(1234, 0), buf_size=4096, bitgen_kind=None):
        if bitgen_kind is None:
            bitgen_kind = PCG64

        # cf Numpy-discussion list, K.~Sheppard, R.~Kern, June 29,
2020 and below
        # https://mail.python.org/pipermail/numpy-discussion/2020-June/080794.html
        entropy, num = seed
        seed_seq = SeedSequence(entropy, spawn_key=(num,))
        py_gen = bitgen_kind(seed_seq)

        # store the python object to avoid it being garbage collected
        self.py_gen = py_gen

        capsule = py_gen.capsule
        self.rng = <bitgen_t *>PyCapsule_GetPointer(capsule, capsule_name)
        if not PyCapsule_IsValid(capsule, capsule_name):
            raise ValueError("Invalid pointer to anon_func_state")

        self.buf = np.empty(buf_size, dtype='float64')
        self._fill()

    @cython.boundscheck(False)
    @cython.wraparound(False)
    cdef void _fill(self) nogil:
        self.idx = 0
        for i in range(self.buf.shape[0]):
            self.buf[i] = self.rng.next_double(self.rng.state)

    @cython.boundscheck(False)
    @cython.wraparound(False)
    cdef double uniform(self) nogil:
        if self.idx >= self.buf.shape[0]:
            self._fill()
        cdef double value = self.buf[self.idx]
        self.idx += 1
        return value