[Numpy-discussion] locking np.random.Generator in a cython nogil context?

[Kevin Sheppard]

You need to reacquire the gil, then you can get the lock and rerelease the
gil.

I think this works (on phone, so untested)

with gil:
    with nogil, lock:
        ...

Kevin


On Mon, Dec 14, 2020, 13:37 Evgeni Burovski <evgeny.burovskiy at gmail.com>
wrote:

> Hi,
>
> What would be the correct way of locking the bit generator of
> np.random.Generator in cython's nogil context?
> (This is threading 101, surely, so please forgive my ignorance).
>
> The docs for extending np.random.Generator in cython
> (https://numpy.org/doc/stable/reference/random/extending.html#cython)
> recommend the following idiom for generating uniform variates, where
> the GIL is released and a Generator-specific lock is held:
>
> x = PCG64()
> rng = <bitgen_t *> PyCapsule_GetPointer(x.capsule, capsule_name)
> with nogil, x.lock:
>     rng.next_double(rng.state)
>
> What is the correct way of locking it when already in the nogil
> section (so that x.lock is not accessible)?
>
> The use case is a long-running MC process which generates random
> variates in a tight loop (so the loop is all nogil). In case it
> matters, I probably won't be using python threads, but may use
> multiprocessing.
>
> Basically,
>
>     cdef double uniform(self) nogil:
>         if self.idx >= self.buf.shape[0]:
>             self._fill()
>         cdef double value = self.buf[self.idx]
>         self.idx += 1
>         return value
>
>     cdef void _fill(self) nogil:
>         self.idx = 0
>         # HERE: Lock ?
>         for i in range(self.buf.shape[0]):
>             self.buf[i] = self.rng.next_double(self.rng.state)
>
>
> Thanks,
> Evgeni
>
>
> P.S. The full cdef class, for completeness:
>
> cdef class RndmWrapper():
>     cdef:
>         double[::1] buf
>         Py_ssize_t idx
>         bitgen_t *rng
>         object py_gen  # keep the garbage collector away
>
>    def __init__(self, seed=(1234, 0), buf_size=4096, bitgen_kind=None):
>         if bitgen_kind is None:
>             bitgen_kind = PCG64
>
>         # cf Numpy-discussion list, K.~Sheppard, R.~Kern, June 29,
> 2020 and below
>         #
> https://mail.python.org/pipermail/numpy-discussion/2020-June/080794.html
>         entropy, num = seed
>         seed_seq = SeedSequence(entropy, spawn_key=(num,))
>         py_gen = bitgen_kind(seed_seq)
>
>         # store the python object to avoid it being garbage collected
>         self.py_gen = py_gen
>
>         capsule = py_gen.capsule
>         self.rng = <bitgen_t *>PyCapsule_GetPointer(capsule, capsule_name)
>         if not PyCapsule_IsValid(capsule, capsule_name):
>             raise ValueError("Invalid pointer to anon_func_state")
>
>         self.buf = np.empty(buf_size, dtype='float64')
>         self._fill()
>
>     @cython.boundscheck(False)
>     @cython.wraparound(False)
>     cdef void _fill(self) nogil:
>         self.idx = 0
>         for i in range(self.buf.shape[0]):
>             self.buf[i] = self.rng.next_double(self.rng.state)
>
>     @cython.boundscheck(False)
>     @cython.wraparound(False)
>     cdef double uniform(self) nogil:
>         if self.idx >= self.buf.shape[0]:
>             self._fill()
>         cdef double value = self.buf[self.idx]
>         self.idx += 1
>         return value
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at python.org
> https://mail.python.org/mailman/listinfo/numpy-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/numpy-discussion/attachments/20201214/776e4fa5/attachment.html>