folly:fibers (https://github.com/facebook/folly/tree/master/folly/fibers ) is a C++ package for lightweight, cooperatively scheduled threads. We have an application which extends this to CPython by adding the following save/restore code around task function invocation: auto tstate = PyThreadState_Get(); CHECK_NOTNULL(tstate); auto savedFrame = tstate->frame; auto savedExcType = tstate->exc_type; auto savedExcValue = tstate->exc_value; auto savedExcTraceback = tstate->exc_traceback; func(); tstate->frame = savedFrame; tstate->exc_type = savedExcType; tstate->exc_value = savedExcValue; tstate->exc_traceback = savedExcTraceback; (here func is a boost::python::object) This does not work in PyPy 5.9 immediately because the thread state object does not expose these fields nor are there accessor methods. Is there a way to get similar effect in PyPy? Thanks david
Hi David. You're probably completely aware that such calls (using cpyext) would completely kill performance, right? I didn't look in detail, but in order to get performance, you would need to write such calls using cffi. Best regards, Maciej Fijalkowski On Mon, Oct 16, 2017 at 7:18 PM, David Callahan <dcallahan@fb.com> wrote:
Hi Maciej, thanks for the reply I have been warned before about the performance of cpyext but we have hundreds of source dependencies and removing such reliance does not help us in the short term. Thus, performance is not a concern yet since we are not even at a point to run the code and know what is expensive and what is irrelevant. At this point, it is not clear to me how to mimic this code using the cpyext later at all. Or if it is even necessary if PyPy has a different architecture for exception handling. --david On 10/17/17, 12:10 PM, "Maciej Fijalkowski" <fijall@gmail.com> wrote: Hi David. You're probably completely aware that such calls (using cpyext) would completely kill performance, right? I didn't look in detail, but in order to get performance, you would need to write such calls using cffi. Best regards, Maciej Fijalkowski On Mon, Oct 16, 2017 at 7:18 PM, David Callahan <dcallahan@fb.com> wrote: > folly:fibers (https://github.com/facebook/folly/tree/master/folly/fibers ) > is a C++ package for lightweight, cooperatively scheduled threads. We have > an application which extends this to CPython by adding the following > save/restore code around task function invocation: > > > > auto tstate = PyThreadState_Get(); > > CHECK_NOTNULL(tstate); > > auto savedFrame = tstate->frame; > > auto savedExcType = tstate->exc_type; > > auto savedExcValue = tstate->exc_value; > > auto savedExcTraceback = tstate->exc_traceback; > > func(); > > tstate->frame = savedFrame; > > tstate->exc_type = savedExcType; > > tstate->exc_value = savedExcValue; > > tstate->exc_traceback = savedExcTraceback; > > > > (here func is a boost::python::object) > > > > This does not work in PyPy 5.9 immediately because the thread state object > does not expose these fields nor are there accessor methods. > > > > Is there a way to get similar effect in PyPy? > > > > Thanks > > david > > > > > > > > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev@python.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mailman_listinfo_pypy-2Ddev&d=DwIBaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=lFyiPUrFdOHdaobP7i4hoA&m=J_q51G31j5IKPKCE_2ZALuqBrWgdBs58pczmLTt_ml8&s=ZPehk3U44BJaOoWPII5iCNEv91mTCk_VoMR4Xq9Oh9k&e= >
Hi David, On 16 October 2017 at 21:18, David Callahan <dcallahan@fb.com> wrote:
This will never work inside PyPy, even using cpyext, because cpyext's notion of exceptions is different from the internal one inside PyPy. Instead, you should look at the ``_continuation`` module or more likely the ``greenlet`` module, which is built-in in PyPy (and available as a CPython extension module, too, if you like). It should give you a replacement, though of course you need to adapt the callers. If you really want to use folly:fibers or a specific package manipulating the C stack, then I'm afraid it's quite hard. The ``greenlet`` module in PyPy is based on ``stacklet``, which is an internal part of PyPy doing the same thing using the necessary bookkeeping for PyPy, which is different than the one for CPython you included in your e-mail. If you really want to look there, you'd need to replicate that logic around folly:fibres (search for all "stacklet" file names in several places in the PyPy source code). A bientôt, Armin.
Hi David. You're probably completely aware that such calls (using cpyext) would completely kill performance, right? I didn't look in detail, but in order to get performance, you would need to write such calls using cffi. Best regards, Maciej Fijalkowski On Mon, Oct 16, 2017 at 7:18 PM, David Callahan <dcallahan@fb.com> wrote:
Hi Maciej, thanks for the reply I have been warned before about the performance of cpyext but we have hundreds of source dependencies and removing such reliance does not help us in the short term. Thus, performance is not a concern yet since we are not even at a point to run the code and know what is expensive and what is irrelevant. At this point, it is not clear to me how to mimic this code using the cpyext later at all. Or if it is even necessary if PyPy has a different architecture for exception handling. --david On 10/17/17, 12:10 PM, "Maciej Fijalkowski" <fijall@gmail.com> wrote: Hi David. You're probably completely aware that such calls (using cpyext) would completely kill performance, right? I didn't look in detail, but in order to get performance, you would need to write such calls using cffi. Best regards, Maciej Fijalkowski On Mon, Oct 16, 2017 at 7:18 PM, David Callahan <dcallahan@fb.com> wrote: > folly:fibers (https://github.com/facebook/folly/tree/master/folly/fibers ) > is a C++ package for lightweight, cooperatively scheduled threads. We have > an application which extends this to CPython by adding the following > save/restore code around task function invocation: > > > > auto tstate = PyThreadState_Get(); > > CHECK_NOTNULL(tstate); > > auto savedFrame = tstate->frame; > > auto savedExcType = tstate->exc_type; > > auto savedExcValue = tstate->exc_value; > > auto savedExcTraceback = tstate->exc_traceback; > > func(); > > tstate->frame = savedFrame; > > tstate->exc_type = savedExcType; > > tstate->exc_value = savedExcValue; > > tstate->exc_traceback = savedExcTraceback; > > > > (here func is a boost::python::object) > > > > This does not work in PyPy 5.9 immediately because the thread state object > does not expose these fields nor are there accessor methods. > > > > Is there a way to get similar effect in PyPy? > > > > Thanks > > david > > > > > > > > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev@python.org > https://urldefense.proofpoint.com/v2/url?u=https-3A__mail.python.org_mailman_listinfo_pypy-2Ddev&d=DwIBaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=lFyiPUrFdOHdaobP7i4hoA&m=J_q51G31j5IKPKCE_2ZALuqBrWgdBs58pczmLTt_ml8&s=ZPehk3U44BJaOoWPII5iCNEv91mTCk_VoMR4Xq9Oh9k&e= >
Hi David, On 16 October 2017 at 21:18, David Callahan <dcallahan@fb.com> wrote:
This will never work inside PyPy, even using cpyext, because cpyext's notion of exceptions is different from the internal one inside PyPy. Instead, you should look at the ``_continuation`` module or more likely the ``greenlet`` module, which is built-in in PyPy (and available as a CPython extension module, too, if you like). It should give you a replacement, though of course you need to adapt the callers. If you really want to use folly:fibers or a specific package manipulating the C stack, then I'm afraid it's quite hard. The ``greenlet`` module in PyPy is based on ``stacklet``, which is an internal part of PyPy doing the same thing using the necessary bookkeeping for PyPy, which is different than the one for CPython you included in your e-mail. If you really want to look there, you'd need to replicate that logic around folly:fibres (search for all "stacklet" file names in several places in the PyPy source code). A bientôt, Armin.
participants (3)
-
Armin Rigo -
David Callahan -
Maciej Fijalkowski