If you need to share objects across threads, then there will be contention, regardless of how many interpreters there are, or which processes they are in. As a rule, we don't use that many python objects. Most of the time a
If the additional resource consumption is irrelevant, what's the objection to spinning up a new processes? The additional resource consumption of a new python interpreter is irrelevant, but the process as a whole needs a lot of extra data making a new process rather costly. Plus there are issues of licensing, synchronization and load balancing
Hello Mark, and thanks for your suggestions. However, I'm afraid I haven't explained our use of python well enough. On 11/06/2020 12:59, Mark Shannon wrote: script calls C++ functions, operating on C++ data. Perhaps with a small snippet I will explain myself better : hcpi='INFLEUR' n_months=3 base_infl=hs_base(hcpi, n_months, 0) im=hs_fs(hcpi,'sia','m',n_months,0) ip=hs_fs(hcpi,'sia','m',n_months-1,0) ir=im+(hs_range()[1].day-1)/month_days(hs_range()[1])*(ip-im) return ir/base_infl # double this is a part of a inflation estimation used in pricing an inflation-linked bond. hcpi and n_months are really parameters of the script and the hs_ functions are all implemented in C++. Some are very small and fast like hs_range, others are much more complex and slow (hs_fs), so we wrap them with Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS. As you see, here python is used more to direct C++, than manipulate objects. At GUI level things work a bit differently, but here we just tried to avoid building and destroying a lot of ephemeral python objects (unneeded anyway, because all subsequent processing is done by C++). This python script is only a part of a larger processing done in parallel by several threads, each operating in distinct instruments. Evaluating an instrument could involve zero, one, or several of those scripts. During evaluation an instrument is bound to a single thread, so from the point of view of python threads share nothing. that are much easier to resolve (for our system, at least) with threads than processes. Still, we /do/ use multiple processes, but those tend to be across administrative boundaries, or for very specific issues. Ciao, Riccardo