On 11/06/2020 2:50 pm, Riccardo Ghetta wrote:
Hello Mark, and thanks for your suggestions. However, I'm afraid I haven't explained our use of python well enough.
If you need to share objects across threads, then there will be contention, regardless of how many interpreters there are, or which processes they are in. As a rule, we don't use that many python objects. Most of the time a
On 11/06/2020 12:59, Mark Shannon wrote: script calls C++ functions, operating on C++ data. Perhaps with a small snippet I will explain myself better :
hcpi='INFLEUR' n_months=3 base_infl=hs_base(hcpi, n_months, 0) im=hs_fs(hcpi,'sia','m',n_months,0) ip=hs_fs(hcpi,'sia','m',n_months-1,0) ir=im+(hs_range()[1].day-1)/month_days(hs_range()[1])*(ip-im) return ir/base_infl # double
this is a part of a inflation estimation used in pricing an inflation-linked bond. hcpi and n_months are really parameters of the script and the hs_ functions are all implemented in C++. Some are very small and fast like hs_range, others are much more complex and slow (hs_fs), so we wrap them with Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS. As you see, here python is used more to direct C++, than manipulate objects. At GUI level things work a bit differently, but here we just tried to avoid building and destroying a lot of ephemeral python objects (unneeded anyway, because all subsequent processing is done by C++). This python script is only a part of a larger processing done in parallel by several threads, each operating in distinct instruments. Evaluating an instrument could involve zero, one, or several of those scripts. During evaluation an instrument is bound to a single thread, so from the point of view of python threads share nothing.
If the additional resource consumption is irrelevant, what's the objection to spinning up a new processes? The additional resource consumption of a new python interpreter is irrelevant, but the process as a whole needs a lot of extra data making a new process rather costly.
Starting a new process is cheap. On my machine, starting a new Python process takes under 1ms and uses a few Mbytes. The overhead largely comes from what you do with the process. The additional cost of starting a new interpreter is the same regardless of whether it is in the same process or not. There should be no need to start a new application process for a new Python interpreter.
Plus there are issues of licensing, synchronization and load balancing that are much easier to resolve (for our system, at least) with threads than processes.
Would this prevent CPython starting new processes, or is this just for processes managed by your application?
Still, we /do/ use multiple processes, but those tend to be across administrative boundaries, or for very specific issues.
Ciao, Riccardo