using python interpreters per thread in C++ program

[sturlamolden]

On 8 Sep, 04:22, ganesh <ganeshbo... at gmail.com> wrote:

> My application is a TCP server having multiple client connectons. C++
> PTHREADS are for each connected socket and the message received on the
> socket is evaluated by python functions.
> If I use only one process level python interpreter, then every thread
> has to lock the GIL & so blocking the other threads from executing the
> python code even if it is not the same python function that locking
> thread is calling.

Usually, TCP-servers are I/O bound. You can safely use a single Python
process for this. A function evaluating a request will hold the GIL
for a while (but not until it's done). But most other threads will be
blocked waiting for I/O. Thus, there will be little contention for the
GIL anyway, and it should not affect scalability much. Only when
multiple requests are processed simultaneously will threre be
contention for the GIL. You can create high-performance TCP servers in
plain Python using e.g. Twisted. If you are in the strange situation
that a TCP server is compute-bound, consider using multiple processes
(os.fork or multiprocessing).


> I think there is no way that we can achieve this because of the GIL
> being a process level state. Least I can do is have one python
> interpreter initialized in main thread and lock the GIL in every
> thread for python calls.

I think you will find out your server is indeed I/O bound, like 99.9%
or all other TCP servers on this planet. Try to embed a single
interpreter first. Use the simplified GIL API I showed you. Most
likely you will find that it suffice.

If you need something more scalable, associate each pthread with a
separate Python process - e.g. using a named pipe on Windows or Unix
domain socket on Linux.