[capi-sig] Embedded Python in C application

[Eljay Love-Jensen]

Hi everyone,

First, my apologies if I'm in the wrong forum for my "embedding Python in a
C application" questions.  Please redirect me if I've wandered into the
wrong place.

I have two needs for using Python in my application that I hope has an easy
answer without rewriting Python's internals.

I need to use Python* in a multi-threaded application, where separate
threads may be working on very long lasting Python scripts, and other
threads may be involved in short Python scripts.  None of the Python scripts
running concurrently have any shared state with any of the other Python
scripts running concurrently.  Number of threads is in the 100-1000 range.

I need to manage Python's use of the heap by providing a memory pool for
Python to use, rather than allowing Python to use malloc/free.  This is to
prevent memory fragmentation, and to allow easy disposal of a memory pool
used for a closed Python interpreter instance.

A quick view of Py_Initialize() indicates that Python does not return some
sort of "Py_State" pointer which represents the entire state of a Python
interpreter.  (Nor some sort of Py_Alloc().)  Nor accepts a custom
malloc/free function pointers.  Hmmm.

Does anyone have experience with using Python in this fashion?

(If relevant, it will be Python 3.x not Python 2.x.)

Thanks,
--Eljay

* Doesn't HAVE to be Python.  Could be JavaScript or Lua or whatnot.  My
preferences these days is a Python solution.