[capi-sig] Embedded Python in C application

[Adam Olsen]

On Fri, Sep 26, 2008 at 9:16 AM, Eljay Love-Jensen <eljay at adobe.com> wrote:
> Hi everyone,
>
> First, my apologies if I'm in the wrong forum for my "embedding Python in a
> C application" questions.  Please redirect me if I've wandered into the
> wrong place.
>
> I have two needs for using Python in my application that I hope has an easy
> answer without rewriting Python's internals.
>
> I need to use Python* in a multi-threaded application, where separate
> threads may be working on very long lasting Python scripts, and other
> threads may be involved in short Python scripts.  None of the Python scripts
> running concurrently have any shared state with any of the other Python
> scripts running concurrently.  Number of threads is in the 100-1000 range.
>
> I need to manage Python's use of the heap by providing a memory pool for
> Python to use, rather than allowing Python to use malloc/free.  This is to
> prevent memory fragmentation, and to allow easy disposal of a memory pool
> used for a closed Python interpreter instance.
>
> A quick view of Py_Initialize() indicates that Python does not return some
> sort of "Py_State" pointer which represents the entire state of a Python
> interpreter.  (Nor some sort of Py_Alloc().)  Nor accepts a custom
> malloc/free function pointers.  Hmmm.
>
> Does anyone have experience with using Python in this fashion?

Don't use multiple interpreters.  They're not really separate, they're
buggy, they offer *NO* advantage to you over just using multiple
threads.

Likewise, you can't force memory to be freed, as it'd still be used by python.

The only way to force cleanup is to spawn a subprocess.  This'd also
let you use multiple cores.  You can probably mitigate the startup
cost by having a given subprocess run several short scripts or one
long script.


-- 
Adam Olsen, aka Rhamphoryncus