[Help Request] Embedding Python in a CPP Application Responsibly & Functionally

[John McCardle]

Greetings,

I'm working on embedding a Python interpreter into a C++ application. My 
embedding example program is here, largely taken from Python docs: 
https://gist.github.com/jmccardle/f3f19d3753ae023aa52b927f0d181c43

I'm simply not interested in writing in Lua, so regardless of any 
particular downsides like `sys` in the standard library or performance 
issues, I'm committed to Python itself as what I want to hack in. This 
is for fun.

I started by compiling Python:

`./configure --enable-shared --enable-optimizations`

Then I can compile my example embedded program:

`g++ -I Python-3.11.1/Include -I Python-3.11.1 -L Python-3.11.1 -pthread 
scripting_engine.cpp libpython3.11.a -o scripteng -lm -ldl -lutil`

This is working not so bad! I can control what C++ functionality is 
exposed and I seemingly don't need anything but the Python shared object 
to execute. But the finer details of making this work truly correctly 
are eluding me.

1) To get the compiled Python to run independently, I have to hack 
LD_LIBRARY_PATH to get it to execute. `LD_LIBRARY_PATH=./Python-3.11.1 
./Python-3.11.1/python` . Even when trying to execute from the same 
directory as the binary & executable, I get an error, `/python: error 
while loading shared libraries: libpython3.11.so.1.0: cannot open shared 
object file: No such file or directory`.

2) When running the C++ program that embeds Python, I see these messages 
after initializing:
`Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>`

This is seemingly connected to some issues regarding libraries: When I 
run the Python interpreter directly, I can get some of the way through 
the process of creating a virtual environment, but it doesn't seem to 
leave me with a working pip:

`$ LD_LIBRARY_PATH=./Python-3.11.1 ./Python-3.11.1/python
 >>> import venv
 >>> venv.create("./venv", with_pip=True)
subprocess.CalledProcessError: Command 
'['/home/john/Development/7DRL/cpp_embedded_python/venv/bin/python', 
'-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit 
status 127.`

Meanwhile, if I try to run a script from the C++ program that includes 
`import venv`, I get a traceback about a platform library:

`Traceback (most recent call last):
   File "engine_user.py", line 7, in <module>
     import venv
   File 
"/home/john/Development/7DRL/cpp_embedded_python/Python-3.11.1/Lib/venv/__init__.py", 
line 10, in <module>
     import subprocess
   File 
"/home/john/Development/7DRL/cpp_embedded_python/Python-3.11.1/Lib/subprocess.py", 
line 104, in <module>
     from _posixsubprocess import fork_exec as _fork_exec
ModuleNotFoundError: No module named '_posixsubprocess'
`

3) I'm not sure I even need to be statically linking the interpreter.

My desired end state is this:
* Deploy a C++ program that doesn't rely on a system Python. I'm not 
sure if I need just the shared object / DLLs, or a Python executable in 
a subdirectory - I'd like to "do it the right way".
* C++ program can run a script to create a virtual environment, which 
the embedded Python environment will use. Users can activate the venv 
like any other Python environment and install packages with pip.
* ideally, some sort of "inside-out" runnable mode, where the API 
exposed by the C++ executable is available in that venv, so that I can 
test a script in Thonny or other IDE. I think I'd do this by providing a 
separate test-mode library in the venv, and when C++ executes 
`PyImport_AppendInittab("scriptable", &PyInit_scriptable);` then the 
module of the same name should be overwritten with the C++ program's 
functionality.

I've been through the embedded programming docs a bit, and they seem 
quite good as a reference, but I don't know what I'm doing well enough 
to solve my problems using them. Thanks for reading.

My ultimate goal is to expose features written in C++ for a game engine 
using SFML, and run .py files in a subdirectory to generate maps, 
control NPC dialogue and actions, etc. I'm hoping to have something 
usable for this year's 7DRL, which starts March 4th. I'd like to spend 
the time until then getting this engine working smoothly and porting it 
to Windows, so I can focus exclusively on "game content" for the 
timeboxed 7-day portion.

Kind Regards,
-John McCardle