Initialization hook for extenders

I work on several projects that have initialization requirements that need to grab control after Py_Initialize(), but before any user code runs (via input, script, -c, etc...). Note that these are Python clones that take advantage of an installed python (using its $prefix/lib/pythonx.x/*.py and site-packages/*) We could use PyImport_AppendInittab("sitecustomize",initsitecustomize); But if there already IS customization in sitecustomize.py, I've blown it away (and have to look it up and force an import). And if someone uses the -S flag, I'm screwed. I propose a hook styled after Py_AtExit(func) called Py_AtInit(func) which maintains a list of functions that are called in Py_Initialize right after main and site initializations. If the hook isn't used, then the cost is a single extra function call at initialization. Here's a spurious example: A customer wants a version of python that has all the math functions and his extensions to act like builtins... I would write (without refcnt or error checks ;-): #include "Python.h" static void after_init(void) { PyObject *builtin,*builtin_dict,*math,*math_dict,*user,*user_dict; builtin = PyImport_ImportModule("__builtin__"); builtin_dict = PyModule_GetDict(builtin); math = PyImport_ImportModule("math"); math_dict = PyModule_GetDict(math); user = PyImport_ImportModule("user"); user_dict = PyModule_GetDict(math); PyDict_Update(builtin_dictionary, math_dict); PyDict_Update(builtin_dictionary, user_dict); } int main(int argc, char** argv) { PyImport_AppendInittab("user",inituser); Py_AtInit(after_init); return Py_Main(argc, argv); } voila! An extended Python with new builtins. I actually want this to do some MPI initialization to setup a single user prompt with broadcast which has to run after Py_Initialize() but before the import of readline. I've attached a copy of the patch (also going to patches at sf.net) Pat -- Patrick Miller | (925) 423-0309 | http://www.llnl.gov/CASC/people/pmiller Son, when you grow up you will know who I really am. I am just a child like you who has been forced to act responsibly. -- Rod Byrnes Index: dist/src/Include/pythonrun.h =================================================================== RCS file: /cvsroot/python/python/dist/src/Include/pythonrun.h,v retrieving revision 2.62 diff -c -r2.62 pythonrun.h *** dist/src/Include/pythonrun.h 13 Feb 2003 22:07:52 -0000 2.62 --- dist/src/Include/pythonrun.h 30 Apr 2003 22:04:13 -0000 *************** *** 75,80 **** --- 75,81 ---- PyAPI_FUNC(void) PyErr_Display(PyObject *, PyObject *, PyObject *); PyAPI_FUNC(int) Py_AtExit(void (*func)(void)); + PyAPI_FUNC(int) Py_AtInit(void (*func)(void)); PyAPI_FUNC(void) Py_Exit(int); Index: dist/src/Python/pythonrun.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/pythonrun.c,v retrieving revision 2.193 diff -c -r2.193 pythonrun.c *** dist/src/Python/pythonrun.c 22 Apr 2003 11:18:00 -0000 2.193 --- dist/src/Python/pythonrun.c 30 Apr 2003 22:04:16 -0000 *************** *** 106,111 **** --- 106,135 ---- return flag; } + #define NINITFUNCS 32 + static void (*initfuncs[NINITFUNCS])(void); + static int ninitfuncs = 0; + + int Py_AtInit(void (*func)(void)) + { + if (ninitfuncs >= NINITFUNCS) + return -1; + if (!func) + return -1; + initfuncs[ninitfuncs++] = func; + return 0; + } + + static void initinitialize(void) + { + int i; + for(i=0;i<ninitfuncs;++i) { + initfuncs[i](); + if (PyErr_Occurred()) + Py_FatalError("Py_AtInit: initialization error"); + } + } + void Py_Initialize(void) { *************** *** 182,190 **** --- 206,217 ---- initsigs(); /* Signal handling stuff, including initintr() */ initmain(); /* Module __main__ */ + if (!Py_NoSiteFlag) initsite(); /* Module site */ + initinitialize(); /* Extenstion hooks */ + /* auto-thread-state API, if available */ #ifdef WITH_THREAD _PyGILState_Init(interp, tstate); *************** *** 1418,1423 **** --- 1445,1451 ---- #endif /* MS_WINDOWS */ abort(); } + /* Clean up and exit */

"Martin v. Löwis" wrote:
It's not easy. Not if you simply want to link against an installed Python. Nor so if you want to build against 2.1 2.2 and 2.3 ... libraries. There are subtle changes that bite you in the ass if you don't physically copy the right source forward. We did copy forward main.c, but found that every time we updated Python, we had to "rehack" main to make sure we had all the options and flags and initialization straight. I think the hook is extremely cheap, very short, looks almost exactly like Py_AtExit() and solves the problem directly. Pat -- Patrick Miller | (925) 423-0309 | http://www.llnl.gov/CASC/people/pmiller You can discover more about a person in an hour of play than in a year of discussion. -- Plato, philosopher (427-347 BCE)

Patrick J. Miller wrote:
It's not easy.
Not if you simply want to link against an installed Python.
Why not? Just don't call the function Py_Main.
Nor so if you want to build against 2.1 2.2 and 2.3 ... libraries.
Again, I can't see a reason why that is.
There are subtle changes that bite you in the ass if you don't physically copy the right source forward.
For example?
That is not necessary. What would be the problem if you just left your function as it was in Python 2.1?
I think the hook is extremely cheap, very short, looks almost exactly like Py_AtExit() and solves the problem directly.
Unfortunately, the problem is one that almost nobody ever has, and supporting that API adds a maintenance burden. It is better if the maintenance burden is on your side than on the Python core. If you think you really need this, write a PEP, ask the community, and wait for BDFL pronouncement. I'm still -1. Regards, Martin

Martin, Sorry you disagree. I think that the issue is still important and other pieces of the API are already in this direction. For instance, there is no need to have PyImport_AppendInittab because you can hack config.c (which you can get from $prefix/lib/pythonx.x/config/config.c) and in fact many people did exactly that but it made for a messy extension until the API call made it clean and direct. You don't need Py_AtExit() because you can call through to atexit.register() to put the function in. The list goes on... I still think that Py_AtInit() is clean, symmetric with Py_AtExit(), and solves a big problem for extenders who wish to address localization from within C (as opposed to sitecustomize.py). This is a 10 line patch with 0 runtime impact that requires no maintanence to move forward with new versions. If it were more than that, I could better understand your objections. Hope that I can get you to at least vote 0 instead of -1. Cheers, Pat -- Patrick Miller | (925) 423-0309 | http://www.llnl.gov/CASC/people/pmiller You can never solve a problem on the level on which it was created. -- Albert Einstein, physicist, Nobel laureate (1879-1955)

"Martin v. Löwis" wrote:
It's not easy. Not if you simply want to link against an installed Python. Nor so if you want to build against 2.1 2.2 and 2.3 ... libraries. There are subtle changes that bite you in the ass if you don't physically copy the right source forward. We did copy forward main.c, but found that every time we updated Python, we had to "rehack" main to make sure we had all the options and flags and initialization straight. I think the hook is extremely cheap, very short, looks almost exactly like Py_AtExit() and solves the problem directly. Pat -- Patrick Miller | (925) 423-0309 | http://www.llnl.gov/CASC/people/pmiller You can discover more about a person in an hour of play than in a year of discussion. -- Plato, philosopher (427-347 BCE)

Patrick J. Miller wrote:
It's not easy.
Not if you simply want to link against an installed Python.
Why not? Just don't call the function Py_Main.
Nor so if you want to build against 2.1 2.2 and 2.3 ... libraries.
Again, I can't see a reason why that is.
There are subtle changes that bite you in the ass if you don't physically copy the right source forward.
For example?
That is not necessary. What would be the problem if you just left your function as it was in Python 2.1?
I think the hook is extremely cheap, very short, looks almost exactly like Py_AtExit() and solves the problem directly.
Unfortunately, the problem is one that almost nobody ever has, and supporting that API adds a maintenance burden. It is better if the maintenance burden is on your side than on the Python core. If you think you really need this, write a PEP, ask the community, and wait for BDFL pronouncement. I'm still -1. Regards, Martin

Martin, Sorry you disagree. I think that the issue is still important and other pieces of the API are already in this direction. For instance, there is no need to have PyImport_AppendInittab because you can hack config.c (which you can get from $prefix/lib/pythonx.x/config/config.c) and in fact many people did exactly that but it made for a messy extension until the API call made it clean and direct. You don't need Py_AtExit() because you can call through to atexit.register() to put the function in. The list goes on... I still think that Py_AtInit() is clean, symmetric with Py_AtExit(), and solves a big problem for extenders who wish to address localization from within C (as opposed to sitecustomize.py). This is a 10 line patch with 0 runtime impact that requires no maintanence to move forward with new versions. If it were more than that, I could better understand your objections. Hope that I can get you to at least vote 0 instead of -1. Cheers, Pat -- Patrick Miller | (925) 423-0309 | http://www.llnl.gov/CASC/people/pmiller You can never solve a problem on the level on which it was created. -- Albert Einstein, physicist, Nobel laureate (1879-1955)
participants (3)
-
"Martin v. Löwis"
-
martin@v.loewis.de
-
Patrick J. Miller