I find the current situation when 'embedding' Python rather unsatisfying. 1. Careful preparation (hacking?) of environment variables is needed to avoid that Py_Initialize() takes the default actions needed for the standard Python interpreter (for example, to avoid the default sys.path). 2. A suggestion from /F on c.l.p was to change pythonw.exe so that instead of hiding all output to stderr a console window would be opened to show the tracebacks. This requires to construct an object with a 'write' method doing all this magic, and it must be done after the call to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the code of Py_Main(), it seems impossible without rewriting most of the code. Possible improvements I currently can think of are: - Provide a function like Py_InitializeEx(), which accepts parameters specifying an initial sys.path, optimize flag, verbose flag, and so on. - Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), or let Py_Main() accept a callback function which will be called jsut after Py_Initialize() has been run. I'm not ready to write a PEP, but if there are people sharing this feeling I would suggest we create a wiki entry somewhere acting as a temporary brainstorming area. Thomas
I find the current situation when 'embedding' Python rather unsatisfying.
I'm interested in improve it. Can you explain what exactly your 'embedding' requirements are?
1. Careful preparation (hacking?) of environment variables is needed to avoid that Py_Initialize() takes the default actions needed for the standard Python interpreter (for example, to avoid the default sys.path).
I guess it depends on the embedding needs. Some embedded uses would be quite happy with the default sys.path.
2. A suggestion from /F on c.l.p was to change pythonw.exe so that instead of hiding all output to stderr a console window would be opened to show the tracebacks. This requires to construct an object with a 'write' method doing all this magic, and it must be done after the call to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the code of Py_Main(), it seems impossible without rewriting most of the code.
I'm not sure what this has to do with embedding -- can you clarify?
Possible improvements I currently can think of are:
- Provide a function like Py_InitializeEx(), which accepts parameters specifying an initial sys.path, optimize flag, verbose flag, and so on.
Except for sys.path, you can set most flags directly before calling Py_Initialize(), so I'm not sure if there's a need to pass these into Py_Initialize().
- Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), or let Py_Main() accept a callback function which will be called jsut after Py_Initialize() has been run.
OK, now I'm confused. If you're embedding Python, why would you be using Py_Main() at all?
I'm not ready to write a PEP, but if there are people sharing this feeling I would suggest we create a wiki entry somewhere acting as a temporary brainstorming area.
Go ahead, but count me out of the wiki -- wikis don't work for discussions for me. --Guido van Rossum (home page: http://www.python.org/~guido/)
[Thomas Heller]
I find the current situation when 'embedding' Python rather unsatisfying.
[Guido van Rossum]
I'm interested in improve it. Can you explain what exactly your 'embedding' requirements are?
see below.
1. Careful preparation (hacking?) of environment variables is needed to avoid that Py_Initialize() takes the default actions needed for the standard Python interpreter (for example, to avoid the default sys.path).
I guess it depends on the embedding needs. Some embedded uses would be quite happy with the default sys.path.
Sure, but others not.
Possible improvements I currently can think of are:
- Provide a function like Py_InitializeEx(), which accepts parameters specifying an initial sys.path, optimize flag, verbose flag, and so on.
Except for sys.path, you can set most flags directly before calling Py_Initialize(), so I'm not sure if there's a need to pass these into Py_Initialize().
I want to specify the Python path before calling Py_Initialize(), and I don't want it to use *any* environment variables which (again) set the flags. So I thought of Py_InitializeEx() which would take parameters specifying these things, and Py_Initialize() could be changed to call Py_InitializeEx(NULL), maybe.
2. A suggestion from /F on c.l.p was to change pythonw.exe so that instead of hiding all output to stderr a console window would be opened to show the tracebacks. This requires to construct an object with a 'write' method doing all this magic, and it must be done after the call to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the code of Py_Main(), it seems impossible without rewriting most of the code.
I'm not sure what this has to do with embedding -- can you clarify?
The Python interpreter (dll) embedded into pythonw.exe ;-)
- Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), or let Py_Main() accept a callback function which will be called jsut after Py_Initialize() has been run.
OK, now I'm confused. If you're embedding Python, why would you be using Py_Main() at all?
I'm probably not good at explaining these things. I was thinking of pythonw.exe (WinMain.c). I want to create a Python object (an stderr writer), and assign this to sys.stderr, without changing too much code. Does it make more sense now? Thomas
[Thomas Heller]
I find the current situation when 'embedding' Python rather unsatisfying.
[Guido van Rossum]
I'm interested in improve it. Can you explain what exactly your 'embedding' requirements are?
see below.
The only clue I find below is that you're trying to inmprove the error reporting of pythonw.exe, which is certainly laudable, but hardly qualifies as "embedding" IMO.
1. Careful preparation (hacking?) of environment variables is needed to avoid that Py_Initialize() takes the default actions needed for the standard Python interpreter (for example, to avoid the default sys.path).
I guess it depends on the embedding needs. Some embedded uses would be quite happy with the default sys.path.
Sure, but others not.
But the only use *you* are interested in is pythonw.exe?
Possible improvements I currently can think of are:
- Provide a function like Py_InitializeEx(), which accepts parameters specifying an initial sys.path, optimize flag, verbose flag, and so on.
Except for sys.path, you can set most flags directly before calling Py_Initialize(), so I'm not sure if there's a need to pass these into Py_Initialize().
I want to specify the Python path before calling Py_Initialize(), and I don't want it to use *any* environment variables which (again) set the flags. So I thought of Py_InitializeEx() which would take parameters specifying these things, and Py_Initialize() could be changed to call Py_InitializeEx(NULL), maybe.
If you don't want the environment to be used, set the global Py_IgnoreEnvironmentFlag before calling Py_Initialize().
2. A suggestion from /F on c.l.p was to change pythonw.exe so that instead of hiding all output to stderr a console window would be opened to show the tracebacks. This requires to construct an object with a 'write' method doing all this magic, and it must be done after the call to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the code of Py_Main(), it seems impossible without rewriting most of the code.
I'm not sure what this has to do with embedding -- can you clarify?
The Python interpreter (dll) embedded into pythonw.exe ;-)
- Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), or let Py_Main() accept a callback function which will be called jsut after Py_Initialize() has been run.
OK, now I'm confused. If you're embedding Python, why would you be using Py_Main() at all?
I'm probably not good at explaining these things.
I was thinking of pythonw.exe (WinMain.c). I want to create a Python object (an stderr writer), and assign this to sys.stderr, without changing too much code.
Does it make more sense now?
Not quite -- I don't understand why *you* don't want the environment variables to be used, if all you want is a better pythonw.exe. If that's your only experience with embedding Python, you can't really speak for other embedders. --Guido van Rossum (home page: http://www.python.org/~guido/)
I want to specify the Python path before calling Py_Initialize(), and I don't want it to use *any* environment variables which (again) set the flags. So I thought of Py_InitializeEx() which would take parameters specifying these things, and Py_Initialize() could be changed to call Py_InitializeEx(NULL), maybe.
If you don't want the environment to be used, set the global Py_IgnoreEnvironmentFlag before calling Py_Initialize().
I didn't know that, thanks for the hint. [...]
Does it make more sense now?
Not quite -- I don't understand why *you* don't want the environment variables to be used, if all you want is a better pythonw.exe.
I did probably mix two different things here.
If that's your only experience with embedding Python, you can't really speak for other embedders.
That may be - my other experiences are py2exe and the embedding done in bdist_wininst. Now knowing Py_IgnoreEnvironmentFlag I would be satisfied with the addition of a PySys_SetPath function ;-). Thomas
Guido van Rossum wrote:
Not quite -- I don't understand why *you* don't want the environment variables to be used, if all you want is a better pythonw.exe.
I haven't embedded Python for a while, but I can sympathize with the desire to ignore environment variables. One approach to embedding is to provide a complete (stripped) copy of Python, inside the target directory of the application. You might even directly link with a static python23.lib, and freeze all modules, instead of linking with python23.dll, and providing a copy of all .py files. In such an environment, you don't want a regular Python installation to interfere with your embedded environment. For example, you may have incorporated the library Foo, but the target system has another (older) copy of Foo in PYTHONPATH. If your embedded application now picks up the older copy of Foo, it fails, and all your testing goes to hell (*) since arbitrary pieces of the library might get arbitrarily replaced with something else. So to allow reproducable results, you need a guarantee that nothing on the target system is incorporated in your application. Regards, Martin (*) DLL hell, to be precise.
[Martin von Löwis]
So to allow reproducable results, you need a guarantee that nothing on the target system is incorporated in your application.
I did not closely follow this thread, and apologise if I repeat arguments which have already been said. While I quite understand the argument and needs which Martin expresses, there are situations where one is happy to tune the behaviour of the embedded Python through the usual environment variables. It might depend if the embedding application intends to either expose or hide the existence of Python. If the application offers Python as an extension language for the system implemented by the application, Python is then exposed, and the user should be able to use environment variable to his/her advantage. If the application uses Python internally, all to itself, then the application likely wants to hide even the fact that it uses Python, exerts full control, or at least decide exactly how much or little the user is allowed to interfere, as Martin explained. If it is given that only one of these avenues is favoured, I would for one prefer that the "exposed" direction be the easier to implement, which may be the statu quo over the current situation. The "hiding" direction might be evaluated so it requires a bit more work for the implementor, about deleting or forcing the values of a few sensitive environment variables. Especially given that a Python hiding application is likely to control other environment variables as well (like maybe those related to loader path search, or localisation). -- François Pinard http://www.iro.umontreal.ca/~pinard
Thomas Heller writes:
I find the current situation when 'embedding' Python rather unsatisfying.
Indeed. See patch 849278 for some of the work we did to deal with this.
1. Careful preparation (hacking?) of environment variables is needed to avoid that Py_Initialize() takes the default actions needed for the standard Python interpreter (for example, to avoid the default sys.path).
I've done this in the patches submitted (by a coworker) above. The idea is to separate the embedded interpreter *completely* from the environment.
2. A suggestion from /F on c.l.p was to change pythonw.exe so that instead of hiding all output to stderr a console window would be opened to show the tracebacks. This requires to construct an object with a 'write' method doing all this magic, and it must be done after the call to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the code of Py_Main(), it seems impossible without rewriting most of the code.
Yes, this is a problem with the embedded interpreter where errors may be written to stderr without regard to the calling application. Further, it is possible for the interpreter to call abort(), which is even worse.
Possible improvements I currently can think of are:
- Provide a function like Py_InitializeEx(), which accepts parameters specifying an initial sys.path, optimize flag, verbose flag, and so on.
See the patch.
- Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), or let Py_Main() accept a callback function which will be called jsut after Py_Initialize() has been run.
Yup.
I'm not ready to write a PEP, but if there are people sharing this feeling I would suggest we create a wiki entry somewhere acting as a temporary brainstorming area.
Absolutely: count me in. -tree -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever"
Indeed. See patch 849278 for some of the work we did to deal with this.
This mostly removes the use of the PYTHONXXX environment variables. Can you explain why you don't want these to work? Also note that you can already disable *all* use of the environment (not just in Py_Initialize()) by setting the global Py_IgnoreEnvironmentFlag to a non-zero value. I'd reject that part of the patch for that reason. Py_SetSysPaths() might be a valuable addition, but the game you play with the prefiX name is abject. :)
1. Careful preparation (hacking?) of environment variables is needed to avoid that Py_Initialize() takes the default actions needed for the standard Python interpreter (for example, to avoid the default sys.path).
I've done this in the patches submitted (by a coworker) above. The idea is to separate the embedded interpreter *completely* from the environment.
2. A suggestion from /F on c.l.p was to change pythonw.exe so that instead of hiding all output to stderr a console window would be opened to show the tracebacks. This requires to construct an object with a 'write' method doing all this magic, and it must be done after the call to Py_Initialize() and before PyRun_SimpleFile(). Now, looking at the code of Py_Main(), it seems impossible without rewriting most of the code.
Yes, this is a problem with the embedded interpreter where errors may be written to stderr without regard to the calling application. Further, it is possible for the interpreter to call abort(), which is even worse.
What do you suggest the interpreter should do instead of calling abort()? This is only called in response to *fatal* errors -- situations where raising an exception is *not* an acceptable alternative for various reasons (maybe because it's in a piece of code that is part of the implementation of exceptions, or maybe because a corruption is detected in the very interpreter environment that's needed to raise an exception). Anout the only alternative I can think of would be a longjmp() call to a handler you have set up, but that can't be safe when the fatal error is discovered in a different thread.
Possible improvements I currently can think of are:
- Provide a function like Py_InitializeEx(), which accepts parameters specifying an initial sys.path, optimize flag, verbose flag, and so on.
See the patch.
- Split Py_Main() into 2 functions Py_Main_Prepare() and Py_Main_Run(), or let Py_Main() accept a callback function which will be called just after Py_Initialize() has been run.
Yup.
What's your embedding environment? Why do you want to use Py_Main() at all? --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum writes:
Indeed. See patch 849278 for some of the work we did to deal with this.
This mostly removes the use of the PYTHONXXX environment variables. Can you explain why you don't want these to work?
If the user installs our application and has a separate Python installation on their machine for which they use those environment variables, bad things can happen. I do not want to be in a position where I'm told our app doesn't work because of environment variables we have no control over, and it is the Wrong Answer to tell the user to change their environment.
Also note that you can already disable *all* use of the environment (not just in Py_Initialize()) by setting the global Py_IgnoreEnvironmentFlag to a non-zero value. I'd reject that part of the patch for that reason.
I did not know about or notice this flag when I made the changes --- in all the places I looked at the time the environment variables were read without such a check, at least as I remember. It certainly makes more sense to use the built in support.
Py_SetSysPaths() might be a valuable addition, but the game you play with the prefiX name is abject. :)
Sure, whatever. These were made for our own use and later submitted, They can be cleaned up as needed.
What do you suggest the interpreter should do instead of calling abort()? This is only called in response to *fatal* errors -- situations where raising an exception is *not* an acceptable alternative for various reasons (maybe because it's in a piece of code that is part of the implementation of exceptions, or maybe because a corruption is detected in the very interpreter environment that's needed to raise an exception).
During initialization if there is a fatal error I can just disable the functionality that uses the interpreter --- having my app abort because Python cannot initialize for whatever reason is a Bad Thing. Similarly with writing messages directly to stderr: I do not necessarily want our users to see these since they don't know what to do with them.
Anout the only alternative I can think of would be a longjmp() call to a handler you have set up, but that can't be safe when the fatal error is discovered in a different thread.
In our use of Python we build without threads, so the longjmp() would work. In the case of threads all bets are off.
What's your embedding environment? Why do you want to use Py_Main() at all?
I don't use Py_Main() in my embedding (multiple Unix flavors, 32- and 64-bit, and Win32 and Win64)... I wasn't reading closely enough. -tree -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever"
This mostly removes the use of the PYTHONXXX environment variables. Can you explain why you don't want these to work?
If the user installs our application and has a separate Python installation on their machine for which they use those environment variables, bad things can happen. I do not want to be in a position where I'm told our app doesn't work because of environment variables we have no control over, and it is the Wrong Answer to tell the user to change their environment.
Fair enough. (Though you could have built your app to work even if Python wrote some random stuff to stderr or optimized its bytecode.)
Also note that you can already disable *all* use of the environment (not just in Py_Initialize()) by setting the global Py_IgnoreEnvironmentFlag to a non-zero value. I'd reject that part of the patch for that reason.
I did not know about or notice this flag when I made the changes --- in all the places I looked at the time the environment variables were read without such a check, at least as I remember. It certainly makes more sense to use the built in support.
Perhaps it's easy to overlook that it's using Py_GETENV() rather than getenv(); this macro takes tests for Py_IgnoreEnvironmentFlag first.
Py_SetSysPaths() might be a valuable addition, but the game you play with the prefiX name is abject. :)
Sure, whatever. These were made for our own use and later submitted, They can be cleaned up as needed.
Please do. Add some docs too -- undocumented new code isn't very welcome, since we've already got enough of that. :-)
What do you suggest the interpreter should do instead of calling abort()? This is only called in response to *fatal* errors -- situations where raising an exception is *not* an acceptable alternative for various reasons (maybe because it's in a piece of code that is part of the implementation of exceptions, or maybe because a corruption is detected in the very interpreter environment that's needed to raise an exception).
During initialization if there is a fatal error I can just disable the functionality that uses the interpreter --- having my app abort because Python cannot initialize for whatever reason is a Bad Thing. Similarly with writing messages directly to stderr: I do not necessarily want our users to see these since they don't know what to do with them.
Anout the only alternative I can think of would be a longjmp() call to a handler you have set up, but that can't be safe when the fatal error is discovered in a different thread.
In our use of Python we build without threads, so the longjmp() would work. In the case of threads all bets are off.
So that's an indirect answer to my question: you'd be happy if instead of calling abort() it would longjmp() to a location you specified? I guess a better (more general) solution would be to have a callback function that's called instead of abort(), and to initialize this with something that calls abort() by default. That's a useful feature; if you upload it to SF with reference to this thread it ought to be accepted. --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum
(not just in Py_Initialize()) by setting the global Py_IgnoreEnvironmentFlag to a non-zero value. I'd reject that part of the patch for that reason.
I did not know about or notice this flag when I made the changes --- in all the places I looked at the time the environment variables were read without such a check, at least as I remember. It certainly makes more sense to use the built in support.
Perhaps it's easy to overlook that it's using Py_GETENV() rather than getenv(); this macro takes tests for Py_IgnoreEnvironmentFlag first.
Py_IgnoreEnvironmentFlag and the -E command line option are both /fairly/ new... 2.2 era? Cheers, mwh -- Worryingly, DEFUN appears to be a function that removes all the fun from something: after using it all your code is converted to C++. -- Tim Bradshaw, comp.lang.lisp
Perhaps it's easy to overlook that it's using Py_GETENV() rather than getenv(); this macro takes tests for Py_IgnoreEnvironmentFlag first.
Py_IgnoreEnvironmentFlag and the -E command line option are both /fairly/ new... 2.2 era?
Yeah, but the patch he suggested was using Py_GETENV(), so it's not like it didn't exist when they did the patch; they must have missed its significance. :-) --Guido van Rossum (home page: http://www.python.org/~guido/)
Guido van Rossum writes:
Perhaps it's easy to overlook that it's using Py_GETENV() rather than getenv(); this macro takes tests for Py_IgnoreEnvironmentFlag first.
Py_IgnoreEnvironmentFlag and the -E command line option are both /fairly/ new... 2.2 era?
Yeah, but the patch he suggested was using Py_GETENV(), so it's not like it didn't exist when they did the patch; they must have missed its significance. :-)
Yes, *they* did miss the significance of the use of the macro. -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever"
participants (6)
-
François Pinard -
Guido van Rossum -
Martin v. Loewis -
Michael Hudson -
Thomas Heller -
Tom Emerson