Module local namespaces
A while ago, I developed a small PyGtk programme that could dynamically reload all the working callbacks and logic while the GUI was still running. I could get away with this because of the flexible way Python loads modules at runtime, but it ended up being a waste of time as implementing it took more time that actually using it. For sanity's sake it quickly becomes clear you almost never want to rely on being able to refer to a half initialized module. And wouldn't it be nice if Python enforced this. My suggestion is that module importing occur in a temporary local namespace that exists only until the end of the module code is executed, then a small function could copy everything from the temporary namespace into the module object. The usual closure semantics would guarantee that top-level functions could still call each other, but they would effectively become immutable after the namespace wraps up. The 'global' keyword could be used at the top level in a module to force it to be defined in the module immediately, and to ensure internal references to the object go through the module object. This would be a big change in module import semantics, but should have remarkably few consequences, as it really is an enforcement mechanism for good style. The copying from the temporary namespace into the module object would be a good place to insert a hook function to filter what objects are actually published to the module. You could by default not copy any object indentified by a leading underscore.
On 1/3/07, Matt Draisey <matt@draisey.ca> wrote:
A while ago, I developed a small PyGtk programme that could dynamically reload all the working callbacks and logic while the GUI was still running. I could get away with this because of the flexible way Python loads modules at runtime, but it ended up being a waste of time as implementing it took more time that actually using it. For sanity's sake it quickly becomes clear you almost never want to rely on being able to refer to a half initialized module. And wouldn't it be nice if Python enforced this.
How are you having an issue with a partially initialized module? The only way I can see that happening is that you have a circular import dependency where both modules want to execute code that the other one has. And if that happens the answer for that is "don't do it". My suggestion is that module importing occur in a temporary local
namespace that exists only until the end of the module code is executed, then a small function could copy everything from the temporary namespace into the module object. The usual closure semantics would guarantee that top-level functions could still call each other, but they would effectively become immutable after the namespace wraps up.
But why would you want it to be immutable? Being able to change the function in a module and have all uses of it also change can be handy. The 'global'
keyword could be used at the top level in a module to force it to be defined in the module immediately, and to ensure internal references to the object go through the module object.
This would be a big change in module import semantics, but should have remarkably few consequences, as it really is an enforcement mechanism for good style. The copying from the temporary namespace into the module object would be a good place to insert a hook function to filter what objects are actually published to the module. You could by default not copy any object indentified by a leading underscore.
Private namespaces are not exactly a popular thing in Python. =) -Brett
Matt Draisey <matt@draisey.ca> wrote: [snip] -1 . The only thing that possibly should be fixed is that modules should only be inserted into sys.modules after they have been imported completely and correctly.
This would be a big change in module import semantics, but should have remarkably few consequences, as it really is an enforcement mechanism for good style.
How is it an enforcement for good style? If I can do exactly what I'm doing now, then it isn't an enforcement mechanism in any way. Regardless, the way to handle not allowing the rest of your program to muck with partially initialized modules is to not use the standard import machinery. There are various ways of emulating module loading, many of which can allow you to insert the module into sys.modules *after* the module was loaded completely correctly. I actually use one of these methods to handle the dynamic loading and reloading of Python macros in a source code editor, and none of the macros are available in sys.modules .
The copying from the temporary namespace into the module object would be a good place to insert a hook function to filter what objects are actually published to the module. You could by default not copy any object indentified by a leading underscore.
You can already do this with the following code: __gl = globals() for name in __gl.keys(): if name[:1] == '_' and len(name) > 1 and name.count('_') == 1: del __gl[name] del __gl - Josiah
On 1/3/07, Josiah Carlson <jcarlson@uci.edu> wrote:
-1 . The only thing that possibly should be fixed is that modules should only be inserted into sys.modules after they have been imported completely and correctly.
I agree, but there's the wrinkle that during recursive imports you want partially-imported modules to be importable. I believe we once agreed on a solution but I don't recall what it was. Does anybody remember? -- --Guido van Rossum (home page: http://www.python.org/~guido/)
"Guido van Rossum" <guido@python.org> wrote:
On 1/3/07, Josiah Carlson <jcarlson@uci.edu> wrote:
-1 . The only thing that possibly should be fixed is that modules should only be inserted into sys.modules after they have been imported completely and correctly.
I agree, but there's the wrinkle that during recursive imports you want partially-imported modules to be importable. I believe we once agreed on a solution but I don't recall what it was. Does anybody remember?
I don't remember what was agreed upon, but what about something like... try: newmodule = ... sys.modules[name] = newmodule handle_the_import(newmodule, ...) except: del sys.modules[name] raise - Josiah
On 1/3/07, Josiah Carlson <jcarlson@uci.edu> wrote:
"Guido van Rossum" <guido@python.org> wrote:
On 1/3/07, Josiah Carlson <jcarlson@uci.edu> wrote:
-1 . The only thing that possibly should be fixed is that modules should only be inserted into sys.modules after they have been imported completely and correctly.
I agree, but there's the wrinkle that during recursive imports you want partially-imported modules to be importable. I believe we once agreed on a solution but I don't recall what it was. Does anybody remember?
I don't remember what was agreed upon, but what about something like...
try: newmodule = ... sys.modules[name] = newmodule handle_the_import(newmodule, ...) except: del sys.modules[name] raise
My re-implementation does exactly that (I just sometimes postpone the module creation to the top of the handle_the_import function). -Brett
On Wed, 2007-01-03 at 17:17 -0800, Josiah Carlson wrote:
You can already do this with the following code:
__gl = globals() for name in __gl.keys(): if name[:1] == '_' and len(name) > 1 and name.count('_') == 1: del __gl[name] del __gl
- Josiah
No, that is not what I meant. I wasn't talking about deleting temporaries but not publishing private objects. Brett Cannon understood when he said, "Private namespaces are not exactly a popular thing in Python". But functional programming style and generators are all the rage and they often drag in private state via a closure.
On 1/3/07, Matt Draisey <matt@draisey.ca> wrote:
On Wed, 2007-01-03 at 17:17 -0800, Josiah Carlson wrote:
You can already do this with the following code:
__gl = globals() for name in __gl.keys(): if name[:1] == '_' and len(name) > 1 and name.count('_') == 1: del __gl[name] del __gl
- Josiah
No, that is not what I meant. I wasn't talking about deleting temporaries but not publishing private objects. Brett Cannon understood when he said, "Private namespaces are not exactly a popular thing in Python". But functional programming style and generators are all the rage and they often drag in private state via a closure.
True, but you can still get at everything in either (generators, for instance, expose the paused execution frame and gives you access to everything). But I consider these more of an exception instead of the rule. The only way I can see myself approving something like this is if it helped with security (which it might, but I don't know if it would be the best solution). -Brett
participants (4)
-
Brett Cannon -
Guido van Rossum -
Josiah Carlson -
Matt Draisey