Explicitly shared objects with sub modules vs import
While trying to debug a problem and thinking that it may be an issue with circular imports, I come up with an interesting idea that might be of value. It wasn't a circular import problem in this case, but I may have found the actual bug sooner if I didn't need to be concerned about that possibility. I have had some difficulty splitting larger modules into smaller modules in the past where if I split the code by functionality, it doesn't correspond with how the code is organized by dependency. The result is an imported module needs to import the module it's imported into. Which just doesn't feel right to me. The solution I found was to call a function to explicitly set the shared items in the imported module. (The example is from a language I'm experimenting with written in python. So don't be concerned about the shared object names in this case.) In the main module... import parse parse.set_main(List=List, Keyword=Keyword, Name=Name, String=String, Express=Express, keywords=keywords, raise_with=raise_with, nil=nil) And in parse... # Sets shared objects from main module. from collections import namedtuple def set_main(**d): global main main = namedtuple(__name__, d.keys()) for k, v in d.items(): setattr(main, k, v) After this, the sub module access's the parent modules objects with... main.Keyword Just the same as if the parent module was imported as main, but it only shares what is intended to be shared within this specific imported module. I think that is better than using "import from" in the sub module. And an improvement over importing the whole module which can possibly expose too much. The benifits: * The shared items are explicitly set by the parent module. * If an item is missing, it results in a nice error message * Accessing objects works the same as if import was used. * It avoids (most) circular import problems. * It's easier to think about once you understand what it does. The problem is the submodule needs a function to make it work. I think it would be nice if it could be made a builtin but doing that may be tricky. Where I've used "main", it could set the name of the shared parent module(s) automatically. The name of the function probably should be "shared" or "sharing". (Or some other thing that makes sense.) I would like to hear what other here think, and of course if there are any obvious improvements that can be made. Would this be a good candidate for a new builtin? Cheers, Ron
On 30.05.15 18:45, Ron Adam wrote:
While trying to debug a problem and thinking that it may be an issue with circular imports, I come up with an interesting idea that might be of value. It wasn't a circular import problem in this case, but I may have found the actual bug sooner if I didn't need to be concerned about that possibility.
I have had some difficulty splitting larger modules into smaller modules in the past where if I split the code by functionality, it doesn't correspond with how the code is organized by dependency. The result is an imported module needs to import the module it's imported into. Which just doesn't feel right to me.
The solution I found was to call a function to explicitly set the shared items in the imported module.
Why not move all shared objects in common module? Then in both main and parse module you can write from common import *
On 05/30/2015 12:13 PM, Serhiy Storchaka wrote:
On 30.05.15 18:45, Ron Adam wrote:
While trying to debug a problem and thinking that it may be an issue with circular imports, I come up with an interesting idea that might be of value. It wasn't a circular import problem in this case, but I may have found the actual bug sooner if I didn't need to be concerned about that possibility.
I have had some difficulty splitting larger modules into smaller modules in the past where if I split the code by functionality, it doesn't correspond with how the code is organized by dependency. The result is an imported module needs to import the module it's imported into. Which just doesn't feel right to me.
The solution I found was to call a function to explicitly set the shared items in the imported module.
Why not move all shared objects in common module? Then in both main and parse module you can write
from common import *
As I said, sometimes I prefer to organise things by function rather than dependency. The point is this fits a somewhat different pattern than when you have independent common objects. These can be inter-dependent shared objects that would require a circular imports. So common may need an "import __main__ as main" in order for the items that are imported with "import *" to work. One argument might be the organisation of the code is wrong if that is needed, or the may be a better way to organise it. While that is a valid point, it may not be the only factor involved in deciding how to organise the code. I also like to avoid "import *" except when importing very general and common utility functions. ie.. "from math import *". Cheers, Ron
On 05/30/2015 11:45 AM, Ron Adam wrote:
The solution I found was to call a function to explicitly set the shared items in the imported module.
A bit of an improvement... def export_to(module, **d): """ Explitely share objects with imported module. Use this_module.item in the sub-module after item is exported to it. """ from collections import namedtuple namespace = namedtuple("exported", d.keys()) for k, v in d.items(): setattr(namespace, k, v) # Not sure about this. Possibly sys.get_frame would be better. setattr(module, __loader__.name, namespace) And used like this. import sub_mod export_to(sub_mod, foo=foo, bar=bar) Then functions in sub-mod can access the objects as if the sub-module imported the parent module, but only the exported items are visible to the sub module. Again, this is for closely dependent modules that can't easily be split by moving common objects into a mutually imported file, or if it is desired to split a larger module by functionality rather than dependency. There are some limitations, but I think they are actually desirable features. The sub-module can't use exported objects at the top level, and it can't alter the parent modules name space directly. Of course, it could just be my own preferences. I like the pattern of control (the specifying of what gets imported/shared) flowing from the top down. Cheers, Ron
On 1 June 2015 at 13:55, Ron Adam <ron3200@gmail.com> wrote:
Of course, it could just be my own preferences. I like the pattern of control (the specifying of what gets imported/shared) flowing from the top down.
This is actually how we bootstrap the import system in 3.3+ (we inject the sys and os modules *after* the top level execution of the bootstrap module is done, since the "import" statement doesn't work yet at the point where that module is running). However, this trick is never a desirable answer, just sometimes the least wrong choice out of multiple bad options :) Cheers, Nick. P.S. Python 3.5 is also more tolerant of circular imports than has historically been the case: https://bugs.python.org/issue17636 -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On za, 2015-05-30 at 11:45 -0400, Ron Adam wrote:
The solution I found was to call a function to explicitly set the shared items in the imported module.
This reminds me of my horrible april fools hack of 2013 to make Python look more like perl: http://seveas.net/exporter/ -- Dennis Kaarsemaker http://www.kaarsemaker.net
On 06/02/2015 04:23 AM, Dennis Kaarsemaker wrote:
On za, 2015-05-30 at 11:45 -0400, Ron Adam wrote:
The solution I found was to call a function to explicitly set the shared items in the imported module.
This reminds me of my horrible april fools hack of 2013 to make Python look more like perl: http://seveas.net/exporter/
It's the reverse of what this suggestion does, if I'm reading it correctly. It allows called code to alter the callers frame. Obviously that wouldn't be good to do. I think what makes the suggestion in this thread "not good", is that modules have no formal order of dependency. If they did, then it could be restricted to only work in one direction, which means sub-modules couldn't effect parent modules. But python isn't organised that way. All modules are at the same level. Which means they can import from each other... and possibly export to each other too. So it's up to the programmer to restrict what parts effect other parts as if it did have a formal dependency order. Cheers, Ron
participants (4)
-
Dennis Kaarsemaker -
Nick Coghlan -
Ron Adam -
Serhiy Storchaka