Thinking about dead simple import hooks...
Hello, It would be nice to have a simple and easy way to override module importing, instead of that babylon which CPython has in that regard. (Even if initially "limited" to filesystem importing, most people won't ever find that limiting at all.) The desired semantics would be: For "import foo", before "/path1/foo.py", "/path2/foo.py" (where /path1:/path2 come from sys.path) is tried to be imported, a user-defined hook function can be called. It should take just one param - full basename of the module to be imported, e.g. "/path1/foo", "/path2/foo" for the cases above. The hook should either return a module object, the result of the import, or None to signal that standard processing should take place. The hook should be aware that there might have been previous hooks, and it's current hook's responsibility to call them. Example of a hook: def import_hook(path): if os.path.exists(path + ".my_ext"): mod = imp.new_module("") # populate mod here return mod A crucial semantic detail is that all possible module types from one sys.path entry should be tried first, before going to the next entry. (CPython import extensibility instead seems to be built around idea that it's fine to do multiple passes over sys.path, loading only a particular extension module type on each pass, which is not the desired semantics). Questions: 1. Does anybody know such thing to already exist? 2. Would you tweak anything in the API/boilerplate semantics of the above? -- Best regards, Paul mailto:pmiscml@gmail.com
And how would Python compute the "full basename of the file to be imported"? How could it guess among all directories on sys.path the one containing the "file", and check if it is a file or a package without going through the existing mechanism? Maybe this proposal is good - but possibly, just cutingt the "full basename" and pass to the "simple hook" whatever string is on the import statement would be better. On Tue, 8 Dec 2020 at 06:30, Paul Sokolovsky <pmiscml@gmail.com> wrote:
Hello, On Tue, 8 Dec 2020 07:29:15 -0300 "Joao S. O. Bueno" <jsbueno@python.org.br> wrote:
And how would Python compute the "full basename of the file to be imported"?
The way it does it usually.
It would not guess anything, it would iterate thru sys.path, as it usually does, and would give the hook a chance to see if the needed file exists. This proposal is also about *modules* and treats packages just as "hierarchical structure of modules" (Python does more than that, but this is "dead simple import hooks").
That's already possible, just hook into __import__() (or one of the other extension points provided by importlib). It's exactly the "convert full hierarchical module name into the actual file path" what's the boring part of it, and what I want to leave to the existing import system. [] -- Best regards, Paul mailto:pmiscml@gmail.com
On Tue, Dec 8, 2020 at 9:54 PM Paul Sokolovsky <pmiscml@gmail.com> wrote:
That involves a number of searches for exact file and/or directory names, so you can't just search a path arbitrarily. I would suggest that this is simultaneously too generic to be meaningful AND insufficiently generic to be useful. It'd be best to just write what you want directly, using the tools in importlib. ChrisA
Hello, On Tue, 8 Dec 2020 22:32:42 +1100 Chris Angelico <rosuav@gmail.com> wrote:
I usually like such statements myself, but that one made me wonder... Do you have an example (in the "single file == single module" domain) where it will be "insufficiently generic to be useful"?
It'd be best to just write what you want directly, using the tools in importlib.
Of course, the implementation of the above for CPython would be based on importlib. And other implementations could implement that "directly", for the background idea of it being how to implement import hooks as frugally as possible, and easily allow to do most of what could be wanted of single-file import hooking. In that regard, the question is again about API such import hooks offer. For example, how do you feel about "The hook should be aware that there might have been previous hook(s), and it's current hook's responsibility to call them." clause?
ChrisA
-- Best regards, Paul mailto:pmiscml@gmail.com
Hello, On Wed, 9 Dec 2020 02:01:58 +1100 Chris Angelico <rosuav@gmail.com> wrote:
I have written this (what my initial mail talks about) for both Pycopy and CPython. Now I wonder if I shouldn't really follow "the most minimal possible implementation" (the usual Pycopy criteria) and upgrade API a just little bit. Likely, the code byte count will still decide, but I'd like to get a second opinion from usability PoV, etc.
ChrisA
-- Best regards, Paul mailto:pmiscml@gmail.com
And how would Python compute the "full basename of the file to be imported"? How could it guess among all directories on sys.path the one containing the "file", and check if it is a file or a package without going through the existing mechanism? Maybe this proposal is good - but possibly, just cutingt the "full basename" and pass to the "simple hook" whatever string is on the import statement would be better. On Tue, 8 Dec 2020 at 06:30, Paul Sokolovsky <pmiscml@gmail.com> wrote:
Hello, On Tue, 8 Dec 2020 07:29:15 -0300 "Joao S. O. Bueno" <jsbueno@python.org.br> wrote:
And how would Python compute the "full basename of the file to be imported"?
The way it does it usually.
It would not guess anything, it would iterate thru sys.path, as it usually does, and would give the hook a chance to see if the needed file exists. This proposal is also about *modules* and treats packages just as "hierarchical structure of modules" (Python does more than that, but this is "dead simple import hooks").
That's already possible, just hook into __import__() (or one of the other extension points provided by importlib). It's exactly the "convert full hierarchical module name into the actual file path" what's the boring part of it, and what I want to leave to the existing import system. [] -- Best regards, Paul mailto:pmiscml@gmail.com
On Tue, Dec 8, 2020 at 9:54 PM Paul Sokolovsky <pmiscml@gmail.com> wrote:
That involves a number of searches for exact file and/or directory names, so you can't just search a path arbitrarily. I would suggest that this is simultaneously too generic to be meaningful AND insufficiently generic to be useful. It'd be best to just write what you want directly, using the tools in importlib. ChrisA
Hello, On Tue, 8 Dec 2020 22:32:42 +1100 Chris Angelico <rosuav@gmail.com> wrote:
I usually like such statements myself, but that one made me wonder... Do you have an example (in the "single file == single module" domain) where it will be "insufficiently generic to be useful"?
It'd be best to just write what you want directly, using the tools in importlib.
Of course, the implementation of the above for CPython would be based on importlib. And other implementations could implement that "directly", for the background idea of it being how to implement import hooks as frugally as possible, and easily allow to do most of what could be wanted of single-file import hooking. In that regard, the question is again about API such import hooks offer. For example, how do you feel about "The hook should be aware that there might have been previous hook(s), and it's current hook's responsibility to call them." clause?
ChrisA
-- Best regards, Paul mailto:pmiscml@gmail.com
Hello, On Wed, 9 Dec 2020 02:01:58 +1100 Chris Angelico <rosuav@gmail.com> wrote:
I have written this (what my initial mail talks about) for both Pycopy and CPython. Now I wonder if I shouldn't really follow "the most minimal possible implementation" (the usual Pycopy criteria) and upgrade API a just little bit. Likely, the code byte count will still decide, but I'd like to get a second opinion from usability PoV, etc.
ChrisA
-- Best regards, Paul mailto:pmiscml@gmail.com
participants (3)
-
Chris Angelico -
Joao S. O. Bueno -
Paul Sokolovsky