On Fri, Sep 3, 2021 at 5:32 AM Paul Moore <p.f.moore@gmail.com> wrote:
On Fri, 3 Sept 2021 at 10:29, Simon Cross <hodgestar+pythondev@gmail.com> wrote:
I think adding a meta path importer that reads from a standard optimized format could be a great addition.
I think the biggest open question would be "what benefits does this have over the existing zipimport?"
+1
As you mentioned in your email, this is a big detour from the current start-up performance work, so I think practically the people working on performance are unlikely to take a detour from their detour right now.
Agreed, it would probably have to be an independent development initially. If it delivers better performance, then switching the startup work to use it would give a second set of performance improvements, which no-one is going to object to. Similarly, if it's simpler to manage, then the maintainability benefits could justify switching over.
+1
* Write the meta path importer in a separate package (it sounds like you've already done a lot of the work and gained a lot of understanding of the issues while writing PyOxidizer!)
This is the key thing, though. The import machinery allows new importers to be written as standalone modules, so I'd strongly recommend that the proposed format/importer gets developed as a PyPI module initially, with the PEP then being simply a proposal that the module gets added to the stdlib and/or built into the interpreter.
FWIW, I'm a big fan of folks taking advantage of the flexibility of the import machinery and writing importers like this (especially ones that folks must explicitly enable). As noted elsewhere, it would need to prove its worth before we consider putting it into importlib.
The key argument would be bootstrapping, IMO. I would definitely expect interest in something like this to be lower if it's an external module (needing a dependency to load your other dependencies is suboptimal). Conversely, though, if no-one shows any interest in a PyPI version of this idea, that would strongly imply that it's not as useful in practice as you'd hoped.
Excellent point!
In particular, I'd involve the maintainers of pyinstaller in the design. If a new "frozen module importer" mechanism isn't of interest to them, it's probably not going to get the necessary support to be worth adding to the stdlib.
+1
On a personal note, I love the flexibility of Python's import system, and I've always wanted to write importers for additional storage formats (import from a sqlite database, for instance). But I've never actually done so, because a zipfile is basically always sufficient for any practical use case I've had. One day I hope to find a real use case, though :-)
Cool! I'd love to see what you make. -eric