Thanks for all the replies, everyone! I'll reply to a few comments individually. But I first wanted to address the common theme around zipimport.
First, one idea that nobody mentioned (and came to me after reading the replies) was to possibly leverage zipimport for freezing the standard library instead of extending the frozen importer. I strongly feel this option is preferable to extending the frozen importer with additional functionality. I suspect the Python core developers would prefer to close importer feature gaps / bugs with zipimport over the frozen importer. And since zipimport is actually usable without having to build your own binaries, improvements to zipimport would more significantly benefit the larger Python ecosystem. If zipimporter gained the ability to open a zip archive residing in a memory address or a PyObject implementing the buffer protocol, [parts of] the standard library could be persisted as a zip file in libpython and the frozen importer would be limited to bootstrapping the import subsystem, just like it is today. This avoids adding additional complexity (like supporting __file__ and __cached__) to the frozen importer. And it keeps the standard library using a commonly-used importer in the standard library, just like it is today with PathFinder.
Onto the bigger question that can be summarized as "why not use zipimport: why do we need something different?" I sympathize with this reasoning. zipimport exists, it is functional, and it is widely used and has demonstrated value.
Performance is a major reason to build something better than zipimport.
I response to your replies, I implemented a handful of benchmarks for oxidized_importer and also implemented a pure Rust implementation of a zip file importer to collect some hard data. You can reproduce my results by cloning
https://github.com/indygreg/PyOxidizer.git and running `cargo bench -p pyembed-bench`. At time of writing, the benchmarks materialize the full standard library on the filesystem, in a zip archive (with no compression), and in the "Python packed resources" format. It then fires up a Python interpreter and imports ~450 modules comprising the standard library. I encourage you to obtain your own numbers and look at the benchmark code to better understand testing methodology. But here are some results from Linux on my Ryzen 5950x.
* zipimporter is slower than PathFinder to import the entirety of the standard library when the disk cache is hot. 201.81ms for zipimporter vs 174.06ms for PathFinder.
* My pure Rust zip importer is faster than zipimporter and PathFinder for the same operation. 161.67ms when reading zip data from memory; 164.45ms when using buffered filesystem I/O (8kb read operations).
* OxidizedFinder + Python packed resources are the fastest of all. 121.07ms loading from memory.
* Parsing/indexing the container formats is fast in Rust. Python packed resources parses in 107.69us and indexes in 200.52us (0.2ms). A zip archive table of contents is parsed in 809.61us and indexes in 1.205ms. If that same zip archive is read/seeked using filesystem I/O, the numbers go up to 4.6768ms and 5.1591ms.
* Starting and finalizing a Python interpreter takes 11.930ms with PathFinder and 4.8887ms with OxidizedFinder.
I won't post the full set of numbers for Windows, but they are generally higher, especially if filesystem I/O is involved. PathFinder is still faster than zipimporter, however. And zipimporter's relative slowness compared to OxidizedFinder is more pronounced.
There are many interesting takeaways from these numbers. But here are what I think are the most important:
* The Rust implementation of a zip importer trouncing performance of zipimporter probably means zipimporter could be made a lot faster (I didn't profile to measure why zipimporter is so slow. But I suspect its performance is hindered by being implemented in Python.)
* OxidizedFinder + Python packed resources are still significantly faster than the next fastest solution (Rust implemented zip importer).
* The overhead of reading and parsing the container format can matter. PyOxidizer built binaries can start and finalize a Python interpreter in <5ms (this ignores new process overhead). ~1.2ms for the Rust zip importer to index the zip file is a significant percentage!
Succinctly, today zipimporter is somewhat slow when you aren't I/O constrained. The existence proof of a faster Rust implementation implies it could be made significantly faster. Is that "good enough" to forego standard library inclusion of a yet more efficient solution? That's a healthy debate to have. You know which side I'm on :) But it would probably be prudent to optimize zipimporter before investing in something more esoteric.
Onto the individual replies.
My quick reaction was somewhat different - it would be a great idea, but it’s entirely possible to implement this outside the stdlib as a 3rd party module. So the fact that no-one has yet done so means there’s less general interest than the OP is suggesting.
Let me slightly push back on the "less general interest" assertion. While oxidized_importer is an existence proof that this is possible today, its upside today is limited because there is still a heavy dependence on a Python install being present and in a usable and well-defined state. This is difficult to achieve in practice and is why many distributed Python applications include their own Python distribution: it's the only way to be sure.
Even if you bundle your own unmodified Python distribution, the upside of something like oxidized_importer by itself is limited because you have to accommodate the modules in the standard library that are imported during interpreter initialization. Today, in order to import the entirety of the standard library from something other than .py files, you need to rely on zipimporter or a custom built binary that injects a meta path importer during interpreter startup. The latter is what PyOxidizer built executables do.
I think the current limitations preventing 3rd party meta path finders from being used exclusively constrain the upside of these tools. If we get to a point where a subset of the stdlib is "frozen" into the binary and PathFinder isn't used at all during startup before your __main__ code runs, then I think we'll finally be at a place where alternative 3rd party finders are viable and start seeing wider adoption. A potential feature request here would be a way to inject a sys.meta_path or sys.path_hooks entry during interpreter initialization, before any non-builtin extension modules are imported. If you could do this via environment variables, command line arguments, shebang tricks, or likewise, that opens up a lot of possibilities for enabling 3rd party meta path importers.
Something else to factor in here is that many people don't realize things like oxidized_importer are even possible! The importing mechanism is complex and implementing a conformant meta path importer is hard. But I do believe there is a latent market need here. I suspect if I spent the time to polish oxidized_importer a bit and actually spent effort to "market" it, it would probably see adoption in some of the larger Python projects out there where the performance/simplicity benefits would matter to end-users. But, that's all speculation: I understand there's a bar that needs to be cleared to justify complexity. I have more work to do here.
But would the downside of it not being possible to manage the format
with existing standard tools outweigh that?
This is a fair call out. I agree that the ubiquity of zip files is a major selling point. There would likely be a high hurdle to clear to justify introducing a non-standard format versus reusing something like zip files.
At the (relative) extreme is to throw out the existing frozen module
approach (or even the "unmarshal + exec" approach of source-based
modules) and replace it with something more efficient and/or more
compatible (and cross-platform). From what I understood, this is the
main focus of this thread.
Just to be clear, oxidized_importer + Python packed resources still retain the "unmarshal + exec" solution: it's the file container format that's different. (From my benchmarking and proof of existence in Facebook/Instagram land, we know that there are more efficient solutions for "unmarshal + exec" and I'm excited to see people poking around here!)
> a) backwards incompatible changes to the C API to support
additional metadata on frozen modules (or at the very least a
supplementary API that fragments what a "frozen" module is).
What part of the C-API, specifically?
The interpreter configuration and initialization APIs. If you extend the frozen struct to capture more metadata, that's an API break.
> You end up slowly reimplementing the importing mechanism in C (remember Python 2?) or disappoint users.
I'm not sure I follow. What part of the import system would be
reimplemented in C? The frozen importer is written in pure Python
with a few small helpers written in C. I expect that nearly all
necessary changes would happen in Lib/importlib/_bootstrap.py and not
Python/import.c.
I think I overspoke here, not realizing how much of the import machinery is in fact implemented in Python. (I even thought aspects of the zip importer were still implemented in C.)