[Python-Dev] Complexity of import.c

[Just van Rossum]

James C. Ahlstrom wrote:

> I see that my patch to import.c has been criticized
> as adding to the complexity of an already complex module.
> I don't entirely agree.  If you want to have the standard
> library available as a zip module, you must have zip file
> import available in C.

This has not been questioned.

> The patch I proposed in PEP 273
> was a minimalist patch; it adds as little as possible
> while still keeping package import and preserving all
> import semantics.

I think my counterpatch (even if it's not done yet) shows this is simply not
true.

> IMHO, the real complexity of import.c comes not from
> the double loop
>    for path in sys.path:
>       for suffix in (".pyc", ".pyo",...):
>           fopen(...)
> and not from the caching of zip archive names.  It comes
> from the use of import.c to perform package imports.
> Package imports are performed by creating recursive calls
> to functions that would otherwise be flat, and by replacing
> sys.path with the package path. 

Erm, I think package import is very recursive by *nature*. I'm not sure what
function you mean that could be flat.

> This is darned confusing.

So true!

> At the time PEP 273 was discussed, I proposed moving package
> imports out of import.c into a Python module such as Gordon's
> iu.py or Greg's imputil.py.  This was rejected because package
> users feared that package imports would be slowed down.  The
> speed of Python startup and import was a concern.

It's also dubious how this would help: the current code works well and is
stable. I would fear that moving any of this to Python would cause bootstrapping
problems.

> I understand that people want to generalize zip imports, and
> create a better import hook mechanism.  Some feel that
> now is the time to do it.
> 
> This is my humble opinion:
> 
> Replacing import.c plus zip import with an equally complex
> import.c is a fundamental mistake.

Yes, and luckily noone is proposing to do that.

[ ... ]
> My patch modified many other Python .c and .py files to
> solve several difficult bootstrap problems.  If you replace
> just import.c you will get a painful lesson in bootstrapping.

I've learned them indeed over the last couple of days, but I didmanage to get it
working by only touching import.c and importdl.h. But my current working version
also touches pythonrun.[hc], as it has a new _PyImportHooks_Init function. (The
reason this was neccesary was I wanted a subclass of ImportError in the
zipimport module; I learned the hard way that you can't create new exceptions
from C before the exceptions module is loaded... So I had to move my init code
from _PyImport_Init to a separate function, which is called by Py_Initialize().)

> You probably need these other patches even if you reject my
> import.c patch.

I doubt it.

Just