On Wed, Jul 14, 2010 at 05:15, Nick Coghlan <ncoghlan@gmail.com> wrote:
On Wed, Jul 14, 2010 at 9:05 PM, Steve Holden <steve@holdenweb.com> wrote:
>> I have stopped fixing bugs related to this in import.c because of the
>> annoying issues it causes and I expect the correct approach to gain
>> traction at some point (plus get importlib bootstrapped in so I don't
>> have to care about import.c anymore).
>>
> It's only a matter of time until someone decides to provide a C
> implementation of importlib ;-)

A C accelerated version of importlib would probably be an awful lot
cleaner than the current import implementation. While the import code
isn't quite the mess that we sometimes make it out to be (it does
basically work after all, and most of the "problems" lie in dim dark
corners that 99% of developers will never get close to), but it has
definitely suffered from an accumulation of features on top of a core
approach that has been pushed far beyond what it was originally
designed to support.

So my dream is to finally get full compatibility for importlib in 3.3 (probably won't hit 3.2 as it requires me changing marshal.loads to take a file path argument) and then try to bootstrap it in. Now bootstrapping can be done with actually a minimal amount of C code as I can simply make the bytecode for importlib a literal in C, get that loaded, and then import importlib as found on the file system to allow people to override things. But obviously I could also identify the true bottlenecks through profiling and provide acceleration where useful. Trick is being reasonable about this so as to not put other VMs at a disadvantage by making the bootstrap solution too difficult to implement.
 

That said, I believe the limiting factor in import speed is likely to
remain the number of stat calls and other filesystem operations, so it
will be interesting to find out just how significant a slowdown there
is between import.c and importlib. If I'm right about the real source
of bottlenecks in import performance, the difference may be
surprisingly small.

So I started writing benchmark code in anticipation of needing to prove a minimal performance difference to justify bootstrapping importlib. Right now it only compares importing from sys.modules and built-in modules. You can run it with ``./python.exe -m importlib.test.benchmark``. If you add a `-b` option that will use the built-in __import__ implementation. I still need to benchmark loading source, bytecode, writing bytecode, and loading extensions. Maybe I will finish writing the benchmark code as the thing I do while at EuroPython (that and finally getting to reviewing http://bugs.python.org/issue2919 so that cProfile and profile can merge, unless someone beats me to it, in which case I would be grateful =).