[Distutils] Pypy and site.py and makepath

[P.J. Eby]

At 05:42 PM 3/13/2010 +0100, Lennart Regebro wrote:
>On Sat, Mar 13, 2010 at 16:07, P.J. Eby <pje at telecommunity.com> wrote:
> > 2. It is finding a valid site.py, but the standard import protocols (i.e.,
> > imp.find_module/load_module and
> > path_importer_cache[modulename].load_module()) are not working correctly
> > when called on an existing module.
>
>Yeah, this seems like the problem. It imports it, but neither globals
>or locals change. It's hard to say exactly what the problem is as I
>can't debug what happens in CPython, as no matter what I put into the
>local site.py, nothing seems to happen. So I't hard to me to check
>what the behaviour *should* be, but it seems clear that this is where
>the difference is.

What the behavior should be is that when you call load_module (either 
on an importer instance or the imp.load_module function), and the 
module already exists in sys.modules, that's the module object that 
should be updated.  reload() relies on this, as do various other 
things such as lazy importing tools.

A quick glance at PyPy's source (at 
http://codespeak.net/pypy/dist/pypy/module/imp/ ) reveals that it 
does support reusing a module in sys.modules, but for some reason 
this code is shut off by a keyword argument that defaults to 
false.  This behavior is *definitely* not what CPython's load_module does.

I didn't dig into whether this is also true for any importers that 
PyPy might put into the path importer cache, but it's definitely true 
for imp.load_module.

One could argue, I suppose, whether the imp behavior is official 
Python spec vs. CPython-only feature, but setuptools isn't the only 
thing that relies on it, and PEP 302 *does* makes the reuse behavior 
explicitly required for loader objects.  Also, its absence would make 
it bloody hard (probably impossible, actually) to implement a 
reload() function (either the base one or an improved version) in 
terms of the imp API.

Simplest fix for this in PyPy would probably be to change 
load_module() in modules.imp.interp_inp to call 
importing.load_module() with the 'reuse' argument set to True.  A 
test for this behavior is straightforward: stick a module object into 
sys.modules with a variable set, then use load_module() to load 
something that sets another variable, then check that both variables 
are set in the module.

(Also, it should assert that load_module() returns the module object 
that was *already* in sys.modules, rather than that it just shoved a 
new module into sys.modules -- which is what the current PyPy code 
apparently does.)