2011/7/24 Eric Snow :

On Sat, Jul 23, 2011 at 10:55 PM, Benjamin Peterson wrote:

...

2011/7/23 Eric Snow :

...

The documentation[1] doesn't say, but the implementation of the imp module makes me wonder if sys.modules was not meant to be replaceable.  No doubt this has to do with its tie to the interpreter's modules dict.  I ran into this doing "sys.modules = sys.modules.copy()" to protect the actual sys.modules dict during some import related test cases.  If the modules I imported were extension modules it broke.

So, is sys.modules not meant to be open to re-binding?

Not any more or less than other global mutable objects. You can expect other code to be holding on to old references.

But, isn't sys.modules a little different because other modules (at least the imp module) don't use it.  From what I understand the interpreter object's modules dict (to which sys.modules is initially bound) is used directly instead.  So rebinding sys.modules causes a disconnect.  That's why I am wondering if sys.modules is not meant to be rebound.

Sure. I'm not sure what point you're trying to make, though.

Are there other objects in the interpreter state that are exposed in sys that would have the same problem?

Is it problematic? -- Regards, Benjamin

On Sun, Jul 24, 2011 at 12:54 AM, Nick Coghlan wrote:

On Sun, Jul 24, 2011 at 3:05 PM, Eric Snow wrote:

...

Are there other objects in the interpreter state that are exposed in sys that would have the same problem?

Rebinding (rather than mutating) any global state in any module is always dubious due to the potential for direct references to the previous value. (This is a large part of the reason why imp.reload() often doesn't work in practice)

sys.modules, sys.path, sys.argv, sys.stdout, et al are all subject to that caveat. For mutable state, the onus is typically on the developer performing the substitution to decide whether or not those consequences are acceptable (usually, they're not, hence you get idioms like "sys.path[:] = saved_copy" to revert sys.path to a saved value). For immutable state (such as sys.stdout), where modification in place isn't possible, the obligation often goes the other way, with developers deliberately avoiding cached references in order to correctly react to runtime changes.

I agree with what you are saying wholeheartedly, but still think there is something fishy with the way that sys.modules works. I'll take it from here to a tracker issue though. :) -eric

sys.modules is by far the worst case though, since dropping modules out of that cache is only safe for modules that correctly support imp.reload(). This is not the case for any module that mutates other global state (or is otherwise referenced from other modules), nor does it work properly for extension modules.

Cheers, Nick.

-- Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia

I agree with what you are saying wholeheartedly, but still think there is something fishy with the way that sys.modules works. I'll take it from here to a tracker issue though. :)

Well, there is a short answer to you question: sys.modules is *not* meant to be replaced. Doing so will result in undefined behavior. So there is nothing fishy about it. Regards, Martin