[Python-Dev] Solving the import-deadlock case

[Pascal Chambon]

Hello everyone,

I'd like to bring your attention to this issue, since it touches the 
fundamentals of python's import workflow:
http://bugs.python.org/issue17716

/I've tried to post it on the python-import ML for weeks, but it must 
still be blocked somewhere in a moderation queue, so here I come ^^/

TLDR version: because of the way import current works, if importing a 
package "temporarily" fails whereas importing one of its children 
succeeded, we reach an unusable state, all subsequent attempts at 
importing that package will fail if a "from...import" is used somewhere. 
Typically, it makes a web worker broken, even though the typical 
behaviour of such process woudl be to retry loading, again and again, 
the failing view.

I agree that a module loading should be, as much as possible, "side 
effects free", and thus shouldn't have temporary errors. But well, in 
practice, module loading is typically the time where process-wide 
initialization are done (modifying sys.path, os.environ, instantiating 
connection or thread pools, registering atexit handler, starting 
maintenance threads...), so that case has chances to happen at a moment 
or another, especially if accesses to filesystem or network (SQL...) are 
done at module loading, due to the lack of initialization system at 
upper levels.

That's why I propose modifying the behaviour of module import, so that 
submodules are deleted as well when a parent module import fails. True, 
it means they will be reloaded as well when importing the parent will 
start again, but anyway we already have a "double execution" problem 
with the reloading of the parent module, so it shouldn't make a big 
difference.
The only other solution I'd see would be to SYSTEMATICALLY perform name 
(re)binding when processing a from...import statement, to recover from 
the previously failed initialization. Dunno if it's a good idea.

On a (separate but related) topic, to be safer on module reimports or 
reloadings, it could be interesting to add some "idempotency" to common 
initialization tasks ; for example the "atexit" registration system, 
wouldn't it be worth adding a boolean flag to explicitely skip 
registration if a callable with same fully qualified name is already 
registered.

Do you have opinions on these subjects ?

thanks,
regards,
Pascal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130702/1ecdb18e/attachment.html>