On 22 October 2017 at 04:03, xoviat <xoviat@gmail.com> wrote:

Nick:

That's generally a good idea, but one significant problem that can occur is that the Python import system will cache certain libraries, people will run "pip install," and then they will expect such libraries to be available. I don't even know exactly how the caching for the import system works, so I don't want to go and make claims about it that may be incorrect (maybe you do?). However, it is important to keep that in mind when considering an API.

Yep, since we switched to the implementation that uses fewer stat calls, you need to call `importlib.invalidate_caches()` to be sure your current process will see anything you just installed. However, for modules you haven't previously imported, that's independent of how you installed them - the problem is that the granularity on the import system's automated cache invalidation depends on the granularity of the filesystem's mtime records for directories, so you either have to sleep for a few seconds (since the mtime resolution on filesystems like FAT & FAT32 is only a couple of seconds), or else force the cache invalidation.

The problem with the sys.modules cache and already imported libraries is different: for those, you either need to use `importlib.reload(existing_module)` to force an in-place update of the existing namespace, or else `del sys.modules[existing_module.__name__]` to force creation of a new module without affecting old references to it.

It's those subtleties that keep "Restart all Python processes if you expect them to see components you just installed" in place as our default guidance: it's the only option that's guaranteed to work in all cases. While hot reloading can be made to work, it has assorted caveats and restrictions that you need to learn in order to do it correctly and reliably (many of them are actually pretty similar to the self-imposed restrictions needed to make lazy loading work reliably).

However, none of that impacts the question of whether `pip.main()` runs code in the current process or implicitly runs it in a subprocess - `pip` doesn't import the modules it installs either way, so it will all look the same as far as the import system is concerned.

Cheers,

Nick.