[Python-Dev] Planning on removing cache invalidation for file finders

[Brett Cannon]

As of right now, importlib keeps a cache of what is in a directory for its
file finder instances. It uses mtime on the directory to try and detect
when it has changed to know when to refresh the cache. But thanks to mtime
granularities of up to a second, it is only a heuristic that isn't totally
reliable, especially across filesystems on different OSs.

This is why importlib.invalidate_caches() came into being. If you look in
our test suite you will see it peppered around where a module is created on
the fly to make sure that mtime granularity isn't a problem. But it
somewhat negates the point of the mtime heuristic when you have to make
this function call regardless to avoid potential race conditions.

http://bugs.python.org/issue17330 originally suggested trying to add
another heuristic to determine when to invalidate the cache. But even with
the suggestion it's still iffy and in no way foolproof.

So the current idea is to just drop the invalidation heuristic and go
full-blown reliance on calls to importlib.invalidate_caches() as necessary.
This makes code more filesystem-agnostic and protects people from
hard-to-detect errors when importlib only occasionally doesn't detect new
modules (I know it drove me nuts for a while when the buildbots kept
failing sporadically and only on certain OSs).

I would have just made the change but Antoine wanted it brought up here
first to make sure that no one was heavily relying on the current setup. So
if you have a good, legitimate reason to keep the reliance on mtime for
cache invalidation please speak up. But since the common case will never
care about any of this (how many people generate modules on the fly to
being with?) and to be totally portable you need to call
importlib.invalidate_caches() anyway, it's going to take a lot to convince
me to keep it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20130301/de64d39a/attachment.html>