On 21 October 2017 at 05:26, Doug Hellmann <doug@doughellmann.com> wrote:
I would also like to compare the performance of a few approaches
(1 file per sys.path hash using INI, JSON, and sqlite; one file per
entry on sys.path using the same formats) using a significant number
of plugins (~100?) before we decide.

If you can manage it, you'll want to run at least some of those tests with the plugins and their metadata mounted via a network drive. When the import system switched from multiple stat calls to cached os.listdir() lookups, SSD and spinning disk imports received a minor speedup, but NFS imports improved *dramatically* (folks reported order of magnitude improvements, along the lines of startup times dropping from 2-3 seconds to 200-300 ms).

I'd expect to see a similar pattern here - inefficient file access patterns can be tolerable with an SSD, and even spinning disks, but the higher latency involved in accessing network drives will make you pay for it.

Cheers,
Nick.

--
Nick Coghlan   |   ncoghlan@gmail.com   |   Brisbane, Australia