<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif"><span style="font-family:arial,sans-serif">2017-03-13 18:11 GMT+01:00 Julian Taylor </span><span dir="ltr" style="font-family:arial,sans-serif"><<a href="mailto:jtaylor.debian@googlemail.com" target="_blank">jtaylor.debian@googlemail.com</a>></span><span style="font-family:arial,sans-serif">:</span><br></div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-">On 13.03.2017 16:21, Anne Archibald wrote:<br>
><br>
><br>
> On Mon, Mar 13, 2017 at 12:21 PM Julian Taylor<br>
</span>> <<a href="mailto:jtaylor.debian@googlemail.com">jtaylor.debian@googlemail.com</a> <mailto:<a href="mailto:jtaylor.debian@googlemail.com">jtaylor.debian@<wbr>googlemail.com</a>>><br>
<span class="gmail-">> wrote:<br>
><br>
> Should it be agreed that caching is worthwhile I would propose a very<br>
> simple implementation. We only really need to cache a small handful of<br>
> array data pointers for the fast allocate deallocate cycle that appear<br>
> in common numpy usage.<br>
> For example a small list of maybe 4 pointers storing the 4 largest<br>
> recent deallocations. New allocations just pick the first memory block<br>
> of sufficient size.<br>
> The cache would only be active on systems that support MADV_FREE (which<br>
> is linux 4.5 and probably BSD too).<br>
><br>
> So what do you think of this idea?<br>
><br>
><br>
> This is an interesting thought, and potentially a nontrivial speedup<br>
> with zero user effort. But coming up with an appropriate caching policy<br>
> is going to be tricky. The thing is, for each array, numpy grabs a block<br>
> "the right size", and that size can easily vary by orders of magnitude,<br>
> even within the temporaries of a single expression as a result of<br>
> broadcasting. So simply giving each new array the smallest cached block<br>
> that will fit could easily result in small arrays in giant allocated<br>
> blocks, wasting non-reclaimable memory. So really you want to recycle<br>
> blocks of the same size, or nearly, which argues for a fairly large<br>
> cache, with smart indexing of some kind.<br>
><br>
<br>
</span>The nice thing about MADV_FREE is that we don't need any clever cache.<br>
The same process that marked the pages free can reclaim them in another<br>
allocation, at least that is what my testing indicates it allows.<br>
So a small allocation getting a huge memory block does not waste memory<br>
as the top unused part will get reclaimed when needed, either by numpy<br>
itself doing another allocation or a different program on the system.<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Well, what you say makes a lot of sense to me, so if you have tested that then I'd say that this is worth a PR and see how it works on different workloads.</div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
An issue that does arise though is that this memory is not available for<br>
the page cache used for caching on disk data. A too large cache might<br>
then be detrimental for IO heavy workloads that rely on the page cache.<br></blockquote><div><br></div><div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif">Yeah. Also, memory mapped arrays use the page cache intensively, so we should test this use case and see how the caching affects memory map performance.</div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
So we might want to cap it to some max size, provide an explicit on/off<br>
switch and/or have numpy IO functions clear the cache.<br></blockquote><div><span style="font-family:arial,helvetica,sans-serif"><br></span></div><div><span style="font-family:arial,helvetica,sans-serif"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Definitely dynamically</div> allowing the disabling <div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">this feature would be desirable. That would provide an easy path for testing how it affects performance. Would that be feasible?</div></span><br></div><div><span style="font-family:arial,helvetica,sans-serif"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline"><br></div></span></div><div><span style="font-family:arial,helvetica,sans-serif"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;display:inline">Francesc</div></span></div><div><br></div></div>
</div></div>