[Numpy-discussion] Memory leak?

[Chris Laumann]

Hi all-

Thanks for the info re: memory leak. In trying to work around it, I think I’ve discovered another (still using SuperPack). This leaks ~30MB / run:

hists = zeros((50,64), dtype=int)
for i in range(50):
    for j in range(2**13):
        hists[i,j%64] += 1

The code leaks using hists[i,j] = hists[i,j] + 1 as well. 

Is this the same leak or different? Doesn’t seem to have much in common.

Incidentally, using 

a = ones(v.shape[0])
a.dot(v)

Instead of np.sum (in the previous example that i sent) does not leak. 

Re: superpack
.. As a fairly technically proficient user, I’m aware that the super pack installs dev builds and that they may therefore by somewhat less reliable. I’m okay with that tradeoff and I don’t expect you guys to actually treat the super pack as a stable release — I also try to report that I’m using the superpack when I report bugs. I sometimes run git versions of ipython, numpy, etc in order to fiddle with the code and make tiny bug fixes/contributions myself. I don’t know the statistics re: superpack users but there is no link from scipy.org’s main install page so most new users won’t find it easily. Fonnesbeck’s webpage does say they are dev builds only two sentences into the paragraph.

Best, Chris








-- 
Chris Laumann
Sent with Airmail

On January 31, 2014 at 9:31:40 AM, Julian Taylor (jtaylor.debian at googlemail.com) wrote:

On 31.01.2014 18:12, Nathaniel Smith wrote:  
> On Fri, Jan 31, 2014 at 4:29 PM, Benjamin Root <ben.root at ou.edu> wrote:  
>> Just to chime in here about the SciPy Superpack... this distribution tracks  
>> the master branch of many projects, and then puts out releases, on the  
>> assumption that master contains pristine code, I guess. I have gone down  
>> strange rabbit holes thinking that a particular bug was fixed already and  
>> the user telling me a version number that would confirm that, only to  
>> discover that the superpack actually packaged matplotlib about a month prior  
>> to releasing a version.  
>>  
>> I will not comment on how good or bad of an idea it is for the Superpack to  
>> do that, but I just wanted to make other developers aware of this to keep  
>> them from falling down the same rabbit hole.  
>  
> Wow, that is good to know. Esp. since the web page:  
> http://fonnesbeck.github.io/ScipySuperpack/  
> simply advertises that it gives you things like numpy 1.9 and scipy  
> 0.14, which don't exist. (With some note about dev versions buried in  
> prose a few sentences later.)  
>  
> Empirically, development versions of numpy have always contained bugs,  
> regressions, and compatibility breaks that were fixed in the released  
> version; and we make absolutely no guarantees about compatibility  
> between dev versions and any release versions. And it sort of has to  
> be that way for us to be able to make progress. But if too many people  
> start using dev versions for daily use, then we and downstream  
> dependencies will have to start adding compatibility hacks and stuff  
> to support those dev versions. Which would be a nightmare for  
> developers and users both.  
>  
> Recommending this build for daily use by non-developers strikes me as  
> dangerous for both users and the wider ecosystem.  
>  

while probably not good for the user I think its very good for us.  
This is the second bug I introduced found by superpack users.  
This one might have gone unnoticed into the next release as it is pretty  
much impossible to find via tests. Even in valgrind reports its hard to  
find as its lumped in with all of pythons hundreds of memory arena  
still-reachable leaks.  

Concerning the fix, it seems if python sees tp_free == PYObject_Del/Free  
it replaces it with the tp_free of the base type which is int_free in  
this case. int_free uses a special allocator for even lower overhead so  
we start leaking.  
We either need to find the right flag to set for our scalars so it stops  
doing that, add an indirection so the function pointers don't match or  
stop using the object allocator as we are apparently digging to deep  
into pythons internal implementation details by doing so.  
_______________________________________________  
NumPy-Discussion mailing list  
NumPy-Discussion at scipy.org  
http://mail.scipy.org/mailman/listinfo/numpy-discussion  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20140204/fc5e5da4/attachment.html>