[Numpy-discussion] Memory leak?

[Julian Taylor]

On 04.02.2014 18:26, Chris Laumann wrote:
> Hi all-
> 
> Thanks for the info re: memory leak. In trying to work around it, I
> think I’ve discovered another (still using SuperPack). This leaks ~30MB
> / run:
> 
> hists = zeros((50,64), dtype=int)
> for i in range(50):
>     for j in range(2**13):
>         hists[i,j%64] += 1
> 
> The code leaks using hists[i,j] = hists[i,j] + 1 as well. 
> 
> Is this the same leak or different? Doesn’t seem to have much in common.

Its the same leak, basically all scalar integers appearing somewhere leaked.
It is now fixed in git master. Sorry for the inconvenience I caused.

> 
> Incidentally, using 
> 
> a = ones(v.shape[0])
> a.dot(v)
> 
> Instead of np.sum (in the previous example that i sent) does not leak. 
> 
> Re: superpack
.. As a fairly technically proficient user, I’m aware that
> the super pack installs dev builds and that they may therefore by
> somewhat less reliable. I’m okay with that tradeoff and I don’t expect
> you guys to actually treat the super pack as a stable release — I also
> try to report that I’m using the superpack when I report bugs. I
> sometimes run git versions of ipython, numpy, etc in order to fiddle
> with the code and make tiny bug fixes/contributions myself. I don’t know
> the statistics re: superpack users but there is no link from scipy.org’s
> main install page so most new users won’t find it easily. Fonnesbeck’s
> webpage does say they are dev builds only two sentences into the paragraph.
> 
> Best, Chris
> 
> 
>
> -- 
> Chris Laumann
> Sent with Airmail
> 
> On January 31, 2014 at 9:31:40 AM, Julian Taylor
> (jtaylor.debian at googlemail.com <mailto://jtaylor.debian@googlemail.com>)
> wrote:
> 
>> On 31.01.2014 18:12, Nathaniel Smith wrote:
>> > On Fri, Jan 31, 2014 at 4:29 PM, Benjamin Root <ben.root at ou.edu> wrote:
>> >> Just to chime in here about the SciPy Superpack... this distribution tracks
>> >> the master branch of many projects, and then puts out releases, on the
>> >> assumption that master contains pristine code, I guess. I have gone down
>> >> strange rabbit holes thinking that a particular bug was fixed already and
>> >> the user telling me a version number that would confirm that, only to
>> >> discover that the superpack actually packaged matplotlib about a month prior
>> >> to releasing a version.
>> >>
>> >> I will not comment on how good or bad of an idea it is for the Superpack to
>> >> do that, but I just wanted to make other developers aware of this to keep
>> >> them from falling down the same rabbit hole.
>> >  
>> > Wow, that is good to know. Esp. since the web page:
>> >   http://fonnesbeck.github.io/ScipySuperpack/
>> > simply advertises that it gives you things like numpy 1.9 and scipy
>> > 0.14, which don't exist. (With some note about dev versions buried in
>> > prose a few sentences later.)
>> >  
>> > Empirically, development versions of numpy have always contained bugs,
>> > regressions, and compatibility breaks that were fixed in the released
>> > version; and we make absolutely no guarantees about compatibility
>> > between dev versions and any release versions. And it sort of has to
>> > be that way for us to be able to make progress. But if too many people
>> > start using dev versions for daily use, then we and downstream
>> > dependencies will have to start adding compatibility hacks and stuff
>> > to support those dev versions. Which would be a nightmare for
>> > developers and users both.
>> >  
>> > Recommending this build for daily use by non-developers strikes me as
>> > dangerous for both users and the wider ecosystem.
>> >  
>>
>> while probably not good for the user I think its very good for us.
>> This is the second bug I introduced found by superpack users.
>> This one might have gone unnoticed into the next release as it is pretty
>> much impossible to find via tests. Even in valgrind reports its hard to
>> find as its lumped in with all of pythons hundreds of memory arena
>> still-reachable leaks.
>>
>> Concerning the fix, it seems if python sees tp_free == PYObject_Del/Free
>> it replaces it with the tp_free of the base type which is int_free in
>> this case. int_free uses a special allocator for even lower overhead so
>> we start leaking.
>> We either need to find the right flag to set for our scalars so it stops
>> doing that, add an indirection so the function pointers don't match or
>> stop using the object allocator as we are apparently digging to deep
>> into pythons internal implementation details by doing so.
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion