Not sure why this is filling my sys memory

[Vincent Davis]

Here is a sample of the output, It almost instantly uses 2GB and then starts
using VMem. This is probably the right suggestion but it's another thing to
install

> It's probably also worth being aware of guppy's heapy stuff:

http://guppy-pe.sourceforge.net/heapy_tutorial....<http://guppy-pe.sourceforge.net/heapy_tutorial.html>I
find it quite nice to have the following to get a quick point-in-time
estimate of my app's memory usage:

print 'Current memory usage: %iMB' % (hpy().heap().stat.size/(1024*1024))


19 files of 43 using 3352 bytes of memory
Loading into memory: 3_veg_nm_rep1.txt
3_veg_nm_rep1.txt has 228484 rows of data
3_veg_nm_rep1.txt has 0 rows of masked data
3_veg_nm_rep1.txt has 141 rows of outliers
3_veg_nm_rep1.txt has 0 modified rows of data
280bytes of memory used for 3_veg_nm_rep1.txt

20 files of 43 using 3352 bytes of memory
Loading into memory: 3_veg_nm_rep2.txt
3_veg_nm_rep2.txt has 228484 rows of data
3_veg_nm_rep2.txt has 0 rows of masked data
3_veg_nm_rep2.txt has 119 rows of outliers
3_veg_nm_rep2.txt has 0 modified rows of data
280bytes of memory used for 3_veg_nm_rep2.txt

21 files of 43 using 3352 bytes of memory
Loading into memory: 3_veg_phd_rep1.txt
3_veg_phd_rep1.txt has 228484 rows of data
3_veg_phd_rep1.txt has 0 rows of masked data
3_veg_phd_rep1.txt has 63 rows of outliers
3_veg_phd_rep1.txt has 0 modified rows of data
280bytes of memory used for 3_veg_phd_rep1.txt

22 files of 43 using 6424 bytes of memory
Loading into memory: 3g_c285-11.txt
3g_c285-11.txt has 228484 rows of data
3g_c285-11.txt has 0 rows of masked data
3g_c285-11.txt has 65 rows of outliers
3g_c285-11.txt has 0 modified rows of data
280bytes of memory used for 3g_c285-11.txt

23 files of 43 using 6424 bytes of memory
Loading into memory: 3g_c285-42.txt
3g_c285-42.txt has 228484 rows of data
3g_c285-42.txt has 0 rows of masked data
3g_c285-42.txt has 27 rows of outliers
3g_c285-42.txt has 0 modified rows of data
280bytes of memory used for 3g_c285-42.txt

24 files of 43 using 6424 bytes of memory
Loading into memory: A6AF.txt
A6AF.txt has 228484 rows of data
A6AF.txt has 0 rows of masked data
A6AF.txt has 36 rows of outliers
A6AF.txt has 0 modified rows of data
280bytes of memory used for A6AF.txt

25 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3026_rep1.txt
Grigg_3026_rep1.txt has 228484 rows of data
Grigg_3026_rep1.txt has 0 rows of masked data
Grigg_3026_rep1.txt has 949 rows of outliers
Grigg_3026_rep1.txt has 0 modified rows of data
280bytes of memory used for Grigg_3026_rep1.txt

26 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3026_rep2.txt
Grigg_3026_rep2.txt has 228484 rows of data
Grigg_3026_rep2.txt has 0 rows of masked data
Grigg_3026_rep2.txt has 361 rows of outliers
Grigg_3026_rep2.txt has 0 modified rows of data
280bytes of memory used for Grigg_3026_rep2.txt

27 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3026_rep3_both.txt
Grigg_3026_rep3_both.txt has 228484 rows of data
Grigg_3026_rep3_both.txt has 0 rows of masked data
Grigg_3026_rep3_both.txt has 41 rows of outliers
Grigg_3026_rep3_both.txt has 0 modified rows of data
280bytes of memory used for Grigg_3026_rep3_both.txt

28 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3131_rep1.txt
Grigg_3131_rep1.txt has 228484 rows of data
Grigg_3131_rep1.txt has 0 rows of masked data
Grigg_3131_rep1.txt has 537 rows of outliers
Grigg_3131_rep1.txt has 0 modified rows of data
280bytes of memory used for Grigg_3131_rep1.txt

29 files of 43 using 6424 bytes of memory
Loading into memory: Grigg_3131_rep2.txt
Grigg_3131_rep2.txt has 228484 rows of data
Grigg_3131_rep2.txt has 0 rows of masked data
Grigg_3131_rep2.txt has 238 rows of outliers
Grigg_3131_rep2.txt has 0 modified rows of data
280bytes of memory used for Grigg_3131_rep2.txt

30 files of 43 using 6424 bytes of memory

  *Vincent Davis
720-301-3003 *
vincent at vincentdavis.net
 my blog <http://vincentdavis.net> |
LinkedIn<http://www.linkedin.com/in/vincentdavis>


On Sat, Feb 20, 2010 at 7:40 PM, ssteinerX at gmail.com <ssteinerx at gmail.com>wrote:

>
> On Feb 20, 2010, at 9:21 PM, Vincent Davis wrote:
>
> See this article for some more info about the reported sizes of things:
>> http://www.doughellmann.com/PyMOTW/sys/limits.html
>
>
> <http://www.doughellmann.com/PyMOTW/sys/limits.html>Nice article but I
> must have missed something useful to my current issue. Do I get any hints?
>
>
> Oh, sorry, there was the part about getsizeof() not including attributes
> unless the class supplies a __sizeof__ method and a comment at the bottom:
>
> It's probably also worth being aware of guppy's heapy stuff:
> http://guppy-pe.sourceforge.net/heapy_tutorial....<http://guppy-pe.sourceforge.net/heapy_tutorial.html>
>
> I find it quite nice to have the following to get a quick point-in-time
> estimate of my app's memory usage:
>
> print 'Current memory usage: %iMB' % (hpy().heap().stat.size/(1024*1024))
>
> Put a few of those in at various places in the collection process and you
> should see where you're taking a substantial hit on each pass.
>
> If I remember those numbers right, 5mb != 2.5gb so something along the way
> is doing something very strange.
>
> S
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-list/attachments/20100220/a0850b01/attachment-0001.html>