Python memory use (psyco, C++)

[Delaney, Timothy C (Timothy)]

Roy Smith wrote:

> I understand that psyco significantly increases memory use.  Is that
> for code or data?  More specifically, if I've got a memory intensive
> application (it might use 100's of Mbytes of data), should I expect
> memory use to go up significantly under psyco?

The memory use is primarily related to the code size, assuming that most
functions are always called with the same data types.

If you have functions that are being called with lots of different data
types, memory use will increase accordingly, as psyco creates
specialised versions of functions for the datatypes they are called
with.

> Also, for that memory intensive application, how should I expect
> Python memory use to compare with C++?  I'm really only interested in
> data; the memory needed to store the code is almost certainly
> insignificant in either case.

Python will almost certainly use more memory than C++ if you don't use
brain-dead structures. However, depending on your data it is possible to
use much less memory with Python because it's a lot simpler to share
identical immutable data in Python.

> The data is a large number of small objects, interconnected in various
> data structures, not just one huge block of raw data.

This suggests though that that approach won't be as useful. OTOH, it may
be that the final chunks of data can be shared, even if the structures
holding them can't.

> I know all of the above is very vague, but I'm just trying to get a
> rough idea if a Python implementation is feasable (or at least
> plausable).  If a C++ version takes 300 Mbytes and a Python version
> takes 1 Gig, that's probably not going to work.  Are there any rules
> of thumb I could use to get a first-order estimate?

Unfortunately not. It depends heavily on the type of data. However, it
should (hopefully) be a simple matter to code something up that just
loads the data from disk to try it out.

Tim Delaney