[IPython-dev] ipython memory usage problem
cadourian
chahe.adourian at asc-csa.gc.ca
Fri Oct 23 14:21:39 EDT 2015
I could not provide an example that causes the problem cause the setup to run
it would be too complicated however I think we should be able to make
progress.
I'll explain. I ran two examples. Both examples have the same pseudocode
template
-------------
Procedure A:
-------------
a) Clear the memory on the engines using the following commands
securityFile =
r'%s\EngineConfig\profile_default\security\ipcontroller-client.json' %
projectRoot
rc = Client(securityFile)
idSubset = rc.ids
dview = rc[idSubset]
dview.execute('%reset')
dview.execute('import gc')
dview.execute('gc.collect()')
b) Measure memory usage on each cluster engine
c) call a function x using dview.map_sync(x, ....)
d) Measure memory usage on each cluster engine
#1) memory usage on engines does not increase when Procedure A is called
multiple times. In this case, the function x used in dview.map_sync(x, ....)
allocates an array with numpy and returns it
#2) In this case, memory increases when Procedure A is called multiple
times. The difference is that function x used in dview.map_sync(x, ....)
loads DLLs and executes loaded functions.
What I don't understand is that I'm calling a function (x) which is
obviously causing the problems. But, I would have expected that when the
function call returned, any objects created within would be destroyed.
Moreover, I purposefully called garbage collect to eliminate any remaining
objects before measuring memory usage.
The question is whether IPython handles functions a little differently than
Python? Any suggestions appreciated.
--
View this message in context: http://python.6.x6.nabble.com/ipython-memory-usage-problem-tp5174336p5174675.html
Sent from the IPython - Development mailing list archive at Nabble.com.
More information about the IPython-dev
mailing list