Ram memory not freed after executing python script on ubuntu system (rmlibre)
Inada Naoki
songofacandy at gmail.com
Fri May 29 01:28:59 EDT 2020
pymalloc manages only small blocks of memory.
Large (more than 512 byte) memory blocks are managed by malloc/free.
glibc malloc doesn't return much freed memory to OS.
You can try jemalloc instead of glibc.
On Ubuntu 20.04, you can try it by:
LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so python your-script.py
On Fri, May 29, 2020 at 11:07 AM <rmlibre at riseup.net> wrote:
>
>
> We just ran into this problem when running our aiootp package's memory
> hard password hashing function (https://github.com/rmlibre/aiootp/). The
> memory was not being cleared after the function finished running but the
> script was still live. We tried making sure everything went out of scope
> and deleting things explicitly inside the function, but that didn't
> help. We even tried forcing the garbage collector to free up
> unreferenced memory with import gc; gc.collect(). But that only had a
> small dent in the memory that was being built up.
>
> The most useful answer online had to do with Python's free lists being
> created automatically when very large datasets were being processed
> (https://stackoverflow.com/questions/23937189/how-do-i-use-subprocesses-to-force-python-to-release-memory/24126616#24126616).
>
>
> After putting the memory intensive work into a separate process, as the
> answer suggested, the memory was finally freed after execution.
>
> In our case, we wound up passing the result back in a
> ``multiprocessing.Manager().List()``. Though, passing your whole numpy
> dataset back doesn't seem feasible. I'd recommend doing the necessary
> memory intensive work in a separate process and passing only the
> necessary conclusions back to the main process.
>
> Or you could set up a multiprocessing queue to pass control messages to
> the spawned (daemon) process so it returns desired results on demand.
>
>
> Here's an excerpt from our code after the fix in case it's helpful.
>
> > @classmethod
> > def passcrypt(cls, password, salt, kb=1024, cpu=1024, hardness=256):
> > """
> > The ``passcrypt`` function can be highly memory intensive.
> > These resources may not be freed up, & often are not, because of
> > python quirks around memory management. This is a huge problem.
> > So to force the release of those resources, we run the function
> > in another process which is guaranteed to release them.
> > """
> > cls._validate_passcrypt_args(kb, cpu, hardness)
> > state = Manager().list()
> > process = Process(
> > target=cls._passcrypt,
> > args=(password, salt),
> > kwargs=dict(kb=kb, cpu=cpu, hardness=hardness, state=state),
> > )
> > process.start()
> > process.join()
> > return state.pop()
>
>
>
>
> On 2020-05-28 16:00, python-list-request at python.org wrote:
> > Send Python-list mailing list submissions to
> > python-list at python.org
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> > https://mail.python.org/mailman/listinfo/python-list
> > or, via email, send a message with subject or body 'help' to
> > python-list-request at python.org
> >
> > You can reach the person managing the list at
> > python-list-owner at python.org
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Python-list digest..."
> >
> > Today's Topics:
> >
> > 1. Re: Custom logging function (zljubisic at gmail.com)
> > 2. Ram memory not freed after executing python script on ubuntu
> > system (Rahul Gupta)
> > 3. Re: Ram memory not freed after executing python script on
> > ubuntu system (Chris Angelico)
> > 4. Re: Custom logging function (Peter Otten)
> > 5. Re: Elegant hack or gross hack? TextWrapper and escape codes
> > (Peter Otten)
> > 6. Re: Elegant hack or gross hack? TextWrapper and escape codes
> > (Chris Angelico)
> > 7. Re: Elegant hack or gross hack? TextWrapper and escape codes
> > (Peter Otten)
> > 8. Re: Ram memory not freed after executing python script on
> > ubuntu system (Rahul Gupta)
> > 9. Re: Ram memory not freed after executing python script on
> > ubuntu system (Chris Angelico)
> > 10. Re: Behaviour of os.path.join (BlindAnagram)
> > 11. Constructing mime image attachment (Joseph L. Casale)
> > 12. Re: Behaviour of os.path.join (Eryk Sun)
> > 13. Re: Behaviour of os.path.join (Eryk Sun)
> > 14. Re: Behaviour of os.path.join (BlindAnagram)
> > 15. Re: Behaviour of os.path.join (Eryk Sun)
> --
> https://mail.python.org/mailman/listinfo/python-list
--
Inada Naoki <songofacandy at gmail.com>
More information about the Python-list
mailing list