Ram memory not freed after executing python script on ubuntu system (rmlibre)

[Inada Naoki]

pymalloc manages only small blocks of memory.
Large (more than 512 byte) memory blocks are managed by malloc/free.

glibc malloc doesn't return much freed memory to OS.
You can try jemalloc instead of glibc.

On Ubuntu 20.04, you can try it by:

  LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so python your-script.py

On Fri, May 29, 2020 at 11:07 AM <rmlibre at riseup.net> wrote:
>
>
> We just ran into this problem when running our aiootp package's memory
> hard password hashing function (https://github.com/rmlibre/aiootp/). The
> memory was not being cleared after the function finished running but the
> script was still live. We tried making sure everything went out of scope
> and deleting things explicitly inside the function, but that didn't
> help. We even tried forcing the garbage collector to free up
> unreferenced memory with import gc; gc.collect(). But that only had a
> small dent in the memory that was being built up.
>
> The most useful answer online had to do with Python's free lists being
> created automatically when very large datasets were being processed
> (https://stackoverflow.com/questions/23937189/how-do-i-use-subprocesses-to-force-python-to-release-memory/24126616#24126616).
>
>
> After putting the memory intensive work into a separate process, as the
> answer suggested, the memory was finally freed after execution.
>
> In our case, we wound up passing the result back in a
> ``multiprocessing.Manager().List()``. Though, passing your whole numpy
> dataset back doesn't seem feasible. I'd recommend doing the necessary
> memory intensive work in a separate process and passing only the
> necessary conclusions back to the main process.
>
> Or you could set up a multiprocessing queue to pass control messages to
> the spawned (daemon) process so it returns desired results on demand.
>
>
> Here's an excerpt from our code after the fix in case it's helpful.
>
> >    @classmethod
> >    def passcrypt(cls, password, salt, kb=1024, cpu=1024, hardness=256):
> >        """
> >        The ``passcrypt`` function can be highly memory intensive.
> >        These resources may not be freed up, & often are not, because of
> >        python quirks around memory management. This is a huge problem.
> >        So to force the release of those resources, we run the function
> >        in another process which is guaranteed to release them.
> >        """
> >        cls._validate_passcrypt_args(kb, cpu, hardness)
> >        state = Manager().list()
> >        process = Process(
> >            target=cls._passcrypt,
> >            args=(password, salt),
> >            kwargs=dict(kb=kb, cpu=cpu, hardness=hardness, state=state),
> >        )
> >        process.start()
> >        process.join()
> >        return state.pop()
>
>
>
>
> On 2020-05-28 16:00, python-list-request at python.org wrote:
> > Send Python-list mailing list submissions to
> >       python-list at python.org
> >
> > To subscribe or unsubscribe via the World Wide Web, visit
> >       https://mail.python.org/mailman/listinfo/python-list
> > or, via email, send a message with subject or body 'help' to
> >       python-list-request at python.org
> >
> > You can reach the person managing the list at
> >       python-list-owner at python.org
> >
> > When replying, please edit your Subject line so it is more specific
> > than "Re: Contents of Python-list digest..."
> >
> > Today's Topics:
> >
> >    1. Re: Custom logging function (zljubisic at gmail.com)
> >    2. Ram memory not freed after executing python script on ubuntu
> >       system (Rahul Gupta)
> >    3. Re: Ram memory not freed after executing python script on
> >       ubuntu system (Chris Angelico)
> >    4. Re: Custom logging function (Peter Otten)
> >    5. Re: Elegant hack or gross hack? TextWrapper and escape codes
> >       (Peter Otten)
> >    6. Re: Elegant hack or gross hack? TextWrapper and escape codes
> >       (Chris Angelico)
> >    7. Re: Elegant hack or gross hack? TextWrapper and escape codes
> >       (Peter Otten)
> >    8. Re: Ram memory not freed after executing python script on
> >       ubuntu system (Rahul Gupta)
> >    9. Re: Ram memory not freed after executing python script on
> >       ubuntu system (Chris Angelico)
> >   10. Re: Behaviour of os.path.join (BlindAnagram)
> >   11. Constructing mime image attachment (Joseph L. Casale)
> >   12. Re: Behaviour of os.path.join (Eryk Sun)
> >   13. Re: Behaviour of os.path.join (Eryk Sun)
> >   14. Re: Behaviour of os.path.join (BlindAnagram)
> >   15. Re: Behaviour of os.path.join (Eryk Sun)
> --
> https://mail.python.org/mailman/listinfo/python-list



-- 
Inada Naoki  <songofacandy at gmail.com>