memory consumption

[Stestagg]

On Mon, Mar 29, 2021 at 2:32 PM Alexey <zen.supagood at gmail.com> wrote:

> понедельник, 29 марта 2021 г. в 15:57:43 UTC+3, Julio Oña:
> > It looks like the problem is on celery.
> > The mentioned issue is still open, so not sure if it was corrected.
> >
> > https://manhtai.github.io/posts/memory-leak-in-celery/
>
> As I mentioned in my first message, I tried to run
> this task(class) via Flask API calls, without Celery.
> And results are the same. Flask worker receives the API call and
> executes MyClass().run() inside of view. After a few calls
> worker size increases to 1Gb of RAM. In production I have 8 workers,
>  so in idle they will hold 8Gb.
>


Memory statistics in modern OSs are incredibly complex to reason about.
It's /possible/ that, while it certainly looks bad in the monitoring tools,
there actually isn't a problem here.

Some questions here to help understand more:

1. Do you have any actual problems caused by running 8 celery workers
(beyond high memory reports)? What are they?
2. Can you try a test with 16 or 32 active workers (i.e. number of
workers=2x available memory in GB), do they all still end up with 1gb
usage? or do you get any other memory-related issues running this?

Thanks

Steve



> --
> https://mail.python.org/mailman/listinfo/python-list
>