[Python-Dev] Investigating Python memory footprint of one real Web application

[Ivan Levkivskyi]

Inada-san,

I have made a PR for typing module upstream
https://github.com/python/typing/pull/383
It should reduce the memory consumption significantly (and also increase
isinstance() speed).
Could you please try it with your real code base and test memory
consumption (and maybe speed) as compared to master?

--
Ivan


On 23 January 2017 at 12:25, INADA Naoki <songofacandy at gmail.com> wrote:

> On Fri, Jan 20, 2017 at 8:52 PM, Ivan Levkivskyi <levkivskyi at gmail.com>
> wrote:
> > On 20 January 2017 at 11:49, INADA Naoki <songofacandy at gmail.com> wrote:
> >>
> >> * typing may increase memory footprint, through functions
> >> __attributes__ and abc.
> >>    * Can we add option to remove or lazy evaluate __attributes__ ?
> >
> >
> > This idea already appeared few times. I proposed to introduce a flag
> (e.g.
> > -OOO) to ignore function and variable annotations in compile.c
> > It was decide to postpone this, but maybe we can get back to this idea.
> >
> > In 3.6, typing is already (quite heavily) optimized for both speed and
> > space.
> > I remember doing an experiment comparing a memory footprint with and
> without
> > annotations, the difference was few percent.
> > Do you have such comparison (with and without annotations) for your app?
> > It would be nice to have a realistic number to estimate what would the
> > additional optimization flag save.
> >
> > --
> > Ivan
> >
> >
>
> Hi, Ivan.
>
> I investigated why our app has so many WeakSet today.
>
> We have dozen or hundreds of annotations like Iterable[User] or List[User].
> (User is one example of application's domain object.  There are
> hundreds of classes).
>
> On the other hand, SQLAlchemy calls isinstance(obj,
> collections.Iterable) many times,
> in [sqlalchemy.util._collections.to_list](https://github.com/
> zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/util/_
> collections.py#L795-L804)
> method.
>
> So there are (# of iterable subclasses) weaksets for negative cache,
> and each weakset
> contains (# of column types) entries.  That's why WeakSet ate much RAM.
>
> It may be slowdown application startup too, because thousands of
> __subclasscheck_ is called.
>
> I gave advice to use 'List[User]' instead of List[User] to the team of
> the project,
> if the team think RAM usage or boot speed is important.
>
> FWIW, stacktrace is like this:
>
>   File "/Users/inada-n/local/py37dbg/lib/python3.7/_weakrefset.py", line
> 84
>     self.data.add(ref(item, self._remove))
>   File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 233
>     cls._abc_negative_cache.add(subclass)
>   File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226
>     if issubclass(subclass, scls):
>   File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226
>     if issubclass(subclass, scls):
>   File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 191
>     return cls.__subclasscheck__(subclass)
>   File "venv/lib/python3.7/site-packages/sqlalchemy/util/_collections.py",
> line 803
>     or not isinstance(x, collections.Iterable):
>   File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line
> 1680
>     columns = util.to_list(prop)
>   File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line
> 1575
>     prop = self._property_from_column(key, prop)
>   File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line
> 1371
>     setparent=True)
>   File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line
> 675
>     self._configure_properties()<PasteEnd>
>
> Regards,
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170201/e485be04/attachment.html>