[Python-Dev] Investigating Python memory footprint of one real Web application

INADA Naoki songofacandy at gmail.com
Thu Feb 2 05:07:25 EST 2017


Thanks, Ivan.

I confirmed 3000 negative cache entries were removed by your patch!

https://gist.github.com/methane/3c34f11fb677365a7e92afe73aca24e7



On Thu, Feb 2, 2017 at 1:16 AM, Ivan Levkivskyi <levkivskyi at gmail.com> wrote:
> Inada-san,
>
> I have made a PR for typing module upstream
> https://github.com/python/typing/pull/383
> It should reduce the memory consumption significantly (and also increase
> isinstance() speed).
> Could you please try it with your real code base and test memory consumption
> (and maybe speed) as compared to master?
>
> --
> Ivan
>
>
> On 23 January 2017 at 12:25, INADA Naoki <songofacandy at gmail.com> wrote:
>>
>> On Fri, Jan 20, 2017 at 8:52 PM, Ivan Levkivskyi <levkivskyi at gmail.com>
>> wrote:
>> > On 20 January 2017 at 11:49, INADA Naoki <songofacandy at gmail.com> wrote:
>> >>
>> >> * typing may increase memory footprint, through functions
>> >> __attributes__ and abc.
>> >>    * Can we add option to remove or lazy evaluate __attributes__ ?
>> >
>> >
>> > This idea already appeared few times. I proposed to introduce a flag
>> > (e.g.
>> > -OOO) to ignore function and variable annotations in compile.c
>> > It was decide to postpone this, but maybe we can get back to this idea.
>> >
>> > In 3.6, typing is already (quite heavily) optimized for both speed and
>> > space.
>> > I remember doing an experiment comparing a memory footprint with and
>> > without
>> > annotations, the difference was few percent.
>> > Do you have such comparison (with and without annotations) for your app?
>> > It would be nice to have a realistic number to estimate what would the
>> > additional optimization flag save.
>> >
>> > --
>> > Ivan
>> >
>> >
>>
>> Hi, Ivan.
>>
>> I investigated why our app has so many WeakSet today.
>>
>> We have dozen or hundreds of annotations like Iterable[User] or
>> List[User].
>> (User is one example of application's domain object.  There are
>> hundreds of classes).
>>
>> On the other hand, SQLAlchemy calls isinstance(obj,
>> collections.Iterable) many times,
>> in
>> [sqlalchemy.util._collections.to_list](https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/util/_collections.py#L795-L804)
>> method.
>>
>> So there are (# of iterable subclasses) weaksets for negative cache,
>> and each weakset
>> contains (# of column types) entries.  That's why WeakSet ate much RAM.
>>
>> It may be slowdown application startup too, because thousands of
>> __subclasscheck_ is called.
>>
>> I gave advice to use 'List[User]' instead of List[User] to the team of
>> the project,
>> if the team think RAM usage or boot speed is important.
>>
>> FWIW, stacktrace is like this:
>>
>>   File "/Users/inada-n/local/py37dbg/lib/python3.7/_weakrefset.py", line
>> 84
>>     self.data.add(ref(item, self._remove))
>>   File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 233
>>     cls._abc_negative_cache.add(subclass)
>>   File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226
>>     if issubclass(subclass, scls):
>>   File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226
>>     if issubclass(subclass, scls):
>>   File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 191
>>     return cls.__subclasscheck__(subclass)
>>   File "venv/lib/python3.7/site-packages/sqlalchemy/util/_collections.py",
>> line 803
>>     or not isinstance(x, collections.Iterable):
>>   File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line
>> 1680
>>     columns = util.to_list(prop)
>>   File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line
>> 1575
>>     prop = self._property_from_column(key, prop)
>>   File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line
>> 1371
>>     setparent=True)
>>   File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line
>> 675
>>     self._configure_properties()<PasteEnd>
>>
>> Regards,
>
>


More information about the Python-Dev mailing list