[New-bugs-announce] [issue28509] Key-sharing dictionaries can inrease the memory consumption

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

The size of small key-sharing dictionary (PEP 412) can be larger than the size of normal dictionary.

Python 3.6:

>>> def dictsizes(k):
...     d = {'a%d'%i: None for i in range(k)}
...     class C:
...         def __init__(self):
...             self.__dict__.update(d)
...     return sys.getsizeof(d), sys.getsizeof(C().__dict__)
... 
>>> for i in range(20):
...     print(i, dictsizes(i))
... 
0 (128, 60)
1 (128, 60)
2 (128, 60)
3 (128, 60)
4 (128, 60)
5 (128, 60)
6 (196, 196)
7 (196, 196)
8 (196, 344)
9 (196, 344)
10 (196, 344)
11 (344, 344)
12 (344, 344)
13 (344, 344)
14 (344, 344)
15 (344, 344)
16 (344, 628)
17 (344, 628)
18 (344, 628)
19 (344, 628)

Normal dictionaries of size 8-10 are more compact than corresponding key-sharing dictionaries.

Python 3.5:

>>> for i in range(20):
...     print(i, dictsizes(i))
... 
0 (144, 48)
1 (144, 48)
2 (144, 48)
3 (144, 48)
4 (144, 240)
5 (144, 240)
6 (240, 240)
7 (240, 240)
8 (240, 432)
9 (240, 432)
10 (240, 432)
11 (240, 432)
12 (432, 432)
13 (432, 432)
14 (432, 432)
15 (432, 432)
16 (432, 816)
17 (432, 816)
18 (432, 816)
19 (432, 816)

In Python 3.5 normal dictionaries of size 4-5 and 8-11 are more compact than corresponding key-sharing dictionaries. And note that key-sharing dictionaries of size 0-3 consume more memory on 3.6 that on 3.5. I think we should use more thrifty strategy for allocating key-sharing dictionaries.

----------
components: Interpreter Core
messages: 279215
nosy: Mark.Shannon, benjamin.peterson, inada.naoki, rhettinger, serhiy.storchaka
priority: normal
severity: normal
status: open
title: Key-sharing dictionaries can inrease the memory consumption
type: resource usage

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue28509>
_______________________________________