memory consumption
Alexey
zen.supagood at gmail.com
Wed Mar 31 05:43:47 EDT 2021
среда, 31 марта 2021 г. в 06:54:52 UTC+3, Inada Naoki:
> First of all, I recommend upgrading your Python. Python 3.6 is a bit old.
I was thinking about that.
> As you saying, Python can not return the memory to OS until the whole
> arena become unused.
> If your task releases all objects allocated during the run, Python can
> release the memory.
> But if your task keeps at least one object, it may prevent releasing
> the whole arena (256KB).
>
> Python manages only small (~256bytes) objects. Larger objects is
> allocated by malloc().
> And glibc malloc may not efficient for some usage. jemalloc is better
> for many use cases.
>
> You can get some hints from sys._debugmallocstats(). It prints
> obmalloc (allocator for small objects) stats to stderr.
> Try printing stats before and after 1st run, and after 2nd run. And
> post it in this thread if you can. (no sensible information in the
> stats).
>
> That is all I can advise.
********** Before first run:
class size num pools blocks in use avail blocks
----- ---- --------- ------------- ------------
0 8 5 2404 126
1 16 3 611 148
2 24 13 2103 81
3 32 371 46703 43
4 40 292 29422 70
5 48 233 19499 73
6 56 1360 97903 17
7 64 1614 101654 28
8 72 1964 109947 37
9 80 1056 52783 17
10 88 436 20023 33
11 96 297 12455 19
12 104 266 10095 13
13 112 193 6930 18
14 120 127 4170 21
15 128 217 6712 15
16 136 1299 37669 2
17 144 1223 34239 5
18 152 113 2920 18
19 160 78 1949 1
20 168 1474 35369 7
21 176 54 1237 5
22 184 46 991 21
23 192 42 864 18
24 200 53 1054 6
25 208 39 713 28
26 216 54 955 17
27 224 575 10350 0
28 232 43 724 7
29 240 32 497 15
30 248 73 1153 15
31 256 29 431 4
32 264 25 375 0
33 272 46 637 7
34 280 24 328 8
35 288 20 280 0
36 296 398 5167 7
37 304 21 261 12
38 312 22 256 8
39 320 17 195 9
40 328 18 215 1
41 336 57 675 9
42 344 17 183 4
43 352 18 194 4
44 360 14 153 1
45 368 14 153 1
46 376 15 148 2
47 384 15 148 2
48 392 14 131 9
49 400 15 149 1
50 408 17 147 6
51 416 16 142 2
52 424 25 221 4
53 432 36 317 7
54 440 44 393 3
55 448 45 399 6
56 456 53 420 4
57 464 46 363 5
58 472 36 288 0
59 480 35 274 6
60 488 29 227 5
61 496 29 230 2
62 504 21 161 7
63 512 85 589 6
# arenas allocated total = 776
# arenas reclaimed = 542
# arenas highwater mark = 234
# arenas allocated current = 234
234 arenas * 262144 bytes/arena = 61,341,696
# bytes in allocated blocks = 59,737,176
# bytes in available blocks = 122,848
15 unused pools * 4096 bytes = 61,440
# bytes lost to pool headers = 718,128
# bytes lost to quantization = 702,104
# bytes lost to arena alignment = 0
Total = 61,341,696
3 free PyCFunctionObjects * 48 bytes each = 144
6 free PyDictObjects * 48 bytes each = 288
2 free PyFloatObjects * 24 bytes each = 48
0 free PyFrameObjects * 384 bytes each = 0
14 free PyListObjects * 40 bytes each = 560
7 free PyMethodObjects * 40 bytes each = 280
7 free 1-sized PyTupleObjects * 32 bytes each = 224
10 free 2-sized PyTupleObjects * 40 bytes each = 400
288 free 3-sized PyTupleObjects * 48 bytes each = 13,824
2 free 4-sized PyTupleObjects * 56 bytes each = 112
185 free 5-sized PyTupleObjects * 64 bytes each = 11,840
2 free 6-sized PyTupleObjects * 72 bytes each = 144
0 free 7-sized PyTupleObjects * 80 bytes each = 0
1 free 8-sized PyTupleObjects * 88 bytes each = 88
1 free 9-sized PyTupleObjects * 96 bytes each = 96
0 free 10-sized PyTupleObjects * 104 bytes each = 0
0 free 11-sized PyTupleObjects * 112 bytes each = 0
0 free 12-sized PyTupleObjects * 120 bytes each = 0
1 free 13-sized PyTupleObjects * 128 bytes each = 128
10 free 14-sized PyTupleObjects * 136 bytes each = 1,360
52 free 15-sized PyTupleObjects * 144 bytes each = 7,488
33 free 16-sized PyTupleObjects * 152 bytes each = 5,016
0 free 17-sized PyTupleObjects * 160 bytes each = 0
0 free 18-sized PyTupleObjects * 168 bytes each = 0
0 free 19-sized PyTupleObjects * 176 bytes each = 0
********** After first run:
class size num pools blocks in use avail blocks
----- ---- --------- ------------- ------------
0 8 5 2465 65
1 16 3 620 139
2 24 13 2137 47
3 32 409 47145 4389
4 40 295 29630 165
5 48 235 19638 102
6 56 1517 101292 7932
7 64 1670 103487 1723
8 72 2263 112937 13791
9 80 1076 53473 327
10 88 441 19958 328
11 96 360 14537 583
12 104 271 10243 55
13 112 197 6985 107
14 120 129 4204 53
15 128 221 6808 43
16 136 1309 37909 52
17 144 1231 34430 38
18 152 113 2936 2
19 160 80 1952 48
20 168 1507 35661 507
21 176 55 1210 55
22 184 46 995 17
23 192 42 869 13
24 200 53 1057 3
25 208 39 717 24
26 216 54 957 15
27 224 584 10423 89
28 232 44 732 16
29 240 32 501 11
30 248 73 1156 12
31 256 29 431 4
32 264 26 376 14
33 272 46 643 1
34 280 24 330 6
35 288 21 283 11
36 296 407 5246 45
37 304 21 263 10
38 312 22 257 7
39 320 17 197 7
40 328 19 218 10
41 336 57 682 2
42 344 17 184 3
43 352 18 194 4
44 360 14 153 1
45 368 14 154 0
46 376 15 150 0
47 384 15 149 1
48 392 14 134 6
49 400 15 150 0
50 408 17 147 6
51 416 17 145 8
52 424 29 254 7
53 432 41 365 4
54 440 50 443 7
55 448 52 461 7
56 456 61 488 0
57 464 54 413 19
58 472 41 328 0
59 480 39 309 3
60 488 33 258 6
61 496 32 255 1
62 504 25 193 7
63 512 97 660 19
# arenas allocated total = 47,669
# arenas reclaimed = 47,316
# arenas highwater mark = 10,114
# arenas allocated current = 353
353 arenas * 262144 bytes/arena = 92,536,832
# bytes in allocated blocks = 60,998,912
# bytes in available blocks = 2,086,632
6826 unused pools * 4096 bytes = 27,959,296
# bytes lost to pool headers = 756,768
# bytes lost to quantization = 735,224
# bytes lost to arena alignment = 0
Total = 92,536,832
6 free PyCFunctionObjects * 48 bytes each = 288
80 free PyDictObjects * 48 bytes each = 3,840
3 free PyFloatObjects * 24 bytes each = 72
2 free PyFrameObjects * 384 bytes each = 768
80 free PyListObjects * 40 bytes each = 3,200
256 free PyMethodObjects * 40 bytes each = 10,240
2000 free 1-sized PyTupleObjects * 32 bytes each = 64,000
481 free 2-sized PyTupleObjects * 40 bytes each = 19,240
2000 free 3-sized PyTupleObjects * 48 bytes each = 96,000
15 free 4-sized PyTupleObjects * 56 bytes each = 840
3 free 5-sized PyTupleObjects * 64 bytes each = 192
2000 free 6-sized PyTupleObjects * 72 bytes each = 144,000
2 free 7-sized PyTupleObjects * 80 bytes each = 160
2 free 8-sized PyTupleObjects * 88 bytes each = 176
1 free 9-sized PyTupleObjects * 96 bytes each = 96
0 free 10-sized PyTupleObjects * 104 bytes each = 0
0 free 11-sized PyTupleObjects * 112 bytes each = 0
0 free 12-sized PyTupleObjects * 120 bytes each = 0
0 free 13-sized PyTupleObjects * 128 bytes each = 0
0 free 14-sized PyTupleObjects * 136 bytes each = 0
0 free 15-sized PyTupleObjects * 144 bytes each = 0
0 free 16-sized PyTupleObjects * 152 bytes each = 0
0 free 17-sized PyTupleObjects * 160 bytes each = 0
0 free 18-sized PyTupleObjects * 168 bytes each = 0
0 free 19-sized PyTupleObjects * 176 bytes each = 0
********** After second run:
class size num pools blocks in use avail blocks
----- ---- --------- ------------- ------------
0 8 5 2465 65
1 16 3 620 139
2 24 13 2141 43
3 32 410 47144 4516
4 40 296 29664 232
5 48 235 19638 102
6 56 1554 101318 10570
7 64 1684 103495 2597
8 72 2361 112952 19264
9 80 1078 53472 428
10 88 443 19960 418
11 96 360 14537 583
12 104 272 10243 93
13 112 196 6985 71
14 120 129 4205 52
15 128 222 6810 72
16 136 1310 37909 81
17 144 1231 34430 38
18 152 113 2936 2
19 160 80 1952 48
20 168 1513 35658 654
21 176 55 1210 55
22 184 46 995 17
23 192 42 869 13
24 200 53 1057 3
25 208 39 717 24
26 216 54 957 15
27 224 584 10425 87
28 232 44 732 16
29 240 32 501 11
30 248 73 1156 12
31 256 29 431 4
32 264 26 376 14
33 272 46 643 1
34 280 24 330 6
35 288 21 283 11
36 296 407 5247 44
37 304 21 263 10
38 312 22 257 7
39 320 17 197 7
40 328 19 218 10
41 336 57 682 2
42 344 17 184 3
43 352 18 194 4
44 360 14 153 1
45 368 15 154 11
46 376 15 150 0
47 384 15 149 1
48 392 14 134 6
49 400 15 150 0
50 408 17 147 6
51 416 17 145 8
52 424 29 260 1
53 432 42 374 4
54 440 51 451 8
55 448 53 473 4
56 456 62 493 3
57 464 54 424 8
58 472 42 334 2
59 480 40 316 4
60 488 33 258 6
61 496 33 257 7
62 504 25 199 1
63 512 97 664 15
# arenas allocated total = 63,635
# arenas reclaimed = 63,238
# arenas highwater mark = 10,114
# arenas allocated current = 397
397 arenas * 262144 bytes/arena = 104,071,168
# bytes in allocated blocks = 61,039,064
# bytes in available blocks = 2,734,600
9471 unused pools * 4096 bytes = 38,793,216
# bytes lost to pool headers = 764,976
# bytes lost to quantization = 739,312
# bytes lost to arena alignment = 0
Total = 104,071,168
3 free PyCFunctionObjects * 48 bytes each = 144
80 free PyDictObjects * 48 bytes each = 3,840
4 free PyFloatObjects * 24 bytes each = 96
5 free PyFrameObjects * 384 bytes each = 1,920
80 free PyListObjects * 40 bytes each = 3,200
256 free PyMethodObjects * 40 bytes each = 10,240
2000 free 1-sized PyTupleObjects * 32 bytes each = 64,000
482 free 2-sized PyTupleObjects * 40 bytes each = 19,280
2000 free 3-sized PyTupleObjects * 48 bytes each = 96,000
15 free 4-sized PyTupleObjects * 56 bytes each = 840
3 free 5-sized PyTupleObjects * 64 bytes each = 192
2000 free 6-sized PyTupleObjects * 72 bytes each = 144,000
2 free 7-sized PyTupleObjects * 80 bytes each = 160
2 free 8-sized PyTupleObjects * 88 bytes each = 176
1 free 9-sized PyTupleObjects * 96 bytes each = 96
0 free 10-sized PyTupleObjects * 104 bytes each = 0
0 free 11-sized PyTupleObjects * 112 bytes each = 0
0 free 12-sized PyTupleObjects * 120 bytes each = 0
0 free 13-sized PyTupleObjects * 128 bytes each = 0
0 free 14-sized PyTupleObjects * 136 bytes each = 0
0 free 15-sized PyTupleObjects * 144 bytes each = 0
0 free 16-sized PyTupleObjects * 152 bytes each = 0
0 free 17-sized PyTupleObjects * 160 bytes each = 0
0 free 18-sized PyTupleObjects * 168 bytes each = 0
0 free 19-sized PyTupleObjects * 176 bytes each = 0
More information about the Python-list
mailing list