Program uses twice as much memory in Python 3.6 than in Python 3.5
INADA Naoki
songofacandy at gmail.com
Thu Mar 30 13:34:33 EDT 2017
I reproduced the issue.
This is very usual, memory usage issue. Slashing is just a result of
large memory usage.
After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and
30GB on Python 3.6.
And Python 3.6 starts slashing in 2nd optimization pass.
I enabled tracemalloc while 1st pass. Results is end of this mail.
It seems frozenset() cause regression, but I'm not sure yet.
I don't know what is contents of frozenset yet. (I know almost
nothing about this application).
Jan, do you know about what this is?
Could you make script which just runs `transitive_closure(edges)` with
edges similar to
`log_reduction.py spaun`?
I'll dig into it later, maybe next week.
---
Python 3.6.1
1191896 memory blocks: 22086104.2 KiB
File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 85
reachables[vertex] = frozenset(reachables[vertex])
File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 410
self.dependents = transitive_closure(self.dg.forward)
602986 memory blocks: 51819.1 KiB
File "<string>", line 14
File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 634
first_view=None, v_offset=0, v_size=0, v_base=None)
Python 3.5.3
1166804 memory blocks: 11116407.0 KiB
File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 85
reachables[vertex] = frozenset(reachables[vertex])
File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 410
self.dependents = transitive_closure(self.dg.forward)
602989 memory blocks: 51819.3 KiB
File "<string>", line 14
File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 634
first_view=None, v_offset=0, v_size=0, v_base=None)
More information about the Python-list
mailing list