Program uses twice as much memory in Python 3.6 than in Python 3.5

INADA Naoki songofacandy at gmail.com
Thu Mar 30 13:34:33 EDT 2017


I reproduced the issue.
This is very usual, memory usage issue.  Slashing is just a result of
large memory usage.

After 1st pass of optimization, RAM usage is 20GB+ on Python 3.5 and
30GB on Python 3.6.
And Python 3.6 starts slashing in 2nd optimization pass.

I enabled tracemalloc while 1st pass.  Results is end of this mail.
It seems frozenset() cause regression, but I'm not sure yet.
I don't know what is contents of frozenset yet.  (I know almost
nothing about this application).

Jan, do you know about what this is?
Could you make script which just runs `transitive_closure(edges)` with
edges similar to
`log_reduction.py spaun`?

I'll dig into it later, maybe next week.

---
Python 3.6.1
1191896 memory blocks: 22086104.2 KiB
  File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 85
    reachables[vertex] = frozenset(reachables[vertex])
  File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 410
    self.dependents = transitive_closure(self.dg.forward)
602986 memory blocks: 51819.1 KiB
  File "<string>", line 14
  File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 634
    first_view=None, v_offset=0, v_size=0, v_base=None)

Python 3.5.3
1166804 memory blocks: 11116407.0 KiB
  File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 85
    reachables[vertex] = frozenset(reachables[vertex])
  File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 410
    self.dependents = transitive_closure(self.dg.forward)
602989 memory blocks: 51819.3 KiB
  File "<string>", line 14
  File "/home/inada-n/work/gosmann-frontiers2017/gosmann_frontiers2017/optimized/optimizer.py",
line 634
    first_view=None, v_offset=0, v_size=0, v_base=None)


More information about the Python-list mailing list