[Python-Dev] ctypes, memory mapped files and context manager

[Hans-Peter Jansen]

Hi,

first of all, sorry for being such a pest, but all former attempts to solve 
this issue on other fora has been grinding to a halt.

In short: I try to combine a context manager with ctypes structures on memory 
mapped files in order to manage huge binary files. (An approach, that performs 
great, while easy to use and keeping the resource usage low).

FWIW, the code is targeted for Linux environments running Python3.

The smallest script demonstrating the issue (thanks to Peter Otten):

import ctypes
import mmap

from contextlib import contextmanager

class T(ctypes.Structure):
    _fields = [("foo", ctypes.c_uint32)]


@contextmanager
def map_struct(m, n):
    m.resize(n * mmap.PAGESIZE)
    yield T.from_buffer(m)

SIZE = mmap.PAGESIZE * 2
f = open("tmp.dat", "w+b")
f.write(b"\0" * SIZE)
f.seek(0)
m = mmap.mmap(f.fileno(), mmap.PAGESIZE)

with map_struct(m, 1) as a:
    a.foo = 1
with map_struct(m, 2) as b:
    b.foo = 2


resulting in:
$ python3 mmap_test.py 
Traceback (most recent call last):
  File "mmap_test.py", line 23, in <module>
    with map_struct(m, 2) as b:
  File "/usr/lib64/python3.4/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "mmap_test.py", line 12, in map_struct
    m.resize(n * mmap.PAGESIZE)
BufferError: mmap can't resize with extant buffers exported.


Python2 does not crash, but that's a different story. What happens here is: 
the context manager variable "a" keeps a reference to a memory mapped area 
alive, that results in a unresizable and not properly closable mmap.

Right now, this rather ugly and error prone workaround must be used, that 
renders the purpose of the context manager ad absurdum:

with map_struct(m, 1) as a:
    a.foo = 1
del a
with map_struct(m, 2) as b:
    b.foo = 2
del b

In order to get this working properly, the ctypes mapping needs a method to 
free the mapping actively. E.g.:

@contextmanager
def map_struct(m, n):
    m.resize(n * mmap.PAGESIZE)
    yield T.from_buffer(m)
	 T.unmap_buffer(m)

Other attempts with weakref and the like do not work due to the nature of the 
ctypes types.

My own investigations in the _ctypes module were unsuccessful so far.

Hopefully, somebody in the audience cares enough for this module in order to 
get this fixed up (or probably I'm missing something obvious..).

Any ideas are very much appreciated.

Pete