[python-win32] Opening existing memory mapped files with pywin32

[Eryk Sun]

On 2/16/21, rhyslloyd1 via python-win32 <python-win32 at python.org> wrote:
>
> I am trying to open a memory mapped file using Python. I originally used the
> "mmap" module from Python however I had issues with it because I had to use
> a fixed size for the file even though my goal was to open an existing file
> of which the size would be unknown.

I'm sorry to say that the only reliable solution is to directly call
WinAPI OpenFileMappingW() and MapViewOfFile() using a foreign-function
interface (FFI) package such as ctypes or CFFI. I can write up an
example if you need help.

In theory, shared_memory.SharedMemory() in the multiprocessing package
could support this case. But in practice it's still based on the mmap
module. It tries to improve the situation by opening and mapping an
existing mapping via WinAPI OpenFileMappingW() [1] and MapViewOfFile()
[2] in order to query the region size via VirtualQuery(), but this
isn't reliable because it returns the size rounded up to the nearest
page boundary. Calling mmap.mmap(-1, size, tagname=name) with this
queried size may fail the internal MapViewOfFile() call with
ERROR_ACCESS_DENIED (5) if the size is bigger than the maximum size of
the file mapping. (The underlying status from the NT system call more
clearly describes the failure as STATUS_INVALID_VIEW_SIZE.)

The problem with the standard library mmap module and PyWin32's
mmapfile is that they're designed to open a mapping only via
CreateFileMappingW() [3]. When this function is called with hFile as
INVALID_HANDLE_VALUE (i.e. use the system paging file) and the maximum
size as 0 (i.e. dwMaximumSizeHigh=0, dwMaximumSizeLow=0), it always
fails with ERROR_INVALID_PARAMETER (87). It does not check whether
lpName is the name of an existing mapping before validating the
parameters and failing. Hypothetically, it could have been written to
check for this case, since the file handle is ignored anyway for an
existing mapping. But it is what it is, and has been so for about 30
years.

A high-level library function needs to be aware of this limitation and
handle it appropriately by implementing an open disposition (e.g.
create-new, open-always, open-existing) that can dictate the use of
OpenFileMappingW() in addition to or instead of CreateFileMapping().

---
[1] https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-openfilemappinga
[2] https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-mapviewoffile
[3] https://docs.microsoft.com/en-us/windows/win32/api/memoryapi/nf-memoryapi-createfilemappingw