[Python-Dev] mmap module

[Tim Peters]

[Paul Barrett]
> In the CVS log of the mmapmodule.c, Tim Peters says:
>
> "The code really needs to be rethought from scratch (not by me, though
> ...)."

That was in specific reference to the code I changed, in mmap_find_method.
The difficulty is that mmap is great for "large files", but the code before
my change used a C int for the starting offset and also for the return value;
I boosted those to a C long, which covers 63 bits on 64-bit Linux boxes, but
doesn't help 64-bit Windows at all (where a C long remains 4 bytes).  The
mmap_object struct uses size_t to declare the relevant members, which is
possibly better still than C long, but may still leave platform capabilities
out of reach for large files (e.g., "even Win95" *allows* specifying 64-bit
offsets when creating a mapped file view).  C is a friggin' mess here, and
Python's PyArg_ParseTuple() and Py_BuildValue() don't cater to the full range
of C integral types anyway.  In other words, if this code is ever to reach
its full potential, it "really needs to be rethought from scratch".

> Well, I might be the person to do the rethinking, but I'd first like
> to know what Tim has in mind.

Nothing that you did <wink>.

> I've been playing around with this module lately and tend to agree
> that some enhancements could be made, particularly to prevent "bus
> errors" and "segmentation faults".

When you get one of those, it's a bug in Python!

> The ability to have offsets into a file that are not multiples of the
> system pagesize would also be nice.

It's OS-specific.  Python should grow warts to protect against it on the OSes
that care.

> I'd be willing to submit a PEP on a new mmapmodule, once I know what
> others would like.

Hard to say.  This has the potential to become Python's next thread
subsystem, i.e. an endless and ultimately hopeless x-platform nightmare.  If
you do write a PEP, I vote to say that we'll cover Windows and Linux (and
maybe Mac OS X?) out of the box, but any other platform is at your own risk
(it doesn't really help if somebody pops up volunteering to support a
minority platform, because they eventually go away, their code stops working,
and it never gets fixed -- so it's use-at-your-own-risk in reality
regardless).