[Python-Dev] mmap module

[Paul Barrett]

Tim Peters wrote:
> 
> [Paul Barrett]
> > In the CVS log of the mmapmodule.c, Tim Peters says:
> >
> > "The code really needs to be rethought from scratch (not by me, though
> > ...)."
> 
> That was in specific reference to the code I changed, in mmap_find_method.
> The difficulty is that mmap is great for "large files", but the code before
> my change used a C int for the starting offset and also for the return      > value; I boosted those to a C long, which covers 63 bits on 64-bit Linux     > boxes, but doesn't help 64-bit Windows at all (where a C long remains 4      > bytes).  The mmap_object struct uses size_t to declare the relevant members, > which is possibly better still than C long, but may still leave platform     > capabilities out of reach for large files (e.g., "even Win95" *allows*       > specifying 64-bit offsets when creating a mapped file view).  C is a         > friggin' mess here, and Python's PyArg_ParseTuple() and Py_BuildValue()     > don't cater to the full range of C integral types anyway.  In other words,  > if this code is ever to reach its full potential, it "really needs to be     > rethought from scratch".

OK, thanks for the clarification.

> > The ability to have offsets into a file that are not multiples of the
> > system pagesize would also be nice.
> 
> It's OS-specific.  Python should grow warts to protect against it on the     > OSes that care.

Well, hopefully the OS-differences wouldn't prevent implementing a
more abstract interface.

> > I'd be willing to submit a PEP on a new mmapmodule, once I know what
> > others would like.
> 
> Hard to say.  This has the potential to become Python's next thread
> subsystem, i.e. an endless and ultimately hopeless x-platform nightmare.  If
> you do write a PEP, I vote to say that we'll cover Windows and Linux (and
> maybe Mac OS X?) out of the box, but any other platform is at your own risk
> (it doesn't really help if somebody pops up volunteering to support a
> minority platform, because they eventually go away, their code stops         > working, and it never gets fixed -- so it's use-at-your-own-risk in reality
> regardless).

Yes, I agree.  Windows, Unix/Linux, and Mac OS X should be the
supported platforms.

My intention is not to make major changes to the Python interface, but
to fix bugs and to implement some additional features, such as a
non-pagesize file offset.  I'll try to get something written up in the
near future.

-- 
Paul Barrett, PhD      Space Telescope Science Institute
Phone: 410-338-4475    ESS/Science Software Group
FAX:   410-338-4767    Baltimore, MD 21218