[Python-Dev] mmap module
Paul Barrett
Barrett@stsci.edu
Mon, 21 May 2001 08:17:09 -0400
Tim Peters wrote:
>
> [Paul Barrett]
> > In the CVS log of the mmapmodule.c, Tim Peters says:
> >
> > "The code really needs to be rethought from scratch (not by me, though
> > ...)."
>
> That was in specific reference to the code I changed, in mmap_find_method.
> The difficulty is that mmap is great for "large files", but the code before
> my change used a C int for the starting offset and also for the return > value; I boosted those to a C long, which covers 63 bits on 64-bit Linux > boxes, but doesn't help 64-bit Windows at all (where a C long remains 4 > bytes). The mmap_object struct uses size_t to declare the relevant members, > which is possibly better still than C long, but may still leave platform > capabilities out of reach for large files (e.g., "even Win95" *allows* > specifying 64-bit offsets when creating a mapped file view). C is a > friggin' mess here, and Python's PyArg_ParseTuple() and Py_BuildValue() > don't cater to the full range of C integral types anyway. In other words, > if this code is ever to reach its full potential, it "really needs to be > rethought from scratch".
OK, thanks for the clarification.
> > The ability to have offsets into a file that are not multiples of the
> > system pagesize would also be nice.
>
> It's OS-specific. Python should grow warts to protect against it on the > OSes that care.
Well, hopefully the OS-differences wouldn't prevent implementing a
more abstract interface.
> > I'd be willing to submit a PEP on a new mmapmodule, once I know what
> > others would like.
>
> Hard to say. This has the potential to become Python's next thread
> subsystem, i.e. an endless and ultimately hopeless x-platform nightmare. If
> you do write a PEP, I vote to say that we'll cover Windows and Linux (and
> maybe Mac OS X?) out of the box, but any other platform is at your own risk
> (it doesn't really help if somebody pops up volunteering to support a
> minority platform, because they eventually go away, their code stops > working, and it never gets fixed -- so it's use-at-your-own-risk in reality
> regardless).
Yes, I agree. Windows, Unix/Linux, and Mac OS X should be the
supported platforms.
My intention is not to make major changes to the Python interface, but
to fix bugs and to implement some additional features, such as a
non-pagesize file offset. I'll try to get something written up in the
near future.
--
Paul Barrett, PhD Space Telescope Science Institute
Phone: 410-338-4475 ESS/Science Software Group
FAX: 410-338-4767 Baltimore, MD 21218