[Python-Dev] mmap

[Fredrik Lundh]

Greg wrote:
> Actually, my preference is to see a change to open() rather than a whole
> new module. For example, let's say that you open a file, specifying
> memory-mapping. Then you create a buffer against that file:
> 
>   f = open('foo','rm')  # 'm' means mem-map
>   b = buffer(f)
>   print b[100:200]
> 
> Disclaimer: I haven't looked at the mmap modules (AMK's and Mark's) to see
> what capabilities are in there. They may not be expressable soly as open()
> changes. (adding add'l params for mmap flags might be another way to
> handle this)
> 
> I'd like to see mmap native in Python. I won't push, though, until I can
> run a test to see what kind of savings will occur when you mmap a .pyc
> file and open PyBuffer objects against the thing for the code bytes. My
> hypothesis is that you can reduce the working set of Python (i.e. amortize
> the cost of a .pyc's code over several processes by mmap'ing it); this
> depends on the proportion of code in the pyc relative to "other" stuff.

yes, yes, yes!

my good friend the mad scientist (the guy who writes code,
not the flaming cult-ridden brainwashed script kiddie) has
considered writing a whole new "abstract file" backend, to
entirely get rid of stdio in the Python core.  some potential
advantages:

-- performance (some stdio implementations are slow)
-- portability (stdio doesn't exist on some platforms!)
-- opens up for cool extensions (memory mapping,
   pluggable file handlers, etc).

should I tell him to start hacking?

or is this the same thing as PyBuffer/buffer (I've implemented
PyBuffer support for the unicode class, but that doesn't mean
that I understand how it works...)

</F>

PS. someone once told me that Perl goes "below" the standard
file I/O system.  does anyone here know if that's true, and per-
haps even explain how they're doing that...