mimicking a file in memory

[Grant Edwards]

On 2007-11-20, p. <ppetrick at gmail.com> wrote:

>> By "memory" I presume you mean virtual memory?  RAM with
>> disk-blocks as backing store? On any real OS, tempfiles are
>> just RAM with disk-blocks as backing store.
>>
>> Sound similar? The only difference is the API used to access
>> the bytes.  You want a file-I/O API, so you can either use the
>> extensively tested and and highly optimized filesystem code in
>> the OS to make disk-backed-RAM look like a file, or you can try
>> to write Python code that does the same thing.
>>
>> Which do you think is going to work faster/better?
>>
>> [The kernel is generally better at knowing what needs to be in
>> RAM than you are -- let it do its job.]
>>
>> IOW: just use a temp file.  Life will be simple. The bytes
>> probably won't ever hit the platters (if they do, then that
>> means they would have the other way too).
>
> Grant, are temp files automatically put into ram for all linux
> distros?

All files are put into ram for all linux distros that use
virtual memory.  (You'll know if you're not using virtual.)

> at any rate, i could set up ram disk. much better solution
> than using python...except that i've never done a ram disk
> before. more reading to do...

You don't have set up a ram disk.  You already have one.  All
your disks are ram disks. It's just that some of them have
magnetic platters as backing store so they get preserved during
a reboot.  On some Linux distros, the /tmp directory is a
filesystem without prmanent magnetic backing-store.  On others
it does have a permanent backing store.  If you do a "mount"
command, you'll probably see a "filesystem" who's type is
"tmpfs". That's a filesystem with no permanent magnetic
backing-store[1].

See http://en.wikipedia.org/wiki/TMPFS

/tmp might or might not be in a tmpfs filesystem (depends on
the distro).  In any case, you probably don't need to worry
about it.

Just call tempfile.NamedTemporaryFile() and tell it you want an
unbuffered file (that way you don't have to remember to flush
the file after writing to it).  It will return a file object:

  f = tempfile.NamedTemporaryFile(bufsize=0)

Write the data to that file object and flush it:

  f.write(mydata)

Pass the file's name to whatever broken library it is that
insists on a file name instead of a file-like object:

  brokenLib.brokenModule(f.name).

When you're done, delete the file object:

  del f

NB: This particular approach won't work on Windows.  On Windows
    you'll have to use tempfile.mktemp(), which can have race
    conditions.  It returns a name, so you'll have to create
    the file, write to it, and then pass the name to the broken
    module.


[1] Tmpfs pages use the swap partition for temporary backing
    store the same as for all other memory pages.  If you're
    using tmpfs for big stuff, make sure your swap partition is
    large enough to hold whatever you're doing in tmpfs plus
    whatever normal swapping capacity you need.


------------------------------demo.py------------------------------
def brokenModule(filename):
    f = file(filename)
    d = f.read()
    print d
    f.close()
    
    
import tempfile,os

f = tempfile.NamedTemporaryFile(bufsize=0)
n = f.name
print f,":",n
os.system("ls -l %s\n" % n)

f.write("hello world")
brokenModule(n)

del f
os.system("ls -l %s\n" % n)
------------------------------demo.py------------------------------

If you run this you'll see something like this:

$ python demo.py
<open file '<fdopen>', mode 'w+b' at 0xb7c37728> : /tmp/tmpgqSj8p
-rw------- 1 grante users 0 2007-11-20 17:11 /tmp/tmpgqSj8p
hello world
ls: cannot access /tmp/tmpgqSj8p: No such file or directory

    
-- 
Grant Edwards                   grante             Yow! I want to mail a
                                  at               bronzed artichoke to
                               visi.com            Nicaragua!