Load Python Scripts from Memory

[Alex Martelli]

Dean Ellis wrote:

> I'm new to python and I have a question. Is it possible to load a
> script/module from memory. I have a requirement in the application I
> am building for scripts to be downloaded from a web site and run
> (these can be pre-compiled scripts). I would rather not have to save
> these scripts to disk then load them if possible.

Yes, you set your own function __import__ in lieu of the built-in
one, and in that function you can do absolutely anything you want.

If what you get from the website is simply the source, or the code
object to be executed in order to define your module (serialized and
deserialized as you prefer), then your life is reasonably easy; if
due to some weird constraint you have to get less easily dealt-with
things, such as the memory image of a .pyc file, you may have more
work to do, of course (basically, getting the code-object from the
"pyc in-memory image", after possibly checking and skipping its header), 
but still, it _is_ a feasible task.


> I would also like to add a feature where a compiled module could be
> loaded directly from a resource file.

I'm not quite sure what a "resource file" _IS_.  I know about the
"resources" that can be hidden in a Windows .EXE file, for example
(and I think win32all has code to let you get at them), but of
course those are NOT "a file", so I suspect you may be thinking of
something else.


> Anyone have any ideas on how this could be done?

Once you clarify exactly what constraints you're operating under,
I'm sure some design can be found to satisfy them.

Suppose, for example, that what you want to do is as follows.  If
the Python code at any time imports a module whose name starts with
_web_, then a pickled version of said module (either a source string
or a code object) is obtained (e.g. via urrlib, etc, etc) and must
be placed as a module in sys.modules &c -- and you also want to do
the usual optimization whereby a second import uses the module that
is already in sys.modules, etc, etc.  Let's suppose further that you
need not deal with packages & their complications, reload, and so on.
Oh, and, if the name starts with _res_ instead, you will get its
pickled version via some other way (some kind of function that reads
stuff in memory from a "resource file", whatever THAT may be).

Oh, incidentally, we also assume that you trust the code you receive
*blindly* -- it has somehow been entirely validated, and you're quite
willing to let it do ANYthing the current user is authorized to do
(if that's not the case, then you do have a _serious_ problem...).

Under these conditions you might do something like...:

import __builtin__
base_importer = __builtin__.__import__

import cPickle, new, sys

def __import__(name, *args):
    if name in sys.modules:
        return sys.modules[name]
    if name.startswith('_web_'):
        pickled = get_pickled_from_web(name[5:])
    elif name.startswith('_res_'):
        pickled = get_pickled_from_resources(name[5:])
    else:
        return base_importer(name, *args)
    module = new.module(name)
    unpickled = cPickle.loads(pickled)
    exec unpickled in module.__dict__
    sys.modules[name] = module
    return module

__builtin__.__import__ = __import__


Run this code once, e.g. at program startup, and assuming the
get_pickled_... functions know how to get a pickled sourcecode
string or codeobject from wherever, the rest should work (I'm
typing the code in from memory, not testing it, so if there's
any silly typo I apologize in advance... but still I hope this
can help!).

So, assuming there is something that stops you from using this
very simple solution, you can (with our collective help) move
on from here.  Except that if the problem is that you do NOT
fully trust the code you have been sent, then the situation may
not be solvable to your safisfaction...


Alex