Executing python script stored as a string

[Steven D'Aprano]

On Mon, 31 Aug 2009 16:29:45 -0700, Ecir Hana wrote:

> Hello,
> 
> please, how to execute a python script stored as a string? But let me
> impose several limitations, so simple "exec" wont work:
> 
> - if I understood it correctly defining a function in the string and
> exec-ing it created the function in current scope. This is something I
> really don't want


You can pass in a global and local namespaces to exec as arguments:

>>> x = 4
>>> ns = {'x': 4}
>>> exec "x += 1" in ns
>>> x
4
>>> ns['x']
5


See the docs for details.


> - simple exec also blocks the rest of the program


Run it in a thread.


> - I also would like the string to be able to use and return some parts
> of the caller

You can copy the parts of the current scope into the namespace you pass 
to exec, then later copy the revised values out again.

But are you sure you really want to take this approach? exec is up to ten 
times slower than just executing the code directly. And if the string is 
coming from an untrusted source, it is a *huge* security risk.



> Couple of points:
> 
> - the script in string should behave just like any other ordinary python
> script executed in separate process, except it should also know about a
> function caller "up". Nothing else. (I read that something similar is
> possible while embedding python into your C project - that you could
> invoke the VM and provide some default "imports")

If you want it to execute in a separate *process*, that's a whole 
different question. If you do that, you get separation of code for free, 
as well as separate namespaces. My approach would be to have a special 
module "common" which subprocesses can import, to get access to the 
shared functions. You will probably need to create some sort of message 
passing infrastructure to get results out of the subprocess into the 
parent process. 


> - if the other script runs in separate process how should it call the
> remote function? And how to pass its arguments? I really hope I don't
> have to serialize every communication, maybe I should use threading
> instead of process? 

If you want separate processes, they're *separate*. Threads are not.


> All I want is that running it wont block the caller
> and that it cannot modify callers code/variables/scope (apart from
> calling the predefined callers' functions). Or maybe even better, let it
> block the caller but provide a way to stop its execution?

As far as I know, you can't kill threads, you can only ask them to kill 
themselves.


> - how to know that the script finished? I was thinking about atexit() -
> could it work here?

I doubt it. You would need to poll each thread to see if it has completed.



> Think of it as a text editor with a special ability to execute its
> content, while providing access of some of its functionality to the
> script.

Something like this?

In the text editor, you have contents:

text goes here
and more text
# Python script starts here
x = 'a'
up(x)
print "foo"
# Python script stops here
more text again


and the user selects lines 4 and 5 and chooses the command "Execute". The 
script executes, and its output (foo) is appended to the end of the file:

text goes here
and more text
# Python script starts here
x = 'a'
up(x)
print "foo"
# Python script stops here
more text again
foo

Is this what you mean?

If so, I think you are making this much too complicated for such a simple 
use-case. Just publish an API which the script can use, and have the main 
text editor application specify a "script" namespace containing only that 
API. That could be a module:

>>> import math  # pretend this is your API shared module
>>> exec "myvalue = 42" in math.__dict__
>>> math.myvalue
42



Then execute the text using exec, but don't bother about putting it into 
a thread or subprocess. That just makes it harder to implement, and you 
have to worry about concurrency issues.



-- 
Steven