Compiling snippets of python code

[Scott David Daniels]

Basil Brush wrote:
> I need some advice on compiling snippets of python source at runtime. So
> perhaps some python experts can point the way.
> 
> I am rewriting an old Java app in python. It tries to find solutions to
> computational problems based on the idea of genetic evolution. The Java
> version just used an array of bytes for the genomes. These byte
> sequences were bytecode which told a rudimentary VM (virtual machine)
> which maths functions to call. It used another array as a small chunk of
> virtual memory for doing the calculations.

Unlike Java and C++, functions and methods are first class.  You can
make lists and dictionaries containing functions, and that would be
the normal Python way of handling this, rather than generating source
from snippets.  You should take a look at Numpy if the "maths functions"
are operating on large chunks of data (fft, matrix operations, and the
like).  If the operations are smallish things, pure Python is likely the
way to go.  I don't understand what the "other array as a small chunk of
virtual memory" might be; my first take is YAGNI.

> Having seen the docs on python's compile module, I'm thinking of doing
> it differently in the new python version. My aim is to have the genomes
> made up of a sequence of python source snippets, with each snippet to be
> randomly generated, compiled, and executed at runtime. The advantage of
> this is that it gives me greater flexibility and I can change what
> actions the genomes can do without having to add new functions to my VM.
> 
> But there's some crucial points. When the snippet is executed I would
> need to pass in a list or array, and I need the list back again with any
> changes that were made; this is the virtual memory and how I would get
> the results back, and is also how the snippet would get data from a
> previous snippet in a sequence. 


Here is a toy-like version of what you might want to do:
     # Some operations:
     def double(vec):
         return [x * 2 for x in vec]

     def alternate(vec):
         '''An "unshuffling" step'''
         return vec[::2] + vec[1::2]

     def halve(vec):
         return [x / 2 for x in vec]

     # Some execution lists:
     processing = [double, alternate, halve, alternate]
     new_processing = alternate(processing) # you can shuffle any list

     assert new_processing == [double, halve, alternate, alternate]

     def perform(processes, data):
         for step in processes:
             data = step(data)
         return data

     start = [x * x for x in range(10)]
     assert perform(processing, start) == perform(new_processing, start)
     print start
     mid = perform(processing, start)
     print mid
     print perform(processing, mid)
     print perform(processing * 2, start)

prints:
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]
[0, 16, 64, 9, 49, 4, 36, 1, 25, 81]
[0, 49, 25, 9, 1, 64, 36, 16, 4, 81]
[0, 49, 25, 9, 1, 64, 36, 16, 4, 81]


How would I do this? Also, there is a
> database of input data. Rather than pass all that data to the snippet
> I'd rather have functions in my module which the snippet could call to
> get data. So what's the scope of an executing piece of code? Can it
> access my modules and functions?
First, let go of managing the data allocation.  Second, realize passing
a list or array is simple and cheap, and _not_ proportional to the size
of the list or array.  You are passing the actual object.  You really
need to run through the tutorial; these questions will be answered there.
Sit down with the tutorial and the interpretter, executing anything you
don't quite understand.  I did it in IDLE, but you may prefer straight
command line.

> My linux system has python 2.5 and I noticed that the docs say the
> compile module has been removed from python 3.0. So I'm concerned about
> whether my app would work in 3.0 if I upgraded. Does anyone know if they
> moved the compiler to another module or simply removed it entirely?
Although I don't think you should be working with the compiler, if
you want to use something like that, you'd be better advised to use
the "ast" module first added in 2.6.  But for now, don't borrow trouble
you don't need. Rethink your problem and write it in Python.  Don't try
to do a line-by-line translation, you'll just have a slow awkward Python
program with a Java heart.

--Scott David Daniels
Scott.Daniels at Acm.Org