How to port Python to VxWorks

[Paul Fernhout]

Pythonistas-

I just got a bare bones port of Python running on VxWorks and thought
I'd share some insights in how to do it (OK, and brag a bit too. :-).
For various reasons (the work was done for a client) I can't make the
code available. Still, I thought some general comments on this process
might be useful. Previous postings indicate that there have been at
least two other ports of Python to VxWorks. However, I haven't seen any
code for this anywhere or a discussion of the specific issues for
VxWorks. 

Background: 

VxWorks is a Real Time Operating System (RTOS), meaning that you can in
theory make "real time" applications on it that respond to time critical
events in a (guaranteed) timely fashion. It is typically used to develop
applications for embedded computing devices like cell phones, Mars
rovers, and other "invisible" computers. To the developer, VxWorks looks
a lot like UNIX(tm). VxWorks has been called the "Cadillac" of embedded
OSs.

VxWorks development is often done using a cross compiler, typically
using a product from WindRiver Systems called Tornado, which is a
development shell around the GNU tool suite (GCC, GDB). The board you
are developing for is commonly called the "target"; the workstation you
develop on is called the "host". You download the code compiled on the
host to the target (via serial, ethernet, or whatever).

Why Python on VxWorks? Because there is quite a bit of things to know to
compile under VxWorks, the developer license are expensive, and the
turnaround time to download to a target is long. Python allows one to
prototype under for example Windows and then run the prototype on the
target. It also allows rapid development and refinement of test code,
which can just be downloaded via a terminal emulator to the Python
command prompt on the target (or via other means). Naturally, there is
still lots of stuff to do in C (especially Interrupt Service Routines
(ISRs) and code that must operate quickly). 

Note that Java and Smalltalk on the target would also offer these
advantages. However, Python is a true OO language that is comfortable
looking to C coders, and the Python license is easier to deal with in
many ways for embedded systems than the Java or Smalltalk licenses, and
of course Python has many other uses as well.

Issues:

The Python source for 1.5.2 comes complete with pretty much everything
you need for the port. However, naturally you will have to already be
able to do serial I/O on your hardware and have your own make files.

Because of cross compiling approach (host != target), the normal Python
"configure" script concept doesn't work. Under a typical UNIX, the
configure script will create a bunch of C files, compile them, and
evaluate the results to see how long integers are, what header files are
present, and additional system dependent information -- all
automatically. The output is a "configure.h" file, produced by that
script modifying "configure.h.in". 

Since VxWorks is typically hosted on a machine other than the target
board, this approach won't work, since at best you'd get settings for
the host, not the target. So, you have to go in and manually change
"configure.h", starting from "configure.h.in". 

My initial attempt a while back to start with the Windows version of
configure.h file led to great confusion. I'd strongly suggest there is
no substitute for going line by line down that file and set the proper
flags. That's what I did that worked. 

I referred to the online documentation to determine what
functions are supported. I looked for each header file in the Tornado
directory. I put in type (int, long, long long) sizes from knowledge of
the target processor. This process took several hours (after a first
false start a while back). I commented out undefs I didn't want;
otherwise I #define constants to 1 in most cases. 

I used an existing VxWorks makefile, and modified it to compile all the
*.c files in a directory and link them together with a main program
which called the code in "python.c". 

One of the problems with porting to VxWorks I had the first time I tried
it was not knowing what modules to include. I had first tried a
subtractive approach starting from including everything, and was
overwhelmed with errors. 

This time I tried for the minimal set, by copying files as the compiler
wanted me about missing items, starting from just having "Python.c". I
faked the argv, argc arguments, passing an argc of 0 and an argv of an
array with a "python.exe" string and a 0. (Just a null for argv ends up
with a crash after starting.)

After some experimentation, I ended up with a directory that had
just about all the files from the python directory, the compiler
directory, and a few from the module directory. A few files needed to be
removed, because they duplicate VxWorks functionality; you'll see errors
when you try to link. 

Now that I know what files are needed for a minimal build, I could
create a VxWorks make file that would use the files in place in the
conventional Python directory hierarchy. It would make sense to have a
list like this somewhere.

I was very surprised when it booted the second time. (The first time it
crashed due to passing a null for argv and I had to debug it). It gave
several error messages, due to not having file support on the target (I
think). But it ran interactively on the serial port, and I was able to
add numbers, define functions, and print results. I now can try adding
in more modules. The first time I tried linking in all the modules a
while back (when trying a subtractive approach to porting) I had all
sorts of compile errors in various modules, so I expect getting other
facilities like TCP/IP working may be a bit of a further adventure. 

What surprised me most was that I did not have to do anything special
about "freezing" the application. I did not use freezemain.c or run any
special converters. Of course, as I want more library support from *.py
files, I may have to do this. I'm not even sure at this point what
builtin modules I have to work with.

Overall, this minimal port was doable in a day, but only after much
previous time spent learning about the source, and also learning from a
previous failed attempt. Obviously, getting more Python facilities
operational under VxWorks will take longer.

Particular stumbling blocks:

I didn't know what thread choice to make. I ended up not including a
thread file at all and turning the thread option off. I would like to
improve on this, so I could have real Python threads under VxWorks.

"dup2.c" gave me problems because I didn't have a fcntl function. I
ended up making this just return BAD_RETURN (or whatever the error
constant was in that function). Since I don't have file support anyway
(I think the calls just fail), I figured this would not hurt.

Note, this is written from home from memory. There may have been some
other minor hurdles overcome in the process.

Some architectural suggestions for Python 2 based on Squeak Smalltalk:

Squeak is a portable Smalltalk. I have also put Squeak on embedded
hardware. From this and other experience, I would suggest a few things
to consider for Python architecture in the future (Python 2)?

* Consider having a clear separation between C that does library calls
(files, threads, network, any i/o) and code that does not. When writing
code to do file stuff for example, have it call something like
"PyOS_Open()", where that is defined elsewhere, (as opposed to just
doing an ANSI "open"). This would allow the bulk of the files to be pure
C, and only a few files to have these OS calls. This would localize
porting issues and make the whole task more approachable. Platform
specific modules might not sensibly be done this way; my emphasis here
is on easily porting the core Python system. Obviously, in some cases
this woudl have a perfomance impact; one apporach is to typedef some of
these calls so they can be relaces by ANSI or POSIX calls in place.

In the case of Squeak Smalltalk, the pure C code is actually generated
from a Smalltalk simulator of the VM (written in a Smalltalk subset that
mainly manipulates byte arrays and dispatches bytecodes). The actual
machine dependent part of Squeak (as of about a year ago) is only about
2000 lines of C (implementing bitmap display, file handling, mouse
handling, sound, and networking). This porting task is relatively
straightforward -- just modify these 2000 lines for your system. Note
that Squeak also provides a complete GUI, but with emulated widgets.

* Consider having the Python compiler implemented in Python. It can then
be compiled to Python byte code, so it doesn't have to be linked in. 
It can just be loaded at startup. Of course, this will be slower, but it
will remove one more porting hurdle.

* Size. Squeak can deliver an entire system with GUI, decompiled source
code browsing, compiler, development tools, debugger, and more in a
complete package of about 1 MB to 2 MB. Python seems to be much bigger
(0.5 MB non-debug DLL for Windows, and many MB of source). I think the
complete package was about 600K for the core Python (and VxWorks w/ WDB
debugger agent) without extra modules. That is going to grow rapidly if
I link in more modules. Clearly there is more that can be done to shrink
the base Python system implementation for embedded targets. Some of this
may be a time/space tradeoff. For example, the previous suggestion
regarding the compiler might result in a smaller but slower compiler
system. 

Anyhow, thanks to Guido and others for such a wonderful system!

-Paul Fernhout
Kurtz-Fernhout Software 
=========================================================
Developers of custom software and educational simulations
Creators of the GPL Garden with Insight(TM) garden simulator
http://www.kurtz-fernhout.com