Thanks Martin, Martin v. Löwis wrote:

...

I've started by looking at the parser portion of the code. However I am not certain this is the best place to start. Since there are so many ports I assume there is a well trodden path to completing this kind of task.

I believe this assumption is wrong. There are not many ports, only a handful (or less - Jython, IronPython, PyPy). While Jython and IronPython may have similar implementation strategies, I would expect that PyPy took an entirely different approach.

In any case, there certainly is a step that you apparently failed to perform as the very first step: set some explicit goals. What kind of compatibility do you want to achieve in your port, what other goals would you like to follow?

I thought I'd try and keep my message short so I decided not to go into the explicit objectives. At the most basic it is the ability for developers to run compiled Python as part of the game code. The next step up from that is allowing Python source code to execute and be modified in a 'simple' interactive coding tool: allowing for 'tweaking' code to be implemented outside of the game engine team. Principal constraint: Microsoft support for independent development on the 360 is only provided through the use of their slimmed down .Net compact framework and the XNA Game Studio Express development environment (C# only). This allows Microsoft to implement security within the tool chain and deployment pipeline to sandbox strictly.

IOW, why is IronPython not what you want (it *is* a port of CPython to C#, in a sense), and why is the C# support in PyPy not good enough for you?

The impact, to this project, of the reduced API and strict sandboxing in the 360 dev environment is Python implementations like IronPython are not feasible. IronPython uses the reflection capabilities of C# to interpret directly to CLR. Without reflection IronPython simply cannot operate. Unfortunately the 360 API does not include reflection functionality. I had a look into PyPy and concluded that it could produce a result that would operate however I was less certain about integrating it into a development tool chain for the 360. It seems more likely that a 'C#Python' would result in a cleaner development environment - much like the embedded inclusion of Lua scripting in many games software.

...

I would prefer to break the task into portions that can be verified (tested for correctness) independently or as a stack (one on top of the next). That way I can catch errors early and have more confidence in what I am creating.

As I don't know what you want to achieve, it is difficult to tell you what steps to take.

I assume your implementation would be similar to CPython in that it uses the same byte code format. So one path would be to ignore the compiler at all, and assume that the byte code format is given, i.e. start with port ceval.c.

I'm not sure whether you also want to provide the same low-level API (i.e. whether you want to provide "Embedding and Extending"); it surely can't be the *same* API, since your's will be C#, whereas CPython's is, well, C. If you implement ceval.c, you will find quickly that you need much of the Objects folder, so implementing the 10 or so most important objects would be the natural starting point (type, int, string, tuple, dict, frame, code, class, method - assuming you would target Python 1.5 first, i.e. no bool, cell, descr, gen, iter, weakref, unicode, object).

...

When I looked through the test suites they all seem to be written in Python. Is there a test suite for the core of CPython i.e. before the C code can interpret Python code?

Yes and no. The core Python is tested through compilation - if it compiles without warnings on the relevant compilers, it is considered good enough to run the Python test suite. For selected features of the interpreter, there are specific tests, in particular test_capi.

The core of CPython (compiler, objects, builtins) is then tested through Python code.

This seems like a sensible way to start since the test harness needs a Python interpreter. Although it seems counter-intuitive to build the bytecode interpreter so that I can test the bytecode compiler...

Regards, Martin

Thanks for the advice Martin. Regards, Ty

Thanks Martin, Martin v. Löwis wrote:

...

I've started by looking at the parser portion of the code. However I am not certain this is the best place to start. Since there are so many ports I assume there is a well trodden path to completing this kind of task.

I believe this assumption is wrong. There are not many ports, only a handful (or less - Jython, IronPython, PyPy). While Jython and IronPython may have similar implementation strategies, I would expect that PyPy took an entirely different approach.

In any case, there certainly is a step that you apparently failed to perform as the very first step: set some explicit goals. What kind of compatibility do you want to achieve in your port, what other goals would you like to follow?

I thought I'd try and keep my message short so I decided not to go into the explicit objectives. At the most basic it is the ability for developers to run compiled Python as part of the game code. The next step up from that is allowing Python source code to execute and be modified in a 'simple' interactive coding tool: allowing for 'tweaking' code to be implemented outside of the game engine team. Principal constraint: Microsoft support for independent development on the 360 is only provided through the use of their slimmed down .Net compact framework and the XNA Game Studio Express development environment (C# only). This allows Microsoft to implement security within the tool chain and deployment pipeline to sandbox strictly.

IOW, why is IronPython not what you want (it *is* a port of CPython to C#, in a sense), and why is the C# support in PyPy not good enough for you?

The impact, to this project, of the reduced API and strict sandboxing in the 360 dev environment is Python implementations like IronPython are not feasible. IronPython uses the reflection capabilities of C# to interpret directly to CLR. Without reflection IronPython simply cannot operate. Unfortunately the 360 API does not include reflection functionality. I had a look into PyPy and concluded that it could produce a result that would operate however I was less certain about integrating it into a development tool chain for the 360. It seems more likely that a 'C#Python' would result in a cleaner development environment - much like the embedded inclusion of Lua scripting in many games software.

...

I would prefer to break the task into portions that can be verified (tested for correctness) independently or as a stack (one on top of the next). That way I can catch errors early and have more confidence in what I am creating.

As I don't know what you want to achieve, it is difficult to tell you what steps to take.

I assume your implementation would be similar to CPython in that it uses the same byte code format. So one path would be to ignore the compiler at all, and assume that the byte code format is given, i.e. start with port ceval.c.

I'm not sure whether you also want to provide the same low-level API (i.e. whether you want to provide "Embedding and Extending"); it surely can't be the *same* API, since your's will be C#, whereas CPython's is, well, C. If you implement ceval.c, you will find quickly that you need much of the Objects folder, so implementing the 10 or so most important objects would be the natural starting point (type, int, string, tuple, dict, frame, code, class, method - assuming you would target Python 1.5 first, i.e. no bool, cell, descr, gen, iter, weakref, unicode, object).

...

When I looked through the test suites they all seem to be written in Python. Is there a test suite for the core of CPython i.e. before the C code can interpret Python code?

Yes and no. The core Python is tested through compilation - if it compiles without warnings on the relevant compilers, it is considered good enough to run the Python test suite. For selected features of the interpreter, there are specific tests, in particular test_capi.

The core of CPython (compiler, objects, builtins) is then tested through Python code.

This seems like a sensible way to start since the test harness needs a Python interpreter. Although it seems counter-intuitive to build the bytecode interpreter so that I can test the bytecode compiler...

Regards, Martin

Thanks for the advice Martin. Regards, Ty