[python-win32] C extension class wrapping and sequence questions

[Jens B. Jorgensen]

Mark English wrote:

>Hi,
>I'm a bit stumped about how to develop this problem, so I'll overburden
>you with questions and information, and appreciate any answers you may
>have.
>
>We use a networking "Stream" to send and receive "Messages".
>I want to wrap streams and messages in Python, so that ultimately I can
>do the following at a prompt:
>
>  
>
>>msgMine = PyMessages.MsgFoo(args_msg)
>>streamMine = PyStream.Stream(args_stream)
>>streamMine.Send(msgMine)
>>    
>>
>
>There are a lot of messages and associated structures, currently
>implemented in C++. The C++ is actually auto-generated from a
>proprietary format. There has been an earlier attempt to wrap the
>messages in Python which works to some extent, although it requires a
>fairly high-level interface that runs in an executable with Python
>embedded in it, and is ultimately pretty difficult to use. It's also
>hand-coded, and therefore error-prone.
>
>I'd like to auto-generate my code, and have some experience of Python,
>and plenty of experience with C++, mostly under Windows. I have very
>little experience with writing Python C extensions.
>
>So here are the questions:
>1) How should I aim to construct my code ? Should the bulk of the
>intelligence lie in the Python code, ultimately resulting in something I
>can pass to a C-extension "Stream" type which will know how to create
>the appropriate C++ "Message" and send it off ? This means no access to
>the underlying C++ implementation of messages which may not be a great
>loss seeing as how it is autogenerated anyway. On the other hand, it
>also means no access to all the associated C++ message handling code
>which IS useful, and not really easily wrapped in its own set of Python
>classes.
>  
>
I don't think there's any easy rule. I look at it more this way: I want 
to end up with a nice Python interface (nice means object that work like 
you'd want a Python object to work. This rules out SWIG. By the way even 
though the extension API is C I write *all* of my extensions in C++ 
including classes, etc. I say if there's useful code in C++ then use it. 
But consider what you'd like the Python API to look like. Will doing 
that require a lot code? A lot of people do write a thin wrapper module 
that's pretty nearly 1-to-1 to the C/C++ interface they're trying to get 
in Python and then write some python code around that to make things 
look nice.

If you have some system that auto-generates C++ how hard would it be to 
get that same code to generate a Python extension written in C++? This 
might be the best solution of all!

>Alternatively, I could make the Python "Message" classes thin-wrappers
>around the C++ code. Each "Message" class would have its own Python
>type, and the C module would export constructors for the types. This
>seems reasonably attractive. I could then override "__setattr__()" so
>that programmers could either write:
>  
>
>>msgMine.SetFoo("Bar")
>>    
>>
>OR
>  
>
>>msgMine.strFoo = "Bar"
>>    
>>
>and the underlying implementation would call the same code in C++.
>Because this is autogenerated, I don't mind going this route.
>
>2) How do I handle "Message" instances which contain sequences
>(especially sequences of auto-generated structures) ? For example, if
>the "Message" contains an array of "Record" instances, then this is
>fine:
>  
>
>>msgMine.arrayRec[0].strFoo = "Bar"
>>strSomething = msgMine.arrayRec[0].strFoo
>>    
>>
>because I can return a sequence of thin wrapper objects for
>"msgMine.arrayRec" from my C module.
>
>Similarly this also is fine:
>  
>
>>arrayRec = PyRecords.Record(args_record) msgMine.SetArrayRec(index,
>>arrayRec)
>>    
>>
>since I am accessing the array through the owner "Message".
>
>However, I can't see an easy way around this problem:
>  
>
>>arrayRec = PyRecords.Record(args_record) #Construct a new record 
>>object msgMine.arrayRec[0] = arrayRec #Put it in the sequence directly
>>    
>>
>
>If I simply build a sequence of thin wrappers for "msgMine.arrayRec",
>then setting one of its members is effectively changing a temporary
>object. I could write my own sequence wrapping object (like a Python
>class inheriting from "UserList"), perhaps by providing implementations
>for the sequence type, so that setting an attribute actually calls a
>method in the owner C++ "Message" object. This seems like hard work for
>something that would be easy to do in Python.
>  
>
Although I haven't done it myself to date it seems to be straightforward 
to implement the sequence "api" in a Python object. In practical terms 
this means your object type structure contains a non-null 
PySequenceMethods structure containing function pointers that perform 
the various sequence operations you support. In your case I would 
imagine that your python Message object would have an attribute arrayRec 
that would either be another object that would pretty much just 
implement the sequence operations. That object could keep a pointer to 
the real message object to handle the insertion and deletion.

>Alternatively, if I go with a Python-centric implementation then code
>like that above isn't a problem. The message doesn't get converted to
>its equivalent C++ object until the stream needs to do so.
>
>Trying to frame the questions a bit more generally then, when wrapping
>an existing C++ class hierarchy, how do you choose between thin Python
>wrappers (which may not leverage all that Python has to offer), and
>"more Python less C++" (which may leave out some of the C++
>functionality). Also, how do C extension types present instances which
>contain sequences of other instances ?
>
>I suspect if you're any good you do it either way you prefer and don't
>mess anything up, or leave anything out, but for a first-timer this
>isn't really an option, and I'd like to get some minimal functionality
>up and about.
>
>I'm looking at following the approach used in win32ui where types are
>factories, and contain enough mapping intelligence to go back and forth
>between C and python. I should be able to autogenerate the types. This
>doesn't solve my owner-of-a-sequence problem though, and I'm worried
>I'll hit other as yet unforeseen problems down the road. Similarly the
>SWIG implementation seems to make arrays read-only, and you have to
>write your own code with typemaps if you want to expand on that
>implementation.
>
>Any thoughts, or pointers to modules which solve similar problems ?
>
>  
>
What you're trying to do can be difficult. I can imagine if you've got a 
complex object library where the C++ objects are doing lots of stuff 
together it's going to be tough to figure out where the Python interface 
figures in. As an overall approach maybe you think of the Python object 
interface as more or less a view object graph of C++ objects you are 
manipulating. You create the python objects as needed to provide that 
view of the objects but still just let the C++ objects works as 
normally. I can see how problems like what to do when a Python Record 
object is created from Python as in your above example as compared to 
when a Record is created within your C++ code in some spot. One might be 
built with a Python wrapper that's attached to it while the other does 
not. In that respect then perhaps this idea of the python objects as a 
view of the underlying objects may work best. Treat the Python objects 
as transient artifacts whose lifetime may or may not be coincident with 
the lifetime of the underlying C++ objects. Create them as needed and 
destroy them as needed. This still leaves some problems with how to 
handle c++ objects that are only referenced through the Python objects. 
How do we know when to delete the C++ object? Well, I think I have an 
idea what your trying to accomplish and unfortunately I haven't tried to 
tackle something like this these are the best suggestions I can give.

-- 
Jens B. Jorgensen
jens.jorgensen at tallan.com

"With a focused commitment to our clients and our people, we deliver value through customized technology solutions."