[Baypiggies] [Fwd: C wrappers and the proxy dilemma]

[Ken Seehart]

Kevin B wrote:
> Looks good.  Somehow, the __overload__ also has to catch all stuff 
> coming back from c.
> In your example.. if instead of GetChildren, there was a 
> c_foo.GetAchild(). Then __overload__ would have to know the return 
> type of the c_foo function (I guess it could check the PyObject and 
> see that it is c_foo type), then automatically wrap it in a python_foo 
> object imediately on coming out of the C code.  In other words, there 
> would never be a c_foo object in python code, all c_foo objects woudl 
> automatically be wrapped/unrapped at the C Module/python boundry.
>
> Seems possible with a language change.  But will they do it... or 
> better if someone answers your final question and gives us a solution 
> without changing the language.  Nice write up.  Thanks for all the help!
>
> Kevin
>
Actually, it doesn't need to know anything about the type of any return 
values, and it never has to check anything like that at runtime.  This 
solution doesn't involve any wrappers (i.e. this solution has nothing to 
do with composition/proxies).  The point is that __overload__ actually 
modifies the behavior of the c_foo type to make it quack like Foo.  
Everything that returns c_foo instances continues to return c_foo 
instances.  Everything that takes c_foo instances as arguments continues 
to use c_foo arguments.  However all instances of c_foo magically 
acquire the attributes of Foo.  Note also that even when you call the 
Foo constructor you get a c_foo instance that gets initialized with 
Foo.__init__.  No more dualism.

In your example, the c_foo.GetAchild() would resolve 'GetAchild' by 
first searching class Foo, (where it doesn't find it), then it checks 
type c_foo and finds it.  c_foo.GetAchild() returns a c_foo instance.

Now I'm thinking that maybe this could be done without changing Python.  
It would involve some tricky hacking with struct _typeobject  in the 
Python/C API, but it seems like it should be possible to come up with a 
general solution.  Even so, I think it may be worth a PEP anyway because 
it would make wrapping C libraries in general significantly easier and 
faster.

- Ken
> Ken Seehart wrote:
>> You inspired me to write this on the main Python list.
>>
>>> Anyone who has wrapped C or C++ libraries has encountered the proxy 
>>> dilemma. 
>>>
>>> A long time ago, I naively thought that I could start by deriving my 
>>> high level python class from the c-python type, but this leads to 
>>> many difficult problems because several of the underlying methods 
>>> return the low level objects.  After reading up on the subject, I 
>>> learned that the "correct" solution is to use composition rather 
>>> than inheritance.  It makes sense, but it is nevertheless rather 
>>> annoying.
>>>
>>> An excellent example of this is the wxPython library, which uses 
>>> composition (proxies) and even solves the OOR (original object 
>>> return) problem that is associated with this kind of proxy oriented 
>>> solution.
>>>
>>> Conclusion: There is no really good solution to this kind of thing 
>>> that doesn't get messy.  Wrapping C or C++ in python becomes 
>>> difficult whenever you have methods that return (or take as 
>>> arguments) instances of the objects that you are trying to wrap in 
>>> high level python.  Any solution seems to add more overhead than is 
>>> desirable (in terms of both programmer time and run time).
>>>
>>> This problem comes up over and over again, so perhaps it is even 
>>> worth a PEP if a change to the python language would facilitate a 
>>> more convenient solution to this nagging problem.
>>>
>>> First, mentally set aside the idea of using composition.  I 
>>> understand the set of problems that it solves, but it also creates a 
>>> new set of problems (such as function call overhead on every method 
>>> call).  I also understand why composition is better than 
>>> inheritance.  But in order to contemplate this proposal, it is 
>>> necessary to temporarily set aside the composition idea.  This 
>>> proposal involves taking another look at something slightly more 
>>> akin to the inheritance approach that we all gave up on before.
>>>
>>> Okay, so here is a somewhat radical proposal:
>>>
>>> Add a writable  __overload__ attribute to low level python type.  
>>> This would be assigned a python 'type' instance (or None to have no 
>>> effect).  The effect is to place __overload__ at the /beginning /of 
>>> the __mro__ of the low level type, making it possible to directly 
>>> alter the behavior of the base python type without using 
>>> inheritance.  This means that instances of the base type acquire the 
>>> attributes of our high level python class.  It is important that the 
>>> __overload__ type is checked /before /the actual type.
>>>
>>> Yeah, it sounds a bit scary at first.  A bit too loose maybe.  So 
>>> the type definition should have to explicitly enable this feature to 
>>> prevent people from doing bad things.  But dwelling too much on the 
>>> scariness seems somewhat non-pythonic.  It is more important that we 
>>> can do really good things than that we prevent ourselves from going 
>>> out of our way to do bad things (as long as it is not too /easy /to 
>>> do bad things).
>>>
>>> So here are some of the benefits:
>>>
>>> 1. Eliminates the duality between proxy objects and low level objects.
>>>
>>> 2. Increased speed on methods whose syntaxes are not being altered 
>>> (because no wrapper is needed).
>>>
>>> 3. Much less work than writing wrappers for every method (even if 
>>> such a task is partially automated).
>>>
>>> Usage example:
>>>
>>> from foolib import c_foo
>>>
>>> class Foo(c_foo):
>>>     """Overload of c_foo"""
>>>
>>>     def __new__(typ, *args, **kwargs):
>>>         return c_foo(*args, **kwargs)
>>>
>>>     def __init__(self, parrot_state):
>>>         c_foo.__init__(self)
>>>         self.parrot_state = parrot_state
>>>
>>>     def myfunc(self):
>>>         return 5 * self.juju(self.parrot_state) # juju method is 
>>> defined in foolib
>>>
>>> # This makes all c_foo instances into Foo instances:  Poof!
>>> c_foo.c_footype.__overload__ = Foo
>>>
>>>
>>>
>>> x = Foo('not dead yet') # actually creates a c_foo instance that 
>>> looks like a Foo instance
>>>
>>> # c_foo.Add is defined in foolib and takes c_foo instances
>>> x.Add(Foo('strange'))
>>> x.Add(Foo('missing'))
>>>
>>> # c_foo.GetChildren is defined in foolib and returns c_foo instances
>>> for y in x.GetChildren():
>>>     print y.myfunc() # Wow, I am magically accessing Foo.myfunc
>>>
>>>
>>> Yeah I know, that's an infinitely recursive inheritance; but that's 
>>> okay, we just implement the __overload__ feature such that the MRO 
>>> thingy will do the right thing for us.  The semantics will be that 
>>> Foo is derived from c_foo, and all instances of c_foo behave as if 
>>> the are instances of Foo.
>>>
>>> I used the term "radical" earlier, to describe this idea.  What I 
>>> meant was that it seems to violate some basic Object Oriented 
>>> principles.  However, when evaluating solutions I try not to apply 
>>> any principles too dogmatically.  Theory is only relevant when it 
>>> has /practical /consequences.  So I recommend not quoting theory 
>>> without also defending the relevance of the theory to the case in 
>>> point.  IMO, anything that eliminates thousands of lines of 
>>> boilerplate code is worth bending theory a little.
>>>
>>> Is it possible to get the same syntactic/semantic results without 
>>> changing python?
>>>
>>> - Ken

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.python.org/pipermail/baypiggies/attachments/20070622/feb2686d/attachment.html