[Python-Dev] Bridging strings from Python to other languages

[Bill Bumgarner]

[This is a continuation of the thread that Just mentions below -- 
"NSString & mutability".  I finally had a chance to write enough code 
to figure out where the walls were that I kept bloodying my head 
against.  I believe I also came up with a more constructive way to 
think about this whole problem. More below -- the end of the 'bridging 
strings' section contains a series of what I believe are issues with 
the current Python implementation that should/may/could be addressed in 
a future version]

On Tuesday, Feb 4, 2003, at 22:13 US/Eastern, Just wrote on python-dev:
> (The use case is this. The PyObjC project marries Objective-C with
> Python. This is cool as it gives us direct access to almost all of
> Cocoa, the native OSX GUI interface. However, Cocoa defines its own
> string type and for reasons that are waaay beyond the scope of this 
> post
> (check the archives of the pyobjc-dev list if you're really really
> interested; see a recent thread called "NSString & mutability") it
> appears a bad idea to _convert_ these strings to Python unicode 
> strings.
> So we need to wrap them. Yet they should work as much like unicode
> strings as possible...)

Let me rephrase the problem in slightly different terms.   This will be 
long winded-- skip down to the 'bridging strings' section if you don't 
want to go through the initial discussion of the challenges of bridging 
two runtimes...

In creating a bridge between Python and other languages-- in this case, 
Objective-C-- the general goal is to provide seamless connectivity 
between the two runtime environments.   That is, you want to have a 
proxy to objects or structures found in the 'alien' runtime available 
in the 'native' runtime in a fashion that makes the proxy convenient to 
use.   Generally, this means that the proxy should act as much like the 
'native' runtime up to the point where it starts to obfuscate the 
behavior of the 'alien' runtime.   While a decent bridging and proxying 
mechanism can make "crossing the bridge" easy to do, one can never 
avoid the fact that there really is a bridge and on the other side 
there really is an 'alien' runtime.

Now-- there are a number of different ways to proxy objects/structures 
between the two runtimes:

- pure proxy:   the object/structure to be bridged is represented by a 
proxy that handles all requests for information or invocation of 
functions/methods by converting the request/invocation into a form that 
can be understood on the other side of the bridge.

Example -- the following creates a python native proxy to the alien 
Objective-C NSMutableArray instance (ignore that it really creates an 
NSCFArray-- that is an internal-to-Foundation implementation detail 
that is irrelevant).  The expression 'a.count()' actually causes the 
'count' Objective-C method to be invoked through the proxy a:

 >>> from Foundation import *
 >>> a = NSMutableArray.array()
 >>> type(a)
<objective-c class NSCFArray at 0x466d0>
 >>> a.count()
0
 >>> len(a)
0

The len(a) is just a demonstration of how far the proxying can go by 
defining the appropriate internal(?) attributes on the proxy.

- pure conversion:   the object/structure to be bridged is converted to 
the native type as it crosses the bridge.

Example -- the NSString is currently bridged to the Python String class 
such that string instances are converted to their native types as they 
cross the bridge [at least, this is the case in CVS -- I now have a 
proxy class that can wrap a Python PyString/PyUnicode instance and 
present it is a standard NSString instance on the ObjC side.  Avoids 
lots of unnecessary data copying when going from Python->Objective-C, 
but it needs a bunch of cleanup before I commit.]:

In the following, I create a new Objective-C NSString instance and 
assign it to 's'.  What results is a copy of the contents of the 
NSString instnace shoved into a normal Python string.

 >>> s = NSString.stringWithString_("Foobar")
 >>> type(s)
<type 'str'>

- mixed conversion/proxy:   this is a suboptimal case.   It generally 
converts to a native type in one direction, but potentially not in the 
other or not fully.

Example -- NSNumber is currently in this category.   It should change 
eventually, but there are issues to deal with:

 >>> a = NSArray.arrayWithObject_(1)
 >>> a[0]
<NSCFNumber objective-c instance 0x6704b0>
 >>> a[0] + 1
2

---

One of the key challenges is that proxying effectively causes two 
references to any given object to exist;   the native object reference 
and the 'alien' reference through the proxy.   Care must be taken to 
ensure that a single reference on either side of the bridge is enough 
to preserve both components of the hunk of data while also ensuring 
that the existence of a proxy without references does not prevent the 
item from being collected [may sound confusing:  consider the situation 
where a Python class is subclassed in the alien environment or 
vice-versa -- you effectively end up with instances that have part of 
their implementation in one runtime and the other part in the other 
runtime.  It can lead to issues.].

In general, these kinds of issues can be worked through by leveraging 
mechanisms such as weak references.  By providing a callback on the 
finalization of an object, it is possible to ensure that the 
alien-to-python component of the instance is destroyed, as well.

A final challenge is that sometimes an object's type or contents are 
completely irrelevant to a piece of code.   It is the object reference 
itself that is meaningful.  In these situations, if an object is passed 
across the bridge and back, what should come back really should be what 
went across in the first place-- if not, the contents may have been 
preserved, but the object's original identity has been lost.

Sometimes an object is just an object.

---

Strings provide a particular set of challenges in that no two runtime 
environments present exactly the same set of features in their string 
handling API, yet every runtime has some kind of a string API and, 
invariably, that API is very much at the core of the runtime.   The 
addition of Unicode to every string API over the last decade+ has not 
made things any simpler.

In python, strings are immutable and can encapsulate non-unicode data.  
  A separate unicode-- also immutable-- type is provided to encapsulate 
unicode data, but the standard string type can also encapsulate unicode 
data in certain circumstances [at least, it appears that PyString will 
happily consume and represent UTF8].

In Objective-C [and other languages], there is a single String class 
that can encapsulate both ASCII and unicode data in many different 
encodings.   Furthermore, there is a subclass of String that provides 
additional mutability API -- an instance of the mutable string class 
can have its contents changed by the developer while the identity of 
the object remains the same (unlike python where appending "b" to "a" 
results in a new string "ab").

To further complicate matters, most typed languages support the concept 
of 'upcasting' -- that is, of casting a particular instance to actually 
be an instance of a superclass.   For Objective-C, it can mean that a 
method that is declared as returning an immutable string or array 
actually returns a mutable string or array instance -- as long as the 
developer pays attention to the compiler warnings and doesn't do any 
stupid casting of their own, everything is fine.   Java offers similar 
casting "features".

- bridging strings -

So, how to bridge strings in such an environment?   In all cases, we 
can [fortunately] assume that strings pass across the bridge in one of 
a few choke points in the code -- that there is always a location to 
add a little bit of logic with which to help bridge the string [or any 
other random object].

The goal is to bridge strings in a fashion such that (not really in 
order of importance):

     (1) only one hunk of memory is used to contain the data within the 
string

     (2) conversion is kept to a minimum, if present at all, because 
strings will be passed back-and-forth across the bridge very frequently

     (3) identity is maintained;   pass a string with id() 7570720 from 
Python into the alien runtime and subsequently from the alien runtime 
back into python and the same string instance with id() 7570720 really 
should come back

     (4) 'alien' string specific API can still be used;   the 
Objective-C NSString provides a very rich API, including localization 
features that are not available in pure python.

For Python->Objective-C, bridging strings has proven to be fairly easy. 
  (1), (2), and (4) are quite straightforward.   (3) is not done yet.

For Objective-C->Python, bridging strings is not so easy.  The 
difficulty is compounded by certain features of the Python 
string/unicode APIs.

(1) is pretty easy -- the challenge is to figure out which API to call 
on the Python side such that the resulting Python object does not copy 
and re-encode the data.   If that is unavoidable, the cost of encoding 
or conversion (2) should be minimized [hopefully with a cache so that 
cost of conversion is paid once, then never again for immutable string 
instances].  There is also the ongoing challenge of determining when to 
use the PyString vs. PyUnicode APIs;  it seems that unicode objects are 
not welcome everywhere that string objects are?

(4) is actually quite easy and has been available for some time through 
the use of unbound methods.   However, the current implementation in 
CVS will always cause the python string to be converted to an NSString, 
the method invoked, and then the result-- if any and if a string-- is 
converted back to a python string.

(3) is not so easy-- at least, not from what I have determined so far.  
Most of the issues seem to be due to limitations in Python (which is 
really just another way of saying "I don't know enough to approach this 
problem from the right direction"):

     - can't use weakref because one can't have a weak reference to a 
string or unicode object.   This means that a callback when a string 
ref is finalized on the python side is not possible.   It also means 
that creating a hash between ObjC string instances and Python string 
instances can't be done without using strong references, thereby 
creating the potential for leaking memory.

     - can't subclass string (but can unicode) to provide a class that 
acts exactly like a regular string while containing a reference to the 
foreign string object.  There doesn't appear to be anywhere to hide a 
hunk of data in the string instance, either.

     - can't use the character buffer APIs because a character buffer 
cannot be used consistently throughout the python APIs in the same 
places as a string.  Using str() to turn a char buffer into a string 
violates (1) [and doesn't make much sense anyway].

End result -- it is very difficult to preserve the association between 
an alien string instance and a PyString instance consistently.  Even if 
PyString instances provide very thorough and consistent hashing 
behavior where two strings with the same contents always hash the same, 
the same cannot be said of all alien environments.   Even when it is 
true, there are cases where the developer may be relying on the 
identity of the object to not change outside of their control.

Mutable strings obviously present issues of their own, but they are not 
particularly relevant to discussion on python-dev outside of how future 
development might make the support of such common idiosyncrasies a bit 
more straightforward. Ideally, one could have an object on the python 
side that looks/feels/smells like a string instance, but whose contents 
may change.  This creates any number of exciting problems.  To further 
compound problems, anything that is declared as returning an NSString 
*may* return an NSMutableString at whim.  It doesn't happen often, but 
when it does, if the handling of mutable vs. immutable strings is too 
radically different, it'll cause code to blow up in highly unexpected 
and very difficult to debug ways.

Rambling on....
b.bum