[Python-Dev] Choosing a best practice solution for Python/extension modules

[Brett Cannon]

On Sat, Feb 21, 2009 at 11:43, <glyph at divmod.com> wrote:

> On 07:07 pm, brett at python.org wrote:
>
>> On Sat, Feb 21, 2009 at 09:17, Jean-Paul Calderone <exarkun at divmod.com
>> >wrote:
>>
>
>  But there is another issue with this: the pure Python code will never call
>>>> the extension code because the globals will be bound to _pypickle and
>>>> not
>>>> _pickle. So if you have something like::
>>>>
>>>
>   # _pypickle
>>>>  def A(): return _B()
>>>>  def _B(): return -13
>>>>
>>>
>   # _pickle
>>>>  def _B(): return 42
>>>>
>>>
>   # pickle
>>>>  from _pypickle import *
>>>>  try: from _pickle import *
>>>>  except ImportError: pass
>>>>
>>>
>  This is really the same as any other high-level/low-level
>>> library split.  It doesn't matter that in this case, one
>>> low-level implementation is provided as an extension module.
>>> Importing the low-level APIs from another module and then
>>> using them to implement high-level APIs is a pretty common,
>>> simple, well-understood technique which is quite applicable
>>> here.
>>>
>>
>  But that doesn't provide a clear way, short of screwing with sys.modules,
>> to
>> get at just the pure Python implementation for testing when the extensions
>> are also present. The key point in trying to figure this out is to
>> facilitate testing since the standard library already uses the import *
>> trick in a couple of places.
>>
>
> You don't have to screw with sys.modules.  The way I would deal with
> testing this particular interaction would be a setUp that replaces pickle._A
> with _pypickle._A, and a tearDown that restores the original one.
>
> Twisted's TestCase has specific support for this.  You would spell it like
> this:
>
>   import _pypickle
>   # ...
>   testCase.patch(pickle, '_A', _pypickle._A)
>
> You can read more about this method here:
>
>
> http://python.net/crew/mwh/apidocs/twisted.trial.unittest.TestCase.html#patch


My worry with this approach is that while this works nicely if you are only
overriding a single function, having to do this for all functions and
classes in order to make sure you are testing the extension code with all
the extension code instead of intermingled extension/Python code. So a
function that did this automatically for the entire module would be needed,
which is like what I proposed in my use_extension function.

I am seeing two approaches emerging. One is where pickle contains all Python
code and then uses something like use_extension to make sure the original
Python objects are still reachable at some point. This has the drawback that
you have to use some function to make the extensions happen and there is
some extra object storage.

The other approach is having pickle contain code known not to be overridden
by anyone, import _pypickle for stuff that may be overridden, and then
import _pickle for whatever is available. This approach has the perk of
using a standard practice for how to pull in different implementation. But
the drawback, thanks to how globals are bound, is that any code pulled in
from _pickle/_pypickle will not be able to call into other optimized code;
it's a take or leave it once the call chain enters one of those modules as
they will always call the implementations in the module they originate from.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20090221/6ed52e48/attachment.htm>