[Python-Dev] Choosing a best practice solution for Python/extension modules

[Nick Coghlan]

Brett Cannon wrote:
> 
> 
> On Sat, Feb 21, 2009 at 11:43, <glyph at divmod.com
> <mailto:glyph at divmod.com>> wrote:
> 
>     On 07:07 pm, brett at python.org <mailto:brett at python.org> wrote:
> 
>         On Sat, Feb 21, 2009 at 09:17, Jean-Paul Calderone
>         <exarkun at divmod.com <mailto:exarkun at divmod.com>>wrote:
> 
> 
>                 But there is another issue with this: the pure Python
>                 code will never call
>                 the extension code because the globals will be bound to
>                 _pypickle and not
>                 _pickle. So if you have something like::
> 
> 
>                  # _pypickle
>                  def A(): return _B()
>                  def _B(): return -13
> 
> 
>                  # _pickle
>                  def _B(): return 42
> 
> 
>                  # pickle
>                  from _pypickle import *
>                  try: from _pickle import *
>                  except ImportError: pass
> 
> 
>             This is really the same as any other high-level/low-level
>             library split.  It doesn't matter that in this case, one
>             low-level implementation is provided as an extension module.
>             Importing the low-level APIs from another module and then
>             using them to implement high-level APIs is a pretty common,
>             simple, well-understood technique which is quite applicable
>             here.
> 
> 
>         But that doesn't provide a clear way, short of screwing with
>         sys.modules, to
>         get at just the pure Python implementation for testing when the
>         extensions
>         are also present. The key point in trying to figure this out is to
>         facilitate testing since the standard library already uses the
>         import *
>         trick in a couple of places.
> 
> 
>     You don't have to screw with sys.modules.  The way I would deal with
>     testing this particular interaction would be a setUp that replaces
>     pickle._A with _pypickle._A, and a tearDown that restores the
>     original one.
> 
>     Twisted's TestCase has specific support for this.  You would spell
>     it like this:
> 
>       import _pypickle
>       # ...
>       testCase.patch(pickle, '_A', _pypickle._A)
> 
>     You can read more about this method here:
> 
>     http://python.net/crew/mwh/apidocs/twisted.trial.unittest.TestCase.html#patch
> 
> 
> My worry with this approach is that while this works nicely if you are
> only overriding a single function, having to do this for all functions
> and classes in order to make sure you are testing the extension code
> with all the extension code instead of intermingled extension/Python
> code. So a function that did this automatically for the entire module
> would be needed, which is like what I proposed in my use_extension function.
> 
> I am seeing two approaches emerging. One is where pickle contains all
> Python code and then uses something like use_extension to make sure the
> original Python objects are still reachable at some point. This has the
> drawback that you have to use some function to make the extensions
> happen and there is some extra object storage.
> 
> The other approach is having pickle contain code known not to be
> overridden by anyone, import _pypickle for stuff that may be overridden,
> and then import _pickle for whatever is available. This approach has the
> perk of using a standard practice for how to pull in different
> implementation. But the drawback, thanks to how globals are bound, is
> that any code pulled in from _pickle/_pypickle will not be able to call
> into other optimized code; it's a take or leave it once the call chain
> enters one of those modules as they will always call the implementations
> in the module they originate from.

I'd actually say there's a third option which is still viable: continue
with the current Foo/_Foo practice for optimised modules, and provide a
function in test.support to get the original Python version's code out
of Foo.

That actually has the virtue of directly testing that the ImportError
for the missing module is being trapped correctly.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------