With io getting rewritten as an extension module, I think it's time to try to come up with a good best practice scenario for how to be able to control when a module uses a pure Python implementation and when it uses extension module optimizations. This is really only important for testing as if the extension is missing then the pure Python version is just flat-out used.<br>
<br>As an example, let's just go with pickle and the Pickler class, with _pickle as the extension module.<br><br>If you look at examples in the standard library, their seems to be two approaches. One is simply to blast the pure Python version:<br>
<br> class Pickler: pass<br><br> try: from _pickle import Pickler<br> except ImportError: pass<br><br>This is bad, though, as the only way to get a pure Python version for testing is to clear out pickle and _pickle from sys.modules, put None in for sys.modules['_pickle'] and then import pickle again. Yuck.<br>
<br>The other option is to hide the pure Python version::<br><br> class _Pickler: pass<br><br> try: from _pickle import Pickler # pickle actualy imports *<br> except ImportError: Pickler = _Pickler<br><br>Better, but it still means that you are mucking around with hidden names and it hard-codes what part of the module gets replaced (using import * gets around this, but it also blasts things like __doc__ which you probably don't want).<br>
<br>Now, from what I can tell, Antoine is suggesting having _pyio and a _io and then io is simply:<br><br> try: from _io import *<br> except ImportError: from _pyio import *<br><br>That works for testing as you can then have test classes have an attribute for the module to use and then create two subclasses which set what module to use (kind of like how test_warnings currently does it). But this only really works for complete module replacements, not modules like pickle where only key portions have been rewritten (which happens more often than the complete rewrite).<br>
<br>So here is my crazy idea that I came up with late last night (i.e. might not make a lot of sense).<br><br>First, the module with the pure Python code is the main version. At the end of that module, you make a function call: ``use_extension(__name__, '_pickle')``. That function then does some "magic"::<br>
<br>def use_extension(py_name, ext_name):<br> try:<br> ext = importlib.import_module(ext_name)<br> except ImportError:<br> return<br> py = sys.modules[py_name]<br> swapped = {} <br> for name in (x for x in dir(ext) if not x.startswith('__')):<br>
swapped[name] = getattr(py, name)<br> setattr(py, name, getattr(ext, name))<br> py.__extension__ = ext_name, swapped<br><br>You can also have an undo_extension('pickle') and it will unroll what was changed. This makes choosing what version of a module to use very simple in tests as it is a single function call in one direction or another. And doing it this way also allows for different VMs to choose different things to replace. For instance IronPython might decide that most of pickle is fine and only want to change a single function with an extension; this solution lets them do that without it being hard-coded in the standard library. At worst other VMs simply need to refactor the Python code so that there is a class or function that can be replaced.<br>
<br>So go ahead and tear this apart so that we can hopefully reach a consensus that makes sense so that at least testing can easily be done.<br><br>-Brett<br>