[Python-ideas] API for VM branches (was: PEP 3146)

[M.-A. Lemburg]

Collin Winter wrote:
> On Tue, Feb 2, 2010 at 2:02 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> Collin Winter wrote:
> [snip]
>>> If such a restrictive plugin-based scheme had been available when we
>>> began Unladen Swallow, I do not doubt that we would have ignored it
>>> entirely. I do not like the idea of artificially tying the hands of
>>> people trying to make CPython faster. I do not see any part of Unladen
>>> Swallow that would have been made easier by such a scheme. If
>>> anything, it would have made our project more difficult.
>>
>> I don't think that it has to be restrictive - much to the contrary,
>> it would provide a consistent API to those CPython internals and
>> also clarify the separation between the various parts. Something
>> which currently does not exist in CPython.
> 
> We do not need an API to CPython's internals: we are not interfacing
> with them, we are replacing and augmenting them.

Right, and that's the idea behind making the VM pluggable: you
define the boundaries and interfaces of the components you want
to replace and then have Python redirect all calls to your
implementation by linking against your VM instead of the default
one.

>> Note that it may be easier for you (and others) to just take
>> CPython and patch it as necessary. However, this doesn't relieve
>> you from the needed maintenance - which, I presume, is one of the
>> reasons why you are suggesting to merge U-S back into CPython ;-)
> 
> That is incorrect. In the year we have been working on Unladen
> Swallow, we have only updated our vendor branch of CPython 2.6 once,
> going from 2.6.1 to 2.6.4. We have occasionally cherrypicked patches
> from the 2.6 maintenance branch to fix specific problems. The
> maintenance required by upstream CPython changes has been effectively
> zero.

Patch level releases usually don't include any disruptive
changes to the code, so I'm not sure whether your experience
with going from 2.6.1 to 2.6.4 provides a good example of
what maintaining compatibility with Python's main line of
development really means in terms of effort.

There are a few things to consider:

 * if you want to stay compatible with Python, you have to
   provide updated releases of your version for every
   supported Python version (including all patch level releases)

 * you need to build releases with more or less the same
   frequency as CPython

 * you need to closely follow Python trunk development if
   want to avoid lagging behind Python's releases by a few
   months

and all this for a long period of time.

Otherwise, I don't see how you could convince users to
use your branch instead of the Python mainline version.

> We are seeking to merge with CPython for three reasons: 1) verify that
> python-dev is interested in this project, and that we are not wasting
> our time; 2) expose the codebase to a wider, more heterogenous testing
> environment; 3) accelerate development by having more hands on the
> code. Upstream maintenance requirements have had zero impact on our
> planning.

I'm sure that python-dev is interested in your project.

I'm not sure whether it would be fair to go with your approach
instead of the other existing branches, such as Stackless or
wpython.

How easy would it be for those projects to continue supporting
Python if U-S were to be merged into main line Python ?

> In any case, I'll be interested in reading your PEP that outlines how
> the plugin interface should work, which systems will be pluggable, and
> exactly how Unladen Swallow, WPython and Stackless would benefit.
> Let's move further discussion of this to python-ideas until there's
> something more concrete here. The py3k-jit branch will live long
> enough that we could update it to work with a plugin system, assuming
> it is demonstrated to be beneficial.

Perhaps I'll write a PEP outlining the idea in more detail.
However, without support from developers of such VMs, I don't
see much reason in trying to design something that will not
eventually get used.

I have a feeling that I'm not getting the point across:
by making it easier for VM developers to hook into the CPython
system, we'd all benefit.

VM developers could rely on a consistent interface to develop against,
python-dev would know where non-mainline VMs hook into the system
and users could choose the VM that best fits their needs.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Feb 03 2010)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

::: Try our new mxODBC.Connect Python Database Interface for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/