[Numpy-discussion] Merging the refactor.

Jason McCampbell jmccampbell at enthought.com
Thu Nov 11 17:37:53 EST 2010


Hi Chuck, Pauli,

This is indeed a good time to bring this up as we are in the process fixing
Python 3 issues and then merging changes from the master tree in preparation
for being able to consider merging the work.  More specific comments inline
below.

Regards,
Jason


On Thu, Nov 11, 2010 at 3:30 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Thu, Nov 11, 2010 at 2:08 PM, Pauli Virtanen <pav at iki.fi> wrote:
>
>> On Thu, 11 Nov 2010 12:38:53 -0700, Charles R Harris wrote:
>> > I'd like to open a discussion about the steps to be followed in merging
>> > the numpy refactor. I have two concerns about this. First, the refactor
>> > repository branched off some time ago and I'm concerned about code
>> > divergence, not just in the refactoring, but in fixes going into the
>> > master branch on github. Second, it is likely that a flag day will look
>> > like the easiest solution and I think we should avoid that.
>>
>> What is a "flag day"?
>>
>>
> It all goes in as one big commit.
>
>
>> > At the moment it seems to me that the changes can be broken up into
>> > three categories:
>> >
>> > 1) Movement of files and resulting changes to the build process.
>> > 2) Refactoring of the files for CPython.
>> > 3) Addition of an IronPython interface.
>>
>
1) and 2) are really the same step as we haven't moved/renamed existing
files but instead moved content from the CPython interface files into new,
platform-independent files.  Specifically, there is a new top-level
directory 'libndarray' that contains the platform-independent core.  The
existing CPython interface files remain in place, but much of the
functionality is now implemented by calling into this core.

Unfortunately this makes merging difficult because some changes need to be
manually applied to a different file.  Once all regression tests are passing
on the refactor branch for both Python 2.x and 3.x (3.x is in progress) Ilan
is going to start working on applying all accumulated changes.  The good
news is that 95% of our changes are to core/multiarray and core/umath and
there are relatively few changes to these modules in the master repository.

The IronPython interface lives in its own directory and is quite standalone.
 It just links to the .so from libndarray and just has a Visual Studio
solution -- it is not part of the main build for now to avoid breaking all
of the people who don't care about it.


> > I'd like to see 1) go into the master branch as soon as possible,
>> > followed by 2) so that the changes can be tested and fixes will go into
>> > a common repository. The main github repository can then be branched for
>> > adding the IronPython stuff. In short, I think it would be usefull to
>> > abandon the teoliphant fork at some point and let the work continue in a
>> > fork of the numpy repository.
>>
>> The first step I would like to see is to re-graft the teoliphant branch
>> onto the current Git history -- currently, it's still based on Git-SVN.
>> Re-grafting would make incremental merging and tracking easier. Luckily,
>> this is easy to do thanks to Git's data model (I have a script for it),
>> and I believe it could be useful to do it ASAP.
>>
>>
> I agree that would be an excellent start. Speaking of repo surgery, you
> might  find esr's latest project <http://esr.ibiblio.org/?p=2727> of
> interest.
>

We will take a look at this and the script.  There is also a feature in git
that allows two trees to be grafted together so the refactoring will end up
as a branch on the main repository with all edits. My hope is that we can
roll all of our changes into the main repository as a branch and then
selectively merge to the main branch as desired.  For example, as you said,
the IronPython changes don't need to be merged immediate.

Either way, I fully agree that we want to abandon our fork as soon as
possible.  If anything, it will go along way towards easing the merge
and getting more eyeballs on the changes we have made so far.


On Thu, Nov 11, 2010 at 3:30 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Thu, Nov 11, 2010 at 2:08 PM, Pauli Virtanen <pav at iki.fi> wrote:
>
>> On Thu, 11 Nov 2010 12:38:53 -0700, Charles R Harris wrote:
>> > I'd like to open a discussion about the steps to be followed in merging
>> > the numpy refactor. I have two concerns about this. First, the refactor
>> > repository branched off some time ago and I'm concerned about code
>> > divergence, not just in the refactoring, but in fixes going into the
>> > master branch on github. Second, it is likely that a flag day will look
>> > like the easiest solution and I think we should avoid that.
>>
>> What is a "flag day"?
>>
>>
> It all goes in as one big commit.
>
>
>>  > At the moment it seems to me that the changes can be broken up into
>> > three categories:
>> >
>> > 1) Movement of files and resulting changes to the build process.
>> > 2) Refactoring of the files for CPython.
>> > 3) Addition of an IronPython interface.
>> >
>> > I'd like to see 1) go into the master branch as soon as possible,
>> > followed by 2) so that the changes can be tested and fixes will go into
>> > a common repository. The main github repository can then be branched for
>> > adding the IronPython stuff. In short, I think it would be usefull to
>> > abandon the teoliphant fork at some point and let the work continue in a
>> > fork of the numpy repository.
>>
>> The first step I would like to see is to re-graft the teoliphant branch
>> onto the current Git history -- currently, it's still based on Git-SVN.
>> Re-grafting would make incremental merging and tracking easier. Luckily,
>> this is easy to do thanks to Git's data model (I have a script for it),
>> and I believe it could be useful to do it ASAP.
>>
>>
> I agree that would be an excellent start. Speaking of repo surgery, you
> might  find esr's latest project <http://esr.ibiblio.org/?p=2727> of
> interest.
>
> <snip>
>
> Chuck
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20101111/8d35895b/attachment.html>


More information about the NumPy-Discussion mailing list