[Numpy-discussion] Introduction to Scott, Jason, and (possibly) others from Enthought

[Joe Harrington]

On Tue, 25 May 2010 15:54:26 -0500, Travis Oliphant <oliphant at enthought.com> wrote: 

>On May 25, 2010, at 2:50 PM, Charles R Harris wrote:
>
>>
>>
>> On Tue, May 25, 2010 at 1:37 PM, Travis Oliphant <oliphant at enthought.com 
>> > wrote:
>>
>> Hi everyone,
>>
>> There has been some talk about re-factoring NumPy to separate out the
>> Python C-API layer and make NumPy closer to a C-library.   I know
>> there are a few different ideas about what this means, and also that
>> people are very busy.  I also know there is a NumPy 2.0 release that
>> is in the works.
>>
>> I'm excited to let everyone know that we (at Enthought) have been able
>> to find resources (about 3 man months) to work on this re-factoring
>> project and Scott and Jason (both very experienced C and Python
>> programmers) are actively pursuing it.    My hope is that NumPy 2.0
>> will contain this re-factoring (which should be finished just after
>> SciPy 2010 --- where I'm going to organize a Sprint on NumPy which
>> will include at least date-time improvements and re-factoring work).
>>
>> While we have specific goals for the re-factoring, we want this
>> activity to be fully integrated with the NumPy community and Scott and
>> Jason want to interact with the community as much as feasible as they
>> suggest re-factoring changes (though they both have more experience
>> with phone-conversations to resolve concerns than email chains and so
>> some patience from everybody will be appreciated).
>>
>> Because Jason and Scott are new to this mailing list (but not new to
>> NumPy),  I wanted to introduce them so they would feel more
>> comfortable posting questions and people would have some context as to
>> what they were trying to do.
>>
>> Scott and Jason are both very proficient and skilled programmers and I
>> have full confidence in their abilities.   That said, we very much
>> want the input of as many people as possible as we pursue the goal of
>> grouping together more tightly the Python C-API interface layer to
>> NumPy.
>>
>> I will be involved in some of the discussions, but am currently on a
>> different project which has tight schedules and so I will only be able
>> to provide limited "mailing-list" visibility.
>>
>>
>> I think 2.0 would be a bit early for this. Is there any reason it  
>> couldn't be done in 2.1? What is the planned policy with regards to  
>> the visible interface for extensions? It would also be nice to have  
>> a rough idea of how the resulting code would be layered, i.e., what  
>> is the design for this re-factoring. Simply having a design would be  
>> a major step forward.
>
>The problem with doing it in 2.1 is that this re-factoring will  
>require extensions to be re-built.   The visible interface to  
>extensions will not change, but there will likely be ABI  
>incompatibility.    It seems prudent to do this in NumPy 2.0.    
>Perhaps we can also put in place the ABI-protecting indirection  
>approaches that David C. was suggesting earlier.
>
>Some aspects of the design are still being fleshed out, but the basic  
>idea is to separate out a core library that is as independent of the  
>Python C-API as possible.    There will likely be at least some  
>dependency on the Python C-API (reference counting and error handling  
>and possibly others) which any interface would have to provide in a  
>very simple Python.h -- equivalent, for example.
>
>Our purpose is to allow NumPy to be integrated with other languages or  
>other frameworks systems without explicitly relying on CPython.     
>There are a lot of questions as to how this will work, and so much of  
>that is being worked out.   Part of the reason for this mail is to  
>help ensure that as much of this discussion as possible takes place in  
>public.
>
>-Travis
>
>
>>
>> In any case, I think the primary goal for 2.0 should remain the  
>> python3k port.
>>
>> Chuck
>>
>> _______________________________________________
>> NumPy-Discussion mailing list
>> NumPy-Discussion at scipy.org
>> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>--
>Travis Oliphant
>Enthought Inc.
>1-512-536-1057
>http://www.enthought.com
>oliphant at enthought.com
>

Chuck wants to make 2.0 be the py3k release.  I expressed a desire
some time ago to hold 2.0 until we had reviewed docs, on the basis
that this should have been done for 1.0, and 2.0 is our "real 1.0" for
this and other reasons.  Now Travis wants to refactor the C code in a
way that breaks extensions (another "real 1.0" item).  But Chuck wants
to get his project out the door soon, and not without justification.

NumPy is not a small project anymore and it would make sense not only
to have an open community process (which we sort-of do, except when we
don't), but to define that process and to conduct more of it in plain
view, on the web site, rather than hiding it here in the list archives
where only subscribers are likely to see it.  There are many
interested parties (users) who are not on the lists.  It's fine (even
good) not to commit to specific dates until we know them for sure, but
the process and rough timeline should be laid out publicly.

I hope we can agree on one item, which is that the 2.0 release should
be one that we are not retroactively embarrassed about.  This means it
should be well tested and that the docs should be up to date for the
new stuff, including examples.  Whether the docs need to be reviewed
or not is a different question; they should at least not be wrong.

It's May 25.  SciPy 2010 starts 28 June, just one month out.  Chuck
and Travis, would you be amenable to making a 2.0 release page on the
web site and share there your timelines for making these major
changes, including testing and documenting them?  It would also be
good to see any other planned changes listed out, the 2.0 release
cutoff date, and a timeline for the release testing process.

Thanks,

--jh--