[Numpy-discussion] Github migration?

[Jason McCampbell]

Chuck,

I will update the wiki page on the Numpy developer site that discusses the
refactoring this week.  Right now what's there reflects our plans before
they met the reality of code.  Needless to say, the actual implementation
differs in some of the details.

Here is a very brief overview of the structure:

- The libndarray directory now contains all of the code for the 'core'
library.  This library is independent of Python and implements most of the
array, descriptor, iterator, and ufunc functionality.  The goal is that all
non-trivial behavior should be in here, but in reality some parts are tied
fairly tightly to the CPython interpreter and will take more work to move
into the core.

- numpy/core/src/multiarray and numpy/core/src/umath now implement "just"
the CPython interface to libndarray.  We have preserved both the Python
interface and the C API.  Ideally each C API function is just a simple
wrapper around a call to the core API, though it doesn't always work out
that way. However, a large amount of code has been moved out of these
modules into the core.

- The core is built as a shared library that is independent of any given
interface layer.  That is, the same shared library/DLL can be used with
CPython, IronPython and any other implementation.  Each interface is
required to pass in a set of callbacks for handling reference counting,
object manipulation, and other interface-specific behavior.

- The core implements its own reference counting semantics that happen to
look very much like CPython's.  This was necessary to make the core library
independent of the interface layer and preserve performance (ie, limit the
number of callbacks).  The handshaking between interface and core is a bit
complicated but ends up working nicely and efficiently for both reference
counted and garbage collected systems.  I'll write up the details on the
wiki page.

- As Travis mentioned we are also working on a .NET back end to Cython.
 This lets us port the modules such as MTRAND without having to have two
separate interfaces, a Cython and a .NET version.  Instead, we can modify
the existing .pyx file to use the new core API (should improve performance
in CPython version slightly).  Once done, Cython can generate the .NET and
CPython interfaces from the same .pyx file.

We have done a fair amount of cleanup on the naming conventions but
certainly more needs to be done!

I'll write it up for everyone this week but feel free to email me with other
questions.

Regards,
Jason


On Mon, Aug 23, 2010 at 9:54 PM, Charles R Harris <charlesr.harris at gmail.com
> wrote:

>
>
> On Mon, Aug 23, 2010 at 2:30 PM, Travis Oliphant <oliphant at enthought.com>wrote:
>
>>
>> Hi all,
>>
>> I'm curious as to the status of the Github migration and if there is
>> anything I can do to help.  I have a couple of weeks right now and I would
>> love to see us make the transition of both NumPy and SciPy to GIT.
>>
>> On a slightly related note, it would really help the numpy-refactor
>> project if it received more input from others in the community.   Right now,
>> the numpy-refactor is happening on a public github branch (named
>> numpy-refactor) awaiting numpy itself to be on github.   It would be more
>> useful if the numpy-refactor branch were a regular branch of the github
>> NumPy project.
>>
>> The numpy-refactor project is making great progress and we have a working
>> core library that can be built on Windows, Mac, and Linux.     The goal is
>> for this numpy-refactor to become the basis for NumPy 2.0 which should come
>> out sometime this Fall.     Already, a lot of unit tests have been written
>> and code coverage has increased on the core NumPy code which I think we all
>> agree is "a good thing"   In addition, some lingering reference count bugs
>> (particularly in object arrays) have been found and squashed.
>>
>> There is also some good progress on the Cython backend for .NET which
>> would allow and also put pressure on migration of most of SciPy to
>> Cython-or-Fwrap generated extension modules.
>>
>> I am looking forward to working on NumPy a little more over the coming
>> months.
>>
>> All the best,
>>
>>
> I've been having some fun browsing through the branch but don't have much
> to say on such short acquaintance.
>
> I wonder if the patch in ticket #1085<http://projects.scipy.org/numpy/ticket/1085>might be something you folks could look at now that the loops have been
> moved about and such? Also, it would be nice if the extended comment style
> was rigidly adhered to, although things look pretty good in that department.
> Another nit would be to keep an eye out during the cleanups for "if (blah)
> foo;" if statements and clean them up by putting the foo on a separate line
> when it is convenient to do so. Apart from that it looks like Ilan and Jason
> are really getting into it and doing a nice job of regularizing the naming
> conventions and such which should make the code easier to read and maintain.
> Adding some explanatory comments along the way would also help as it may be
> awhile before someone else looks so closely at the code as to have a good
> understanding of what the different functions do. I know that gets in the
> way of getting the code out, but I throw it out as a wish.
>
> Would it be possible to sketch out how the general layering of
> functionality looks at this point? Is there going to be any cleanup of the
> visible macros?
>
> Chuck
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20100825/4c5e59b2/attachment.html>