[Numpy-discussion] NumPy re-factoring project

[Sturla Molden]

I have a few radical suggestions:

1. Use ctypes as glue to the core DLL, so we can completely forget about 
refcounts and similar mess. Why put manual reference counting and error 
handling in the core? It's stupid.

2. The core should be a plain DLL, loadable with ctypes. (I know David 
Cournapeau and Robert Kern is going to hate this.) But if Python can 
have a custom loader for .pyd files, so can NumPy for it's core DLL. For 
ctypes we just need to specify a fully qualified path to the DLL, which 
can be read from a config file or whatever.

3. ctypes will take care of all the mess regarding the GIL. Again there 
is no need to re-invent the wheel. ctypes.CDLL releases the GIL when 
calling into C, and re-acquires before making callbacks to Python. 
ctypes.PyDLL keeps the GIL for legacy library code that are not threadsafe.

ctypes will also make porting to other Python implementations easier (or 
even other languages: Ruby, JacaScript) easier. Not to mention that it 
will make NumPy impervious to changes in the Python C API.

4. Write the core in C++ or Fortran (95/2003), not C. ANSI C (aka C89). 
Use std::vector<> instead of malloc/free for C++, and possibly templates 
for generics on various dtypes.

5. Allow OpenMP pragmas in the core. If arrays are above a certain size, 
it should switch to multi-threading.

Sturla






Den 10.06.2010 15:26, skrev Charles R Harris:
>
> A few thoughts came to mind while reading the initial writeup.
>
> 1) How is the GIL handled in the callbacks.
> 2) What about error handling? That is tricky to get right, especially 
> in C and with reference counting.
> 3) Is there a general policy as to how the reference counting should 
> be handled in specific functions? That is, who does the reference 
> incrementing/decrementing?
> 4) Boost has some reference counted pointers, have you looked at them? 
> C++ is admittedly a very different animal for this sort of application.
>
> Chuck
>
>
> _______________________________________________
> NumPy-Discussion mailing list
> NumPy-Discussion at scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>    

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20100610/82a67b40/attachment.html>