Request for advice: project to get NumPy working in IronPython
Hi,
At Resolver Systems, we have a product that is written in IronPython - the .NET Python implementation - and allows users to use that language to script a spreadsheet-like interface. Because they're using IronPython, they can access their existing .NET objects and libraries, which has worked out really well for us and for them. But there's an increasing number of users who would like access to CPython C extensions - in particular, NumPy.
We *don't* want to just hack something together that would just allow users of our own product to access NumPy; what we'd really like to do is work on an OSS solution that will allow all IronPython users to access NumPy transparently, just as if they were using CPython.
My question is, is this something that the readers of this list would like to get involved in, either in terms of contibuting, or even in terms on advice? We're under no illusions; this is a significant undertaking. And we're very keen on hearing from those who know NumPy better about how this should be structured, even if people don't have the time to write code for it. In particular:
* How do people feel about a source-code-compatible solution, where (perhaps) we would maintain a project that basically provided an alternative makefile for NumPy - or, even better, we could work with the NumPy developers to contribute a .NET/IronPython package. * Would it be better to try for some kind of "binary compatibility", where we'd write some kind of "glue" that sat between the existing C extension .pyd files and the IronPython engine? Our gut feeling is that this would be much more work, but we might be missing something. * What should the work's relationship to the NumPy project be?
Any thoughts, comments, or advice would be very much appreciated.
Regards,
Giles
Hello
On Mon, 15 Oct 2007, Giles Thomas wrote:
- Would it be better to try for some kind of "binary compatibility", where we'd write some kind of "glue" that sat between the existing C extension .pyd files and the IronPython engine? Our gut feeling is that this would be much more work, but we might be missing something.
Along these lines, what ever happened to Sanghyeon Seo's CPython reflector?
http://lists.ironpython.com/pipermail/users-ironpython.com/2006-June/002503....
I think it might be an avenue worth exploring, since this would also make other native modules available as a by-product. Last I heard, the JRuby folks were also trying something these lines for native MRI modules, using JNA (essentially, a Java equivalent to ctypes).
Good luck with your project. I hope it succeeds.
Cheers,
Albert
Giles Thomas wrote:
Hi,
At Resolver Systems, we have a product that is written in IronPython - the .NET Python implementation - and allows users to use that language to script a spreadsheet-like interface. Because they're using IronPython, they can access their existing .NET objects and libraries, which has worked out really well for us and for them. But there's an increasing number of users who would like access to CPython C extensions - in particular, NumPy.
An IronPython compatible version of NumPy would be great. Of course it could be done by using C# to write NumPy, but I'm not sure that this would really be any less work than creating a "glue" layer that allowed most (or all) C-Python extensions to work with IronPython.
* How do people feel about a source-code-compatible solution, where (perhaps) we would maintain a project that basically provided an alternative makefile for NumPy - or, even better, we could work with the NumPy developers to contribute a .NET/IronPython package.
Hmm.. I don't know enough about C# I guess, but would a different setup.py file really be enough? Would this have some sort of C->C# translator. I would be surprised if that actually worked, though.
Would it be better to try for some kind of "binary compatibility", where we'd write some kind of "glue" that sat between the existing C extension .pyd files and the IronPython engine? Our gut feeling is that this would be much more work, but we might be missing something.
I'm not sure that this would really be any more work than the C->C# translator that you are talking about above, but then again I've never done any C to C# translation.
* What should the work's relationship to the NumPy project be?
This is flexible. It could be distributed with NumPy or else simply advertised by NumPy.
Best regards,
-Travis Oliphant
Is there a C# <-> C (or, I guess, really a .net <-> C ) binding mechanism? Like JNI for Java?
-Chris
Christopher Barker wrote:
Is there a C# <-> C (or, I guess, really a .net <-> C ) binding mechanism? Like JNI for Java?
I don't know much about the whole .Net thing (CLR, in this case), but there is the p-invoke mechanism for that:
http://en.wikipedia.org/wiki/Platform_Invocation_Services
I don't know the performance impact. According to pypy developers, the best way to extend python in C is to use ctypes: this makes interop easier for different VM (Cpython vs CLR in this case). But in the case of numpy, this would mean quite a lot of work, I guess, because numpy is more than just a python extension, it provides a big C Api (which is the real problem in the case of VM adaptation). The best would be of course a solution which enables interop, so that if later, people want to use the work gone into numpy for IronPython, people do not have to start everything from scratch for say Jython or something else. But maybe this is too big of a task.
Giles, maybe you could see with the pypy people if they can give you some advice, too (pypy is an implementation of python in python, which can translate into different platforms, including the CLI):
http://codespeak.net/pypy/dist/pypy/doc/news.html http://lists.ironpython.com/pipermail/users-ironpython.com/2007-March/004685...
cheers,
David
We've created a Google Group to discuss this project, and if anyone's interested in contributing or advising, you're very welcome to sign up: http://groups.google.com/group/c-extensions-for-ironpython/
My replies to everyone's comments below:
David Cournapeau wrote:
Giles, maybe you could see with the pypy people if they can give you
some advice, too
Thanks for the pointer - I'd heard of PyPy, but it hadn't occurred to me that they have valuable knowledge about how to do this kind of thing - obvious now you point it out :-). I'll get in touch with them.
According to pypy developers, the best way to extend python in C is
to use ctypes:
There was an effort to get ctypes working with IronPython a while back - the CPython Reflector by Sanghyeon Seo that Albert Strasheim mentioned - Seo has just joined our mailing list, so we'll certainly start a discussion about that.
Christopher Barker wrote:
Is there a C# <-> C (or, I guess, really a .net <-> C ) binding mechanism? Like JNI for Java?
I don't know much about the whole .Net thing (CLR, in this case), but there is the p-invoke mechanism for that
P/Invoke is indeed the normal way to link C# with C; we use it in our software on those (luckily rare) occasions when we have to step outside the .NET world and talk to the basic Win32 APIs. It's pretty easy to do, and our initial thoughts about the best way to integrate IronPython with NumPy were that we would use P/Invoke to call into the C library (I agree with Travis E. Oliphant that trying to convert the existing C to C# would be a Bad Idea). Using P/Invoke would give us a very rough-and-ready .NET API, which we could then wrap with a Python library that <handwaving>interfaced with the existing CPython library in some manner</handwaving>.
I don't know the performance impact.
I don't think P/Invoke slows down the running of the C code, but of course calls from .NET to native code are quite slow. My expectation so far has been with most Python code using NumPy, the number-crunching will happen at the native code level, with relatively few switches between languages - that is, it is a "chunky" rather than a "chatty" API. Is that a reasonable assumption?
The best would be of course a solution which enables interop, so that if later, people want to use the work gone into numpy for IronPython, people do not have to start everything from scratch for say Jython or something else. But maybe this is too big of a task.
We're aiming to try to solve the "simple" case of one Python implementation, one C extension to start off with. Our intention was to then look at handling a couple of other extensions, to broaden the solution more toward a general solution for loading C extensions into IronPython - I agree that we could usefully also broaden it to handle other Python implementations.
Regards,
Giles
Travis E. Oliphant wrote:
Giles Thomas wrote:
Hi,
At Resolver Systems, we have a product that is written in IronPython - the .NET Python implementation - and allows users to use that language to script a spreadsheet-like interface. Because they're using IronPython, they can access their existing .NET objects and libraries, which has worked out really well for us and for them. But there's an increasing number of users who would like access to CPython C extensions - in particular, NumPy.
An IronPython compatible version of NumPy would be great. Of course it could be done by using C# to write NumPy, but I'm not sure that this would really be any less work than creating a "glue" layer that allowed most (or all) C-Python extensions to work with IronPython.
So can anyone inform will IronPython have bridge to NumPy or something else? And will it be available in nearest future or some weeks/months/years are required? (I'm interested in scikits.openopt being available for IronPython as well, and it requires numpy) And what about pylab for IronPython? Is it work already now, or will be available in nearest future, or the situation is undefined?
Regards, D.
On 10/26/07, dmitrey dmitrey.kroshko@scipy.org wrote:
Travis E. Oliphant wrote:
Giles Thomas wrote:
Hi,
At Resolver Systems, we have a product that is written in IronPython - the .NET Python implementation - and allows users to use that language to script a spreadsheet-like interface. Because they're using IronPython, they can access their existing .NET objects and libraries, which has worked out really well for us and for them. But there's an increasing number of users who would like access to CPython C extensions - in particular, NumPy.
An IronPython compatible version of NumPy would be great. Of course it could be done by using C# to write NumPy, but I'm not sure that this would really be any less work than creating a "glue" layer that allowed most (or all) C-Python extensions to work with IronPython.
So can anyone inform will IronPython have bridge to NumPy or something else? And will it be available in nearest future or some weeks/months/years are required? (I'm interested in scikits.openopt being available for IronPython as well, and it requires numpy) And what about pylab for IronPython? Is it work already now, or will be available in nearest future, or the situation is undefined?
Jim Hugunin, who did the original Numeric, started the IronPython project and was later hired by Microsoft to bring it to completion. He mentioned at one of the SciPy conferences that if he had it to do over again, he would do Numeric in C#. But that is not really an option for us ;)
Chuck
On 10/26/07, Charles R Harris charlesr.harris@gmail.com wrote:
On 10/26/07, dmitrey dmitrey.kroshko@scipy.org wrote:
Travis E. Oliphant wrote:
Giles Thomas wrote:
Hi,
At Resolver Systems, we have a product that is written in IronPython
the .NET Python implementation - and allows users to use that
language
to script a spreadsheet-like interface. Because they're using IronPython, they can access their existing .NET objects and
libraries,
which has worked out really well for us and for them. But there's an increasing number of users who would like access to CPython C extensions - in particular, NumPy.
An IronPython compatible version of NumPy would be great. Of course it could be done by using C# to write NumPy, but I'm not sure that
this
would really be any less work than creating a "glue" layer that
allowed
most (or all) C-Python extensions to work with IronPython.
So can anyone inform will IronPython have bridge to NumPy or something
else?
And will it be available in nearest future or some weeks/months/years are required? (I'm interested in scikits.openopt being available for IronPython as well, and it requires numpy) And what about pylab for IronPython? Is it work already now, or will be available in nearest future, or the situation is undefined?
Jim Hugunin, who did the original Numeric, started the IronPython project and was later hired by Microsoft to bring it to completion. He mentioned at one of the SciPy conferences that if he had it to do over again, he would do Numeric in C#. But that is not really an option for us ;)
FWIW, I once rewrote the core of Numeric in Java for use with Jython (then JPyhton). I gave it up after I became disillusioned with Java, but the code's still out there (http://jnumerical.sourceforge.net/) although I suspect it is pretty dated at this point.
An IronPython compatible version of NumPy would be great. Of course it could be done by using C# to write NumPy, but I'm not sure that this would really be any less work than creating a "glue" layer that allowed most (or all) C-Python extensions to work with IronPython.
I'm curious about why all the discussion is about putting Python and its extensions on top of C# and very little discussion about just using C#-based tools as an extension from CPython.
Python .NET is a great example of what I'm referring to.
The C# language and the CLR does solve some problems, but it does not solve all the problems related to scientific computing that it could. In particular, I wish it's cross-platform visibility where higher. Mono is a great start, but there are a lot of C# libraries that just don't get made to work on Linux or Mac OS X.
The task of moving scipy to sit on top of the CLR seems rather large without some automatic tool to allow calling CPython extensions from the CLR that works in a cross-platform way.
I don't really see the benefit that the CLR offers (unless all the hype is just so you can write code that runs in a browser --- in which case, are you really going to run matrix inversion on the CLR in a browser?)
How does legacy code interact with the "magic" of the CLR?
What are people's opinions about the value of NumPy and SciPy on the CLR?
-Travis O.
On 10/26/07, Travis E. Oliphant oliphant@enthought.com wrote:
An IronPython compatible version of NumPy would be great. Of course it could be done by using C# to write NumPy, but I'm not sure that this would really be any less work than creating a "glue" layer that allowed most (or all) C-Python extensions to work with IronPython.
I'm curious about why all the discussion is about putting Python and its extensions on top of C# and very little discussion about just using C#-based tools as an extension from CPython.
Python .NET is a great example of what I'm referring to.
The C# language and the CLR does solve some problems, but it does not solve all the problems related to scientific computing that it could. In particular, I wish it's cross-platform visibility where higher. Mono is a great start, but there are a lot of C# libraries that just don't get made to work on Linux or Mac OS X.
The task of moving scipy to sit on top of the CLR seems rather large without some automatic tool to allow calling CPython extensions from the CLR that works in a cross-platform way.
I don't really see the benefit that the CLR offers (unless all the hype is just so you can write code that runs in a browser --- in which case, are you really going to run matrix inversion on the CLR in a browser?)
How does legacy code interact with the "magic" of the CLR?
What are people's opinions about the value of NumPy and SciPy on the CLR?
If we were all Microsoft, all the time, it might be worth it. I run Linux myself and haven't installed Windows at home in years, so for me it is a non-starter. Keep it in C and target generic python. Portability and compiler independence is the goal.
Chuck
What are people's opinions about the value of NumPy and SciPy on the CLR?
As someone who uses Numpy/Scipy almost exclusively on Linux workstations or on clusters (in coordination with lots of C code), I wouldn't value NumPy and SciPy on the CLR at all.
I am kind of curious, though, to see how many people _would_ think it would be usefull....
S
Scott Ransom wrote:
What are people's opinions about the value of NumPy and SciPy on the CLR?
As someone who uses Numpy/Scipy almost exclusively on Linux workstations or on clusters (in coordination with lots of C code), I wouldn't value NumPy and SciPy on the CLR at all.
I am kind of curious, though, to see how many people _would_ think it would be usefull....
S
my 2 cents: let me mention that 2/3 of scipy.org/.../openopt webpages visitors have Windows installed I guess whole scipy.org visitors doesn't differ too much. However, scipy.org website admins could see their own statistic and inform us more exactly. Regards, D
Travis E. Oliphant wrote:
An IronPython compatible version of NumPy would be great. Of course it could be done by using C# to write NumPy, but I'm not sure that this would really be any less work than creating a "glue" layer that allowed most (or all) C-Python extensions to work with IronPython.
I'm curious about why all the discussion is about putting Python and its extensions on top of C# and very little discussion about just using C#-based tools as an extension from CPython.
Python .NET is a great example of what I'm referring to.
The C# language and the CLR does solve some problems, but it does not solve all the problems related to scientific computing that it could. In particular, I wish it's cross-platform visibility where higher. Mono is a great start, but there are a lot of C# libraries that just don't get made to work on Linux or Mac OS X.
The task of moving scipy to sit on top of the CLR seems rather large without some automatic tool to allow calling CPython extensions from the CLR that works in a cross-platform way.
This is exactly what Resolver hopes to achieve in the long-run.
I already have a proof-of-concept hack that already allows you to use CPython extensions from IronPython, using the Python.NET work:
http://www.voidspace.org.uk/ironpython/cpython_extensions.shtml
I had matplotlib (using numpy and Tkinter) working - but it is only a proof of concept. :-)
In the long run we would like to provide a mechanism that allows CPython extensions to be used seamlessly from IronPython. We're starting with Numpy (and not by reimplementing in C#) because we have customers who want to use it.
On the CLR subject, the CLR is a great runtime. Advantages include a powerful JIT, no GIL, an accepted (and widely used) technology in the corporate world, and a native GUI library for Windows (better than any I've seen for CPython).
Michael http://www.manning.com/foord
I don't really see the benefit that the CLR offers (unless all the hype is just so you can write code that runs in a browser --- in which case, are you really going to run matrix inversion on the CLR in a browser?)
How does legacy code interact with the "magic" of the CLR?
What are people's opinions about the value of NumPy and SciPy on the CLR?
-Travis O.
Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
What are people's opinions about the value of NumPy and SciPy on the CLR?
If anything, wouldn't the "big win" (if it's a win at all) be to get NumPy/SciPy working on top of the JVM (as T. Hochber tried)? This way it's pretty much universally portable.
I know Jython isn't as up to speed as IronPython, but the folks in Jython land are doing a pretty good job of it these days.
Their next "target" CPython compatibility will definitely be good enough for NumPy since it will be at least CPythoon 2.3 ... maybe even 2.5.
-steve
Travis E. Oliphant wrote:
An IronPython compatible version of NumPy would be great. Of course it could be done by using C# to write NumPy, but I'm not sure that this would really be any less work than creating a "glue" layer that allowed most (or all) C-Python extensions to work with IronPython.
I'm curious about why all the discussion is about putting Python and its extensions on top of C# and very little discussion about just using C#-based tools as an extension from CPython.
Python .NET is a great example of what I'm referring to.
The C# language and the CLR does solve some problems, but it does not solve all the problems related to scientific computing that it could. In particular, I wish it's cross-platform visibility where higher. Mono is a great start, but there are a lot of C# libraries that just don't get made to work on Linux or Mac OS X.
I think it is useful to separate C# and the CLR: you can do C# without the CLR (with mono at least), and you can do CLR without C#. What is C# ? At the core, it is a language with a C-like syntax with automatic memory management, and a huge framework. C is still, by far, the best language for interoperability. None (and this includes C++) comes even close. For a library, C is the best language to use if you care about interoperability, so I don't see how it would be a bad choice.
So I don't see the point of C#, at least related to numpy.
The task of moving scipy to sit on top of the CLR seems rather large without some automatic tool to allow calling CPython extensions from the CLR that works in a cross-platform way.
I don't really see the benefit that the CLR offers (unless all the hype is just so you can write code that runs in a browser --- in which case, are you really going to run matrix inversion on the CLR in a browser?)
I don't think running in a browser is the only useful thing of the CRL; I can see why this would be useful for some, but at least, for me, this is not interesting (in my current, PhD student position at least :) ). In a windows centric world, running on the CLR means having access to the whole windows framework. C# is not really an interesting language, frankly, the only thing which really matters is that it does automatic memory management, and is endorsed by MS, meaning access to the windows ecosystem. This by itself is already so huge that any other things is a detail.
No, the important point is the following: once you have an implementation on the CLR, you can use it from any language. For a less hyped based discussion of this, you can read the B. Meyer article (B. Meyer is quite involved with his company in .net, but at least, this article focus on the technology):
http://archive.eiffel.com/doc/manuals/technology/bmarticles/sd/dotnet.html
As a concrete example, look at F#: if numpy can run on the CLR, you can use it from C#, IronPython and F#. F# is, basically, OCAML on the .NET. And as much as I don't have much interest into C#, I consider F# to be much more interesting (before considering python + numpy as my environment of choice for all my numeric-based projects, I considered OCAML).
http://www.ffconsultancy.com/products/fsharp_for_visualization/index.html
If numpy can be used from F#, this would be interesting. But is this at all possible ? For example, are the semantic compatible (indexing, etc...) ? I have no idea.
Implementation-wise, this is an interesting problem to solve.
cheers,
David
participants (12)
-
Albert Strasheim -
Charles R Harris -
Christopher Barker -
David -
David Cournapeau -
dmitrey -
Giles Thomas -
Michael Foord -
Scott Ransom -
Steve Lianoglou -
Timothy Hochberg -
Travis E. Oliphant