introducing Numpy.net, a pure C# implementation of Numpy
This is a quick note to announce open source numpy.net available at this repository https://github.com/Quansight-Labs/numpy.net This is a pure C# based implementation of Numpy, ported from the underlying C library and the python application layer. Written and tested on .NET standard 2.1 so it will run anywhere. I have the vast majority of the most commonly used functions implemented and unit tested. It supports fully functional slicing and views. Verified to generate binary compatible output with python numpy. I built it because I was tasked with porting a large Python Numpy application to C# and as you may know, there are not a lot of good solutions (until now). My employer graciously agreed to open source it. If you have some time to review it, I'd love to get some feedback from some Numpy experts (I don't consider myself one). I believe it is ready to start building tools like SciPy on top of it. Longer term, I would like to see this ported to Java. I expect pure C# to port easily to Java.
From: NumPy-Discussion <numpy-discussion-bounces+kmckenna=baselinesw.com@python.org> On Behalf Of Charles R Harris Sent: Saturday, March 16, 2019 12:06 PM To: Discussion of Numerical Python <numpy-discussion@python.org> Subject: [SPAM]Re: [Numpy-discussion] introducing Numpy.net, a pure C# implementation of Numpy On Sat, Mar 16, 2019 at 10:02 AM Charles R Harris <charlesr.harris@gmail.com <mailto:charlesr.harris@gmail.com> > wrote: On Sat, Mar 16, 2019 at 4:41 AM <kmckenna@baselinesw.com <mailto:kmckenna@baselinesw.com> > wrote: This is a quick note to announce open source numpy.net <http://numpy.net> available at this repository https://github.com/Quansight-Labs/numpy.net This is a pure C# based implementation of Numpy, ported from the underlying C library and the python application layer. Written and tested on .NET standard 2.1 so it will run anywhere. I have the vast majority of the most commonly used functions implemented and unit tested. It supports fully functional slicing and views. Verified to generate binary compatible output with python numpy. I built it because I was tasked with porting a large Python Numpy application to C# and as you may know, there are not a lot of good solutions (until now). My employer graciously agreed to open source it. If you have some time to review it, I'd love to get some feedback from some Numpy experts (I don't consider myself one). I believe it is ready to start building tools like SciPy on top of it. Longer term, I would like to see this ported to Java. I expect pure C# to port easily to Java. That's interesting. What parts are in C#? Any benchmarks comparing the two implementations? What about BLAS libraries? 1. All of it is in C#. The underlying C code was ported and so was the python layer. 2. Benchmarks? If you mean performance benchmarks, I have nothing official but I will say that the C# implementation is not as fast as the optimized C code and it never will be. The only place you would really notice it is if you were doing a UFUNC operation on a large array. 3. I am not a numpy programmer so I don’t know what BLAS libraries are. I just did a quick google and my guess is that this is not relevant in my implementation. I envision my tool to be used if someone wants to port a numpy application to .NET or wants to do Numpy things in a .NET application, Kevin
On Sat, Mar 16, 2019 at 11:10 AM <kmckenna@baselinesw.com> wrote:
Just reply to the discussion, I may have made a boo boo in replying to your first post. I'm curious. Implementing NumPy in another language seems like quite a bit of work. Did you have any tools to make it easier? I assume the C api is gone, so that the translation is NumPy program specific. I've never used C#, was there a reason to avoid C? What about IronPython? IIRC there was some work to make NumPy run on IronPython before that project was ended. I'm also curious what the application was that made it impossible to stay with python, I suppose the customer wanted C#, but I'd like to know why plain old Python was not an option. Chuck
Just reply to the discussion, I may have made a boo boo in replying to your
Chuck, To start: I'd like to be clear that I'm not trying to speak for the OP, but I thought I'd share my experiences. I'm a civil engineer who adopted Python early in his career and became the "data guy" in the office pretty early on. Our company's IT department manages lots of Windows Servers running SQL Server. In my case, running python apps on our infrastructure just isn't feasible or supported by the IT department. Typically, we move to outside hosting when we're going that route. However, sometimes there are a litany of reasons to stay with our in house infrastructure. In that case, C# makes it very simple to set up and deploy a web API against a SQL Server database. It's very much a walled-garden approach to the web, but it can be quite efficient, especially when you're billing clients by the hour. Additionally, C#'s ORM treats each table in the database pretty much as dataframe. I've ported many of my pandas-based workflows over to C# without much issue. For example in C# I did: var results = StormEvent .Where(se => new List<String> { "CI", "JI" }.Contains(se.MonitoringLocations .sitename)) .Where(se => se.pCoseParameter.pName.ToLower().Contains("flow")) .Where(se => se.wateryear_int.Equals(2014)) .GroupBy( se => new {se.MonitoringLocations.sitename, se.paramName}, (key, df) => new { ml = key.sitename, param = key.paramName total = df.Select(d => d.paramValue).Sum() } ); Whereas in pandas I would do: results = ( StormEvents .merge(MonitoringLocations, on='monlocID', lsuffix='_ml', rsuffix='') .merge(Parameters, on='paramID', lsuffix='_param', rsuffix='') .loc[lambda x: x['sitename'].isin(['CI', 'CI'])] .loc[lambda x: x['param'].str.lower().str.contains("flow")] .loc[lambda x: x['wateryear_int'] == 2014] .groupby(by=['sitename', 'paramID']) .sum()['paramValue'] ) I'm not trying to sell you on C#, I'll always go python if I can. But depending on your organization's infrastructure and project constraints, it can be surprisingly pleasant to work with. The point of all of this is that in those situations, have a numpy-like library would be very nice indeed. I've very excited to hear that the OP's work has been open sourced. Kudos! -Paul
On Mon, Mar 18, 2019 at 1:19 PM Paul Hobson <pmhobson@gmail.com> wrote:
Just curious -- does it have to be C#? or could it be any CLR application -- i.e. IronPython? I imagine you could build a web service pretty easily in IronPython -- though AFAIK, the attempts at getting numpy support (and thus Pandas, etc) never panned out. The point of all of this is that in those situations, have a numpy-like
library would be very nice indeed. I've very excited to hear that the OP's work has been open sourced.
I wonder if the OP's work could be used to make a numpy for Iron Python native to the CLR .... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Mon, Mar 18, 2019 at 1:19 PM Paul Hobson <pmhobson@gmail.com> wrote:
On Wed, Mar 27, 2019 at 10:30 AM Chris Barker <chris.barker@noaa.gov> wrote:
Just curious -- does it have to be C#? or could it be any CLR application -- i.e. IronPython?
I don't do this enough to know this answer to this question. But we had a critical mass of past C# projects to serve as a guide that it's been a relatively low effort to get the C# APIs up and running (e.g., click this button to deploy).
That was the state of things (no numpy or pandas) last time I checked. And C#'s pandas-like approach to DB tables is good enough that I haven't felt the need check again recently. The point of all of this is that in those situations, have a numpy-like
That would be very interesting indeed. -Paul
Thanks for sharing this work with the world, What you did is not an easy task. This makes it easier for people to port NumPy-based Python code to C#. Of course, this group would more likely prefer people to port their C# code to Python + NumPy. Nonetheless, I think there are interesting things to learn from your code for this group as well --- if you could share some of the details of what it was like to port the code. Things like where the difficulties where, any abstraction concepts you developed, as well as things that were confusing, it would be of value to this community. Also, a lot of NumPy code these days uses a lot of other libraries. Did you also port the numpy.linalg or numpy.fft or numpy.stats modules? The numpy.linalg module currently uses low-level libraries in C called BLAS (Basic Linear Algebra Subprograms). There are similar interfaces in C#. Another approach to code-sharing is the XND project (http://xnd.io) which allows the creation of NumPy-like interfaces in any language. Your implementation would be an interesting start to a C# binding of XND. Thanks, -Travis On Sat, Mar 16, 2019 at 5:41 AM <kmckenna@baselinesw.com> wrote:
From: NumPy-Discussion <numpy-discussion-bounces+kmckenna=baselinesw.com@python.org> On Behalf Of Charles R Harris Sent: Saturday, March 16, 2019 12:06 PM To: Discussion of Numerical Python <numpy-discussion@python.org> Subject: [SPAM]Re: [Numpy-discussion] introducing Numpy.net, a pure C# implementation of Numpy On Sat, Mar 16, 2019 at 10:02 AM Charles R Harris <charlesr.harris@gmail.com <mailto:charlesr.harris@gmail.com> > wrote: On Sat, Mar 16, 2019 at 4:41 AM <kmckenna@baselinesw.com <mailto:kmckenna@baselinesw.com> > wrote: This is a quick note to announce open source numpy.net <http://numpy.net> available at this repository https://github.com/Quansight-Labs/numpy.net This is a pure C# based implementation of Numpy, ported from the underlying C library and the python application layer. Written and tested on .NET standard 2.1 so it will run anywhere. I have the vast majority of the most commonly used functions implemented and unit tested. It supports fully functional slicing and views. Verified to generate binary compatible output with python numpy. I built it because I was tasked with porting a large Python Numpy application to C# and as you may know, there are not a lot of good solutions (until now). My employer graciously agreed to open source it. If you have some time to review it, I'd love to get some feedback from some Numpy experts (I don't consider myself one). I believe it is ready to start building tools like SciPy on top of it. Longer term, I would like to see this ported to Java. I expect pure C# to port easily to Java. That's interesting. What parts are in C#? Any benchmarks comparing the two implementations? What about BLAS libraries? 1. All of it is in C#. The underlying C code was ported and so was the python layer. 2. Benchmarks? If you mean performance benchmarks, I have nothing official but I will say that the C# implementation is not as fast as the optimized C code and it never will be. The only place you would really notice it is if you were doing a UFUNC operation on a large array. 3. I am not a numpy programmer so I don’t know what BLAS libraries are. I just did a quick google and my guess is that this is not relevant in my implementation. I envision my tool to be used if someone wants to port a numpy application to .NET or wants to do Numpy things in a .NET application, Kevin
On Sat, Mar 16, 2019 at 11:10 AM <kmckenna@baselinesw.com> wrote:
Just reply to the discussion, I may have made a boo boo in replying to your first post. I'm curious. Implementing NumPy in another language seems like quite a bit of work. Did you have any tools to make it easier? I assume the C api is gone, so that the translation is NumPy program specific. I've never used C#, was there a reason to avoid C? What about IronPython? IIRC there was some work to make NumPy run on IronPython before that project was ended. I'm also curious what the application was that made it impossible to stay with python, I suppose the customer wanted C#, but I'd like to know why plain old Python was not an option. Chuck
Just reply to the discussion, I may have made a boo boo in replying to your
Chuck, To start: I'd like to be clear that I'm not trying to speak for the OP, but I thought I'd share my experiences. I'm a civil engineer who adopted Python early in his career and became the "data guy" in the office pretty early on. Our company's IT department manages lots of Windows Servers running SQL Server. In my case, running python apps on our infrastructure just isn't feasible or supported by the IT department. Typically, we move to outside hosting when we're going that route. However, sometimes there are a litany of reasons to stay with our in house infrastructure. In that case, C# makes it very simple to set up and deploy a web API against a SQL Server database. It's very much a walled-garden approach to the web, but it can be quite efficient, especially when you're billing clients by the hour. Additionally, C#'s ORM treats each table in the database pretty much as dataframe. I've ported many of my pandas-based workflows over to C# without much issue. For example in C# I did: var results = StormEvent .Where(se => new List<String> { "CI", "JI" }.Contains(se.MonitoringLocations .sitename)) .Where(se => se.pCoseParameter.pName.ToLower().Contains("flow")) .Where(se => se.wateryear_int.Equals(2014)) .GroupBy( se => new {se.MonitoringLocations.sitename, se.paramName}, (key, df) => new { ml = key.sitename, param = key.paramName total = df.Select(d => d.paramValue).Sum() } ); Whereas in pandas I would do: results = ( StormEvents .merge(MonitoringLocations, on='monlocID', lsuffix='_ml', rsuffix='') .merge(Parameters, on='paramID', lsuffix='_param', rsuffix='') .loc[lambda x: x['sitename'].isin(['CI', 'CI'])] .loc[lambda x: x['param'].str.lower().str.contains("flow")] .loc[lambda x: x['wateryear_int'] == 2014] .groupby(by=['sitename', 'paramID']) .sum()['paramValue'] ) I'm not trying to sell you on C#, I'll always go python if I can. But depending on your organization's infrastructure and project constraints, it can be surprisingly pleasant to work with. The point of all of this is that in those situations, have a numpy-like library would be very nice indeed. I've very excited to hear that the OP's work has been open sourced. Kudos! -Paul
On Mon, Mar 18, 2019 at 1:19 PM Paul Hobson <pmhobson@gmail.com> wrote:
Just curious -- does it have to be C#? or could it be any CLR application -- i.e. IronPython? I imagine you could build a web service pretty easily in IronPython -- though AFAIK, the attempts at getting numpy support (and thus Pandas, etc) never panned out. The point of all of this is that in those situations, have a numpy-like
library would be very nice indeed. I've very excited to hear that the OP's work has been open sourced.
I wonder if the OP's work could be used to make a numpy for Iron Python native to the CLR .... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
On Mon, Mar 18, 2019 at 1:19 PM Paul Hobson <pmhobson@gmail.com> wrote:
On Wed, Mar 27, 2019 at 10:30 AM Chris Barker <chris.barker@noaa.gov> wrote:
Just curious -- does it have to be C#? or could it be any CLR application -- i.e. IronPython?
I don't do this enough to know this answer to this question. But we had a critical mass of past C# projects to serve as a guide that it's been a relatively low effort to get the C# APIs up and running (e.g., click this button to deploy).
That was the state of things (no numpy or pandas) last time I checked. And C#'s pandas-like approach to DB tables is good enough that I haven't felt the need check again recently. The point of all of this is that in those situations, have a numpy-like
That would be very interesting indeed. -Paul
Thanks for sharing this work with the world, What you did is not an easy task. This makes it easier for people to port NumPy-based Python code to C#. Of course, this group would more likely prefer people to port their C# code to Python + NumPy. Nonetheless, I think there are interesting things to learn from your code for this group as well --- if you could share some of the details of what it was like to port the code. Things like where the difficulties where, any abstraction concepts you developed, as well as things that were confusing, it would be of value to this community. Also, a lot of NumPy code these days uses a lot of other libraries. Did you also port the numpy.linalg or numpy.fft or numpy.stats modules? The numpy.linalg module currently uses low-level libraries in C called BLAS (Basic Linear Algebra Subprograms). There are similar interfaces in C#. Another approach to code-sharing is the XND project (http://xnd.io) which allows the creation of NumPy-like interfaces in any language. Your implementation would be an interesting start to a C# binding of XND. Thanks, -Travis On Sat, Mar 16, 2019 at 5:41 AM <kmckenna@baselinesw.com> wrote:
participants (5)
-
Charles R Harris -
Chris Barker -
kmckenna＠baselinesw.com -
Paul Hobson -
Travis Oliphant