Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching?
Hello folks, Myself a long time Numpy user, and recently done some Haskell stuff, Please have a look at https://github.com/e-wrks/edh/tree/master/Tour#defining-more-magic-methods <https://github.com/e-wrks/edh/tree/master/Tour#defining-more-magic-methods> and https://github.com/e-wrks/edh/tree/master/Tour#indexing <https://github.com/e-wrks/edh/tree/master/Tour#indexing> , any interest to port Numpy to Haskell, with http://hackage.haskell.org/package/accelerate <http://hackage.haskell.org/package/accelerate> and/or http://hackage.haskell.org/package/repa <http://hackage.haskell.org/package/repa> doing the number crunching? Best regards, Compl
Hi Matthew, I hate to say I'm still a noob in the implementation details of Numpy, I just got an interpreter for sth else working, and suddenly realized it can make seemingly better magic methods in Python spirit. Repa and Accelerate per my learning so far, are focused on loop-fusion, i.e. optimization after all intended computations have been expressed and composed together, this seems some different from Numpy/Pandas tradition to implement doable things ex ante, and let the users do things interactively ad-hoc. That said, so I feel Repa will do composed computations more speedy, once properly organized. And with Accelerate, since it does GPU natively, I do feel a sure thing for it to excel CPU implementations. btw I'm not a native English speaker, please don't hesitate to ask me to make myself clearer in my wordings, I'm aware of my flaws in English skills. btw I'm a believer that Brain-Computer-Interface ought to happen sooner, especially non-intrusive ones, I can't wait to program such interfaces for great good and fun. I see you are pioneering in this area, take my admiring, and I started following you on github, looking forward to hear from you more. btw I'm behind China's Great Fire Wall, access to some western sites are banned badly at times, that will explain why I can be unresponsive some times. Cheers, Compl On 2020/1/21 下午10:20, Matthew Brett wrote:
The description you make tell me that this look similar to numexpr[1]. It could be useful that you look it up. [1] https://github.com/pydata/numexpr -----Original Message----- From: NumPy-Discussion <numpy-discussion-bounces+fbastien=nvidia.com@python.org> On Behalf Of Compl Yue Sent: Tuesday, January 21, 2020 10:33 AM To: numpy-discussion@python.org Subject: Re: [Numpy-discussion] Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching? External email: Use caution opening links or attachments Hi Matthew, I hate to say I'm still a noob in the implementation details of Numpy, I just got an interpreter for sth else working, and suddenly realized it can make seemingly better magic methods in Python spirit. Repa and Accelerate per my learning so far, are focused on loop-fusion, i.e. optimization after all intended computations have been expressed and composed together, this seems some different from Numpy/Pandas tradition to implement doable things ex ante, and let the users do things interactively ad-hoc. That said, so I feel Repa will do composed computations more speedy, once properly organized. And with Accelerate, since it does GPU natively, I do feel a sure thing for it to excel CPU implementations. btw I'm not a native English speaker, please don't hesitate to ask me to make myself clearer in my wordings, I'm aware of my flaws in English skills. btw I'm a believer that Brain-Computer-Interface ought to happen sooner, especially non-intrusive ones, I can't wait to program such interfaces for great good and fun. I see you are pioneering in this area, take my admiring, and I started following you on github, looking forward to hear from you more. btw I'm behind China's Great Fire Wall, access to some western sites are banned badly at times, that will explain why I can be unresponsive some times. Cheers, Compl On 2020/1/21 下午10:20, Matthew Brett wrote:
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion ----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------
Yes, I do feel NumExpr does very similar things as Repa, while Repa works rather statically for Haskell code, NumExpr is much more interactive. Well, honestly as a long time Numpy user, I'm aware of NumExpr since early days I learning Numpy, but never ever resorted to NumExpr, it's neither as complete as Numpy, nor as flexible as Numba. I'm neither proficient in Haskell at data exploration for now, attempting improving the tooling (e.g. [hadui](https://github.com/complyue/hadui <https://github.com/complyue/hadui>) ), while still working on an ndarray interface for Haskell to an in house array database (mmap'ed virtual large datafile per se). I miss very much the workflow with Pandas/Numpy/Numba, the interactive aspect is crucial to me and my team's daily work. We have to resort to Haskell because some time ago we needed to start processing irregular event streams (no common dimension along time axis), for thousands of series (instruments in the stock market). While in handling this many number of inputs (passing in a tuple of 1d arrays, Numba treats each array as a separate input variable), Numba will swallow 20+GB ram with 10+ hours in jit compilation without a result. I'm aware of Julia, and feels very pity for my org not able to adopt it for hiring difficulties, while luckily Haskell is agreed to be the base of our next gen internal platform. So I'm currently working hard to bring sufficient interaction abilities with a Haskell tool chain.
Hi Matthew, I hate to say I'm still a noob in the implementation details of Numpy, I just got an interpreter for sth else working, and suddenly realized it can make seemingly better magic methods in Python spirit. Repa and Accelerate per my learning so far, are focused on loop-fusion, i.e. optimization after all intended computations have been expressed and composed together, this seems some different from Numpy/Pandas tradition to implement doable things ex ante, and let the users do things interactively ad-hoc. That said, so I feel Repa will do composed computations more speedy, once properly organized. And with Accelerate, since it does GPU natively, I do feel a sure thing for it to excel CPU implementations. btw I'm not a native English speaker, please don't hesitate to ask me to make myself clearer in my wordings, I'm aware of my flaws in English skills. btw I'm a believer that Brain-Computer-Interface ought to happen sooner, especially non-intrusive ones, I can't wait to program such interfaces for great good and fun. I see you are pioneering in this area, take my admiring, and I started following you on github, looking forward to hear from you more. btw I'm behind China's Great Fire Wall, access to some western sites are banned badly at times, that will explain why I can be unresponsive some times. Cheers, Compl On 2020/1/21 下午10:20, Matthew Brett wrote:
The description you make tell me that this look similar to numexpr[1]. It could be useful that you look it up. [1] https://github.com/pydata/numexpr -----Original Message----- From: NumPy-Discussion <numpy-discussion-bounces+fbastien=nvidia.com@python.org> On Behalf Of Compl Yue Sent: Tuesday, January 21, 2020 10:33 AM To: numpy-discussion@python.org Subject: Re: [Numpy-discussion] Interest to port Numpy to Haskell, with Accelerate/Repa to do the crunching? External email: Use caution opening links or attachments Hi Matthew, I hate to say I'm still a noob in the implementation details of Numpy, I just got an interpreter for sth else working, and suddenly realized it can make seemingly better magic methods in Python spirit. Repa and Accelerate per my learning so far, are focused on loop-fusion, i.e. optimization after all intended computations have been expressed and composed together, this seems some different from Numpy/Pandas tradition to implement doable things ex ante, and let the users do things interactively ad-hoc. That said, so I feel Repa will do composed computations more speedy, once properly organized. And with Accelerate, since it does GPU natively, I do feel a sure thing for it to excel CPU implementations. btw I'm not a native English speaker, please don't hesitate to ask me to make myself clearer in my wordings, I'm aware of my flaws in English skills. btw I'm a believer that Brain-Computer-Interface ought to happen sooner, especially non-intrusive ones, I can't wait to program such interfaces for great good and fun. I see you are pioneering in this area, take my admiring, and I started following you on github, looking forward to hear from you more. btw I'm behind China's Great Fire Wall, access to some western sites are banned badly at times, that will explain why I can be unresponsive some times. Cheers, Compl On 2020/1/21 下午10:20, Matthew Brett wrote:
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion ----------------------------------------------------------------------------------- This email message is for the sole use of the intended recipient(s) and may contain confidential information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message. -----------------------------------------------------------------------------------
Yes, I do feel NumExpr does very similar things as Repa, while Repa works rather statically for Haskell code, NumExpr is much more interactive. Well, honestly as a long time Numpy user, I'm aware of NumExpr since early days I learning Numpy, but never ever resorted to NumExpr, it's neither as complete as Numpy, nor as flexible as Numba. I'm neither proficient in Haskell at data exploration for now, attempting improving the tooling (e.g. [hadui](https://github.com/complyue/hadui <https://github.com/complyue/hadui>) ), while still working on an ndarray interface for Haskell to an in house array database (mmap'ed virtual large datafile per se). I miss very much the workflow with Pandas/Numpy/Numba, the interactive aspect is crucial to me and my team's daily work. We have to resort to Haskell because some time ago we needed to start processing irregular event streams (no common dimension along time axis), for thousands of series (instruments in the stock market). While in handling this many number of inputs (passing in a tuple of 1d arrays, Numba treats each array as a separate input variable), Numba will swallow 20+GB ram with 10+ hours in jit compilation without a result. I'm aware of Julia, and feels very pity for my org not able to adopt it for hiring difficulties, while luckily Haskell is agreed to be the base of our next gen internal platform. So I'm currently working hard to bring sufficient interaction abilities with a Haskell tool chain.
participants (4)
-
Compl Yue -
Frederic Bastien -
Matthew Brett -
YueCompl