PyRSB: Python interface to librsb sparse matrices library
![](https://secure.gravatar.com/avatar/a957fa9e46f46af9f5cfec44b170cf1d.jpg?s=120&d=mm&r=g)
Hi. I'm the author of the high performance multithreaded sparse matrix library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/ I'm *not* a user of SciPy/NumPy/Python, but using Cython I have written a proof-of-concept interface to librsb, named `PyRSB': https://github.com/michelemartone/pyrsb PyRSB is in a prototypal state; e.g. still lacks good error handling. Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'. Advantages over csr_matrix are in fast multithreaded multiplication of huge sparse matrices. Intended application area is iterative solution of linear systems; particularly fast if with symmetric matrices and many rhs. With this email I am looking for prospective: - users/testers - developers (any interest to collaborate/adopt/include the project?) Looking forward for your feedback, Michele
![](https://secure.gravatar.com/avatar/971399cd3d255dfffb7f1924f4595c05.jpg?s=120&d=mm&r=g)
Hi Michele, This is really interesting. I am a co-author of the xtensor project and one thing that could be interesting is to wrap the various sparse matrix data structures in the form of xtensor expressions. A byproduct of doing so is that it would simplify creating bindings for multiple scientific computing languages (Python, Julia, R, and more coming). You can see the blog post http://quantstack.net/c++/2017/05/30/polyglot-scientific-computing-with- xtensor.html for reference... Also, one quick question: is the LGPL license a deliberate choice or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency. If you are the only copyright holder, you would still have the possibility to license it under a more permissive license such as BSD or MIT... Congratulations on the release! Sylvain On Wed, Jun 21, 2017 at 11:51 AM, Michele Martone <michelemartone@users. sourceforge.net> wrote:
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Jun 24, 2017 7:29 AM, "Sylvain Corlay" <sylvain.corlay@gmail.com> wrote: Also, one quick question: is the LGPL license a deliberate choice or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency. If you are the only copyright holder, you would still have the possibility to license it under a more permissive license such as BSD or MIT... Why would LGPL be a problem in a dependency? That doesn't stop you making your code BSD, and it's less restrictive license-wise than depending on MKL or the windows C runtime... -n
![](https://secure.gravatar.com/avatar/ad13088a623822caf74e635a68a55eae.jpg?s=120&d=mm&r=g)
On Sat, Jun 24, 2017 at 3:16 PM, Nathaniel Smith <njs@pobox.com> wrote:
Is scipy still including any LGPL code, I thought not. There might still be some optional dependencies that not many users are using by default. ? Julia packages are mostly MIT, AFAIK. (except for the GPL parts because of cholmod, which we (?) avoid) Josef
![](https://secure.gravatar.com/avatar/b4f6d4f8b501cb05fd054944a166a121.jpg?s=120&d=mm&r=g)
On Sat, 2017-06-24 at 15:47 -0400, josef.pktd@gmail.com wrote:
Well, I don't think scipy has many dependencies (but I would not be surprised if those are LGPL). Not a specialist, but as a dependency it should be fine (that is the point of the L in LGPL after all as far as I understand, it is much less viral). If you package it with your own stuff, you have to make sure to point out that parts are LGPL of course (just like there is a reason you get the GPL printed out with some devices) and if you modify it provide these modifications, etc. Of course you cannot include it into the scipy codebase itself, but there is probably no aim of doing so here, so without a specific reason I would think that LGPL is a great license. - Sebastian
![](https://secure.gravatar.com/avatar/29a8424a5c1ddc5e0e79104965a85011.jpg?s=120&d=mm&r=g)
Does this still apply: https://scipy.github.io/old-wiki/pages/License_Compatibility.html Carl 2017-06-24 22:07 GMT+02:00 Sebastian Berg <sebastian@sipsolutions.net>:
![](https://secure.gravatar.com/avatar/b4f6d4f8b501cb05fd054944a166a121.jpg?s=120&d=mm&r=g)
On Sat, 2017-06-24 at 22:58 +0200, Carl Kleffner wrote:
Does this still apply: https://scipy.github.io/old-wiki/pages/License _Compatibility.html
Of course, but it talks about putting it into the code base of scipy not about being able to use the package in any way in a dependency (i.e. `import package`). - Sebastian
![](https://secure.gravatar.com/avatar/ad13088a623822caf74e635a68a55eae.jpg?s=120&d=mm&r=g)
On Sat, Jun 24, 2017 at 5:26 PM, Sebastian Berg <sebastian@sipsolutions.net> wrote:
But scipy does bundle a lot of external packages and then the license is relevant. I have no idea if this would apply here, but I find Sylvain's question relevant if closer integration and usage within the scientific python, and maybe Julia, community is desired. LGPL is not as bad as GPL but still adds another hurdle, as far as I know the scipy and related history. Josef
![](https://secure.gravatar.com/avatar/a957fa9e46f46af9f5cfec44b170cf1d.jpg?s=120&d=mm&r=g)
On 20170624@16:29, Sylvain Corlay wrote:
Hi Michele,
Hi Sylvain, print(a+a) # + operator and 'print' interfacing ?
Congratulations on the release! Thanks for the interest and welcome constructive feedback :-)
![](https://secure.gravatar.com/avatar/971399cd3d255dfffb7f1924f4595c05.jpg?s=120&d=mm&r=g)
Hi Michele, Regarding xtensor, the article focused on how we can wrap an existing data structure in an "xtensor expression", i.e. something that you can operate upon with the numpy style API in C++ (see the numpy to xtensor cheat sheet http://xtensor.readthedocs.io/en/latest/numpy.html). Since it is an expression system, you can do "a + b * sin(t)" with numpy-style broadcasting and it returns an un-evaluated expression, which is computed only upon access or assignment to a container. "a", "b", and "c" could all be compound expressions, backed by filesystem operations, be an in-memory container, or a wrapper on your sparse matrices... Now, we are working on automating the "*C++ tensor expression-type -> python wrapper with access and math operators" *generation of code, which would be what you are describing below... On the subject of the LGPL, regarding the distinction with the GPL that were brought up earlier in the thread, it is indeed not as problematic as the GPL, but still less permissive as BSD and MIT. In general, I think that using a BSD or MIT license averts these concerns in the Python ecosystem, and is preferable *if you don't care about copyleft*... Best, Sylvain On Sun, Jun 25, 2017 at 12:01 PM, Michele Martone < michelemartone@users.sourceforge.net> wrote:
![](https://secure.gravatar.com/avatar/a957fa9e46f46af9f5cfec44b170cf1d.jpg?s=120&d=mm&r=g)
On 20170624@16:29, Sylvain Corlay wrote:
Hi Michele,
Hi Sylvain, print(a+a) # + operator and 'print' interfacing ?
Congratulations on the release! Thanks for the interest and welcome constructive feedback :-)
![](https://secure.gravatar.com/avatar/971399cd3d255dfffb7f1924f4595c05.jpg?s=120&d=mm&r=g)
Hi Michele, This is really interesting. I am a co-author of the xtensor project and one thing that could be interesting is to wrap the various sparse matrix data structures in the form of xtensor expressions. A byproduct of doing so is that it would simplify creating bindings for multiple scientific computing languages (Python, Julia, R, and more coming). You can see the blog post http://quantstack.net/c++/2017/05/30/polyglot-scientific-computing-with- xtensor.html for reference... Also, one quick question: is the LGPL license a deliberate choice or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency. If you are the only copyright holder, you would still have the possibility to license it under a more permissive license such as BSD or MIT... Congratulations on the release! Sylvain On Wed, Jun 21, 2017 at 11:51 AM, Michele Martone <michelemartone@users. sourceforge.net> wrote:
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Jun 24, 2017 7:29 AM, "Sylvain Corlay" <sylvain.corlay@gmail.com> wrote: Also, one quick question: is the LGPL license a deliberate choice or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency. If you are the only copyright holder, you would still have the possibility to license it under a more permissive license such as BSD or MIT... Why would LGPL be a problem in a dependency? That doesn't stop you making your code BSD, and it's less restrictive license-wise than depending on MKL or the windows C runtime... -n
![](https://secure.gravatar.com/avatar/ad13088a623822caf74e635a68a55eae.jpg?s=120&d=mm&r=g)
On Sat, Jun 24, 2017 at 3:16 PM, Nathaniel Smith <njs@pobox.com> wrote:
Is scipy still including any LGPL code, I thought not. There might still be some optional dependencies that not many users are using by default. ? Julia packages are mostly MIT, AFAIK. (except for the GPL parts because of cholmod, which we (?) avoid) Josef
![](https://secure.gravatar.com/avatar/b4f6d4f8b501cb05fd054944a166a121.jpg?s=120&d=mm&r=g)
On Sat, 2017-06-24 at 15:47 -0400, josef.pktd@gmail.com wrote:
Well, I don't think scipy has many dependencies (but I would not be surprised if those are LGPL). Not a specialist, but as a dependency it should be fine (that is the point of the L in LGPL after all as far as I understand, it is much less viral). If you package it with your own stuff, you have to make sure to point out that parts are LGPL of course (just like there is a reason you get the GPL printed out with some devices) and if you modify it provide these modifications, etc. Of course you cannot include it into the scipy codebase itself, but there is probably no aim of doing so here, so without a specific reason I would think that LGPL is a great license. - Sebastian
![](https://secure.gravatar.com/avatar/29a8424a5c1ddc5e0e79104965a85011.jpg?s=120&d=mm&r=g)
Does this still apply: https://scipy.github.io/old-wiki/pages/License_Compatibility.html Carl 2017-06-24 22:07 GMT+02:00 Sebastian Berg <sebastian@sipsolutions.net>:
![](https://secure.gravatar.com/avatar/b4f6d4f8b501cb05fd054944a166a121.jpg?s=120&d=mm&r=g)
On Sat, 2017-06-24 at 22:58 +0200, Carl Kleffner wrote:
Does this still apply: https://scipy.github.io/old-wiki/pages/License _Compatibility.html
Of course, but it talks about putting it into the code base of scipy not about being able to use the package in any way in a dependency (i.e. `import package`). - Sebastian
![](https://secure.gravatar.com/avatar/ad13088a623822caf74e635a68a55eae.jpg?s=120&d=mm&r=g)
On Sat, Jun 24, 2017 at 5:26 PM, Sebastian Berg <sebastian@sipsolutions.net> wrote:
But scipy does bundle a lot of external packages and then the license is relevant. I have no idea if this would apply here, but I find Sylvain's question relevant if closer integration and usage within the scientific python, and maybe Julia, community is desired. LGPL is not as bad as GPL but still adds another hurdle, as far as I know the scipy and related history. Josef
![](https://secure.gravatar.com/avatar/a957fa9e46f46af9f5cfec44b170cf1d.jpg?s=120&d=mm&r=g)
On 20170624@16:29, Sylvain Corlay wrote:
Hi Michele,
Hi Sylvain, print(a+a) # + operator and 'print' interfacing ?
Congratulations on the release! Thanks for the interest and welcome constructive feedback :-)
![](https://secure.gravatar.com/avatar/971399cd3d255dfffb7f1924f4595c05.jpg?s=120&d=mm&r=g)
Hi Michele, Regarding xtensor, the article focused on how we can wrap an existing data structure in an "xtensor expression", i.e. something that you can operate upon with the numpy style API in C++ (see the numpy to xtensor cheat sheet http://xtensor.readthedocs.io/en/latest/numpy.html). Since it is an expression system, you can do "a + b * sin(t)" with numpy-style broadcasting and it returns an un-evaluated expression, which is computed only upon access or assignment to a container. "a", "b", and "c" could all be compound expressions, backed by filesystem operations, be an in-memory container, or a wrapper on your sparse matrices... Now, we are working on automating the "*C++ tensor expression-type -> python wrapper with access and math operators" *generation of code, which would be what you are describing below... On the subject of the LGPL, regarding the distinction with the GPL that were brought up earlier in the thread, it is indeed not as problematic as the GPL, but still less permissive as BSD and MIT. In general, I think that using a BSD or MIT license averts these concerns in the Python ecosystem, and is preferable *if you don't care about copyleft*... Best, Sylvain On Sun, Jun 25, 2017 at 12:01 PM, Michele Martone < michelemartone@users.sourceforge.net> wrote:
![](https://secure.gravatar.com/avatar/a957fa9e46f46af9f5cfec44b170cf1d.jpg?s=120&d=mm&r=g)
On 20170624@16:29, Sylvain Corlay wrote:
Hi Michele,
Hi Sylvain, print(a+a) # + operator and 'print' interfacing ?
Congratulations on the release! Thanks for the interest and welcome constructive feedback :-)
participants (6)
-
Carl Kleffner
-
josef.pktd@gmail.com
-
Michele Martone
-
Nathaniel Smith
-
Sebastian Berg
-
Sylvain Corlay