PyRSB: Python interface to librsb sparse matrices library

Hi.
I'm the author of the high performance multithreaded sparse matrix library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/
I'm *not* a user of SciPy/NumPy/Python, but using Cython I have written a proof-of-concept interface to librsb, named `PyRSB': https://github.com/michelemartone/pyrsb
PyRSB is in a prototypal state; e.g. still lacks good error handling. Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'. Advantages over csr_matrix are in fast multithreaded multiplication of huge sparse matrices. Intended application area is iterative solution of linear systems; particularly fast if with symmetric matrices and many rhs.
With this email I am looking for prospective: - users/testers - developers (any interest to collaborate/adopt/include the project?)
Looking forward for your feedback, Michele

Hi Michele,
This is really interesting. I am a co-author of the xtensor project and one thing that could be interesting is to wrap the various sparse matrix data structures in the form of xtensor expressions. A byproduct of doing so is that it would simplify creating bindings for multiple scientific computing languages (Python, Julia, R, and more coming). You can see the blog post http://quantstack.net/c++/2017/05/30/polyglot-scientific-computing-with- xtensor.html for reference...
Also, one quick question: is the LGPL license a deliberate choice or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency. If you are the only copyright holder, you would still have the possibility to license it under a more permissive license such as BSD or MIT...
Congratulations on the release!
Sylvain
On Wed, Jun 21, 2017 at 11:51 AM, Michele Martone <michelemartone@users. sourceforge.net> wrote:
Hi.
I'm the author of the high performance multithreaded sparse matrix library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/
I'm *not* a user of SciPy/NumPy/Python, but using Cython I have written a proof-of-concept interface to librsb, named `PyRSB': https://github.com/michelemartone/pyrsb
PyRSB is in a prototypal state; e.g. still lacks good error handling. Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'. Advantages over csr_matrix are in fast multithreaded multiplication of huge sparse matrices. Intended application area is iterative solution of linear systems; particularly fast if with symmetric matrices and many rhs.
With this email I am looking for prospective:
- users/testers
- developers (any interest to collaborate/adopt/include the project?)
Looking forward for your feedback, Michele
SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev

On Jun 24, 2017 7:29 AM, "Sylvain Corlay" sylvain.corlay@gmail.com wrote:
Also, one quick question: is the LGPL license a deliberate choice or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency. If you are the only copyright holder, you would still have the possibility to license it under a more permissive license such as BSD or MIT...
Why would LGPL be a problem in a dependency? That doesn't stop you making your code BSD, and it's less restrictive license-wise than depending on MKL or the windows C runtime...
-n

On Sat, Jun 24, 2017 at 3:16 PM, Nathaniel Smith njs@pobox.com wrote:
On Jun 24, 2017 7:29 AM, "Sylvain Corlay" sylvain.corlay@gmail.com wrote:
Also, one quick question: is the LGPL license a deliberate choice or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency. If you are the only copyright holder, you would still have the possibility to license it under a more permissive license such as BSD or MIT...
Why would LGPL be a problem in a dependency? That doesn't stop you making your code BSD, and it's less restrictive license-wise than depending on MKL or the windows C runtime...
Is scipy still including any LGPL code, I thought not. There might still be some optional dependencies that not many users are using by default. ? Julia packages are mostly MIT, AFAIK. (except for the GPL parts because of cholmod, which we (?) avoid)
Josef
-n
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

On Sat, 2017-06-24 at 15:47 -0400, josef.pktd@gmail.com wrote:
On Sat, Jun 24, 2017 at 3:16 PM, Nathaniel Smith njs@pobox.com wrote:
On Jun 24, 2017 7:29 AM, "Sylvain Corlay" <sylvain.corlay@gmail.com
wrote:
Also, one quick question: is the LGPL license a deliberate choice or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency. If you are the only copyright holder, you would still have the possibility to license it under a more permissive license such as BSD or MIT...
Why would LGPL be a problem in a dependency? That doesn't stop you making your code BSD, and it's less restrictive license-wise than depending on MKL or the windows C runtime...
Is scipy still including any LGPL code, I thought not. There might still be some optional dependencies that not many users are using by default. ? Julia packages are mostly MIT, AFAIK. (except for the GPL parts because of cholmod, which we (?) avoid)
Well, I don't think scipy has many dependencies (but I would not be surprised if those are LGPL). Not a specialist, but as a dependency it should be fine (that is the point of the L in LGPL after all as far as I understand, it is much less viral). If you package it with your own stuff, you have to make sure to point out that parts are LGPL of course (just like there is a reason you get the GPL printed out with some devices) and if you modify it provide these modifications, etc.
Of course you cannot include it into the scipy codebase itself, but there is probably no aim of doing so here, so without a specific reason I would think that LGPL is a great license.
- Sebastian
Josef
-n
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

Does this still apply: https://scipy.github.io/old-wiki/pages/License_Compatibility.html
Carl
2017-06-24 22:07 GMT+02:00 Sebastian Berg sebastian@sipsolutions.net:
On Sat, 2017-06-24 at 15:47 -0400, josef.pktd@gmail.com wrote:
On Sat, Jun 24, 2017 at 3:16 PM, Nathaniel Smith njs@pobox.com wrote:
On Jun 24, 2017 7:29 AM, "Sylvain Corlay" <sylvain.corlay@gmail.com
wrote:
Also, one quick question: is the LGPL license a deliberate choice or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency. If you are the only copyright holder, you would still have the possibility to license it under a more permissive license such as BSD or MIT...
Why would LGPL be a problem in a dependency? That doesn't stop you making your code BSD, and it's less restrictive license-wise than depending on MKL or the windows C runtime...
Is scipy still including any LGPL code, I thought not. There might still be some optional dependencies that not many users are using by default. ? Julia packages are mostly MIT, AFAIK. (except for the GPL parts because of cholmod, which we (?) avoid)
Well, I don't think scipy has many dependencies (but I would not be surprised if those are LGPL). Not a specialist, but as a dependency it should be fine (that is the point of the L in LGPL after all as far as I understand, it is much less viral). If you package it with your own stuff, you have to make sure to point out that parts are LGPL of course (just like there is a reason you get the GPL printed out with some devices) and if you modify it provide these modifications, etc.
Of course you cannot include it into the scipy codebase itself, but there is probably no aim of doing so here, so without a specific reason I would think that LGPL is a great license.
- Sebastian
Josef
-n
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

On Sat, 2017-06-24 at 22:58 +0200, Carl Kleffner wrote:
Does this still apply: https://scipy.github.io/old-wiki/pages/License _Compatibility.html
Of course, but it talks about putting it into the code base of scipy not about being able to use the package in any way in a dependency (i.e. `import package`).
- Sebastian
Carl
2017-06-24 22:07 GMT+02:00 Sebastian Berg <sebastian@sipsolutions.net
: On Sat, 2017-06-24 at 15:47 -0400, josef.pktd@gmail.com wrote:
On Sat, Jun 24, 2017 at 3:16 PM, Nathaniel Smith njs@pobox.com wrote:
On Jun 24, 2017 7:29 AM, "Sylvain Corlay" <sylvain.corlay@gmail
.com
wrote:
Also, one quick question: is the LGPL license a deliberate
choice
or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency.
If
you are the only copyright holder, you would still have the possibility to license it under a more permissive license such
as
BSD or MIT...
Why would LGPL be a problem in a dependency? That doesn't stop
you
making your code BSD, and it's less restrictive license-wise
than
depending on MKL or the windows C runtime...
Is scipy still including any LGPL code, I thought not. There might still be some optional dependencies that not many
users
are using by default. ? Julia packages are mostly MIT, AFAIK. (except for the GPL parts because of cholmod, which we (?) avoid)
Well, I don't think scipy has many dependencies (but I would not be surprised if those are LGPL). Not a specialist, but as a dependency it should be fine (that is the point of the L in LGPL after all as far as I understand, it is much less viral). If you package it with your own stuff, you have to make sure to point out that parts are LGPL of course (just like there is a reason you get the GPL printed out with some devices) and if you modify it provide these modifications, etc.
Of course you cannot include it into the scipy codebase itself, but there is probably no aim of doing so here, so without a specific reason I would think that LGPL is a great license.
- Sebastian
Josef
-n
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

On Sat, Jun 24, 2017 at 5:26 PM, Sebastian Berg sebastian@sipsolutions.net wrote:
On Sat, 2017-06-24 at 22:58 +0200, Carl Kleffner wrote:
Does this still apply: https://scipy.github.io/old-wiki/pages/License _Compatibility.html
Of course, but it talks about putting it into the code base of scipy not about being able to use the package in any way in a dependency (i.e. `import package`).
But scipy does bundle a lot of external packages and then the license is relevant. I have no idea if this would apply here, but I find Sylvain's question relevant if closer integration and usage within the scientific python, and maybe Julia, community is desired.
LGPL is not as bad as GPL but still adds another hurdle, as far as I know the scipy and related history.
Josef
- Sebastian
Carl
2017-06-24 22:07 GMT+02:00 Sebastian Berg <sebastian@sipsolutions.net
: On Sat, 2017-06-24 at 15:47 -0400, josef.pktd@gmail.com wrote:
On Sat, Jun 24, 2017 at 3:16 PM, Nathaniel Smith njs@pobox.com wrote:
On Jun 24, 2017 7:29 AM, "Sylvain Corlay" <sylvain.corlay@gmail
.com
wrote:
Also, one quick question: is the LGPL license a deliberate
choice
or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency.
If
you are the only copyright holder, you would still have the possibility to license it under a more permissive license such
as
BSD or MIT...
Why would LGPL be a problem in a dependency? That doesn't stop
you
making your code BSD, and it's less restrictive license-wise
than
depending on MKL or the windows C runtime...
Is scipy still including any LGPL code, I thought not. There might still be some optional dependencies that not many
users
are using by default. ? Julia packages are mostly MIT, AFAIK. (except for the GPL parts because of cholmod, which we (?) avoid)
Well, I don't think scipy has many dependencies (but I would not be surprised if those are LGPL). Not a specialist, but as a dependency it should be fine (that is the point of the L in LGPL after all as far as I understand, it is much less viral). If you package it with your own stuff, you have to make sure to point out that parts are LGPL of course (just like there is a reason you get the GPL printed out with some devices) and if you modify it provide these modifications, etc.
Of course you cannot include it into the scipy codebase itself, but there is probably no aim of doing so here, so without a specific reason I would think that LGPL is a great license.
- Sebastian
Josef
-n
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

On 20170624@16:29, Sylvain Corlay wrote:
Hi Michele,
Hi Sylvain,
This is really interesting. I am a co-author of the xtensor project and one thing that could be interesting is to wrap the various sparse matrix data structures in the form of xtensor expressions. A byproduct of doing so is that it would simplify creating bindings for multiple scientific computing languages (Python, Julia, R, and more coming). You can see the blog post http://quantstack.net/c++/2017/05/30/polyglot-scientific-computing-with- xtensor.html for reference...
This article exemplifies manipulation of numerical arrays. Now I ask you: Given an interactive language $L of the one you cite above, can xtensor provide objects with custom type and operators for manipulation in *that* language, like in e.g. the pyrsb case: a=rsb_matrix((4,4)) print(a+a) # + operator and 'print' interfacing ?
Also, one quick question: is the LGPL license a deliberate choice or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency. If you are the only copyright holder, you would still have the possibility to license it under a more permissive license such as BSD or MIT...
No important choice. No problems relicensing the PyRSB prototype to BSD or MIT.
Congratulations on the release!
Thanks for the interest and welcome constructive feedback :-)
Sylvain
On Wed, Jun 21, 2017 at 11:51 AM, Michele Martone <michelemartone@users. sourceforge.net> wrote:
Hi.
I'm the author of the high performance multithreaded sparse matrix library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/
I'm *not* a user of SciPy/NumPy/Python, but using Cython I have written a proof-of-concept interface to librsb, named `PyRSB': https://github.com/michelemartone/pyrsb
PyRSB is in a prototypal state; e.g. still lacks good error handling. Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'. Advantages over csr_matrix are in fast multithreaded multiplication of huge sparse matrices. Intended application area is iterative solution of linear systems; particularly fast if with symmetric matrices and many rhs.
With this email I am looking for prospective:
- users/testers
- developers (any interest to collaborate/adopt/include the project?)
Looking forward for your feedback, Michele
SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev

Hi Michele,
Regarding xtensor, the article focused on how we can wrap an existing data structure in an "xtensor expression", i.e. something that you can operate upon with the numpy style API in C++ (see the numpy to xtensor cheat sheet http://xtensor.readthedocs.io/en/latest/numpy.html). Since it is an expression system, you can do "a + b * sin(t)" with numpy-style broadcasting and it returns an un-evaluated expression, which is computed only upon access or assignment to a container. "a", "b", and "c" could all be compound expressions, backed by filesystem operations, be an in-memory container, or a wrapper on your sparse matrices...
Now, we are working on automating the "*C++ tensor expression-type -> python wrapper with access and math operators" *generation of code, which would be what you are describing below...
On the subject of the LGPL, regarding the distinction with the GPL that were brought up earlier in the thread, it is indeed not as problematic as the GPL, but still less permissive as BSD and MIT. In general, I think that using a BSD or MIT license averts these concerns in the Python ecosystem, and is preferable *if you don't care about copyleft*...
Best,
Sylvain
On Sun, Jun 25, 2017 at 12:01 PM, Michele Martone < michelemartone@users.sourceforge.net> wrote:
On 20170624@16:29, Sylvain Corlay wrote:
Hi Michele,
Hi Sylvain,
This is really interesting. I am a co-author of the xtensor project and
one
thing that could be interesting is to wrap the various sparse matrix data structures in the form of xtensor expressions. A byproduct of doing so is that it would simplify creating bindings for multiple scientific
computing
languages (Python, Julia, R, and more coming). You can see the blog post http://quantstack.net/c++/2017/05/30/polyglot-scientific-computing-with- xtensor.html for reference...
This article exemplifies manipulation of numerical arrays. Now I ask you: Given an interactive language $L of the one you cite above, can xtensor provide objects with custom type and operators for manipulation in *that* language, like in e.g. the pyrsb case: a=rsb_matrix((4,4)) print(a+a) # + operator and 'print' interfacing ?
Also, one quick question: is the LGPL license a deliberate choice or is
it
not important to you? Most projects in the Python scientific stack are
BSD
licensed. So the LGPL choice makes it unlikely that a higher-level
project
adopts it as a dependency. If you are the only copyright holder, you
would
still have the possibility to license it under a more permissive license such as BSD or MIT...
No important choice. No problems relicensing the PyRSB prototype to BSD or MIT.
Congratulations on the release!
Thanks for the interest and welcome constructive feedback :-)
Sylvain
On Wed, Jun 21, 2017 at 11:51 AM, Michele Martone <michelemartone@users. sourceforge.net> wrote:
Hi.
I'm the author of the high performance multithreaded sparse matrix library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/
I'm *not* a user of SciPy/NumPy/Python, but using Cython I have written a proof-of-concept interface to librsb, named `PyRSB': https://github.com/michelemartone/pyrsb
PyRSB is in a prototypal state; e.g. still lacks good error handling. Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'. Advantages over csr_matrix are in fast multithreaded multiplication of huge sparse matrices. Intended application area is iterative solution of linear systems; particularly fast if with symmetric matrices and many rhs.
With this email I am looking for prospective:
- users/testers
- developers (any interest to collaborate/adopt/include the project?)
Looking forward for your feedback, Michele
SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev

On 20170624@16:29, Sylvain Corlay wrote:
Hi Michele,
Hi Sylvain,
This is really interesting. I am a co-author of the xtensor project and one thing that could be interesting is to wrap the various sparse matrix data structures in the form of xtensor expressions. A byproduct of doing so is that it would simplify creating bindings for multiple scientific computing languages (Python, Julia, R, and more coming). You can see the blog post http://quantstack.net/c++/2017/05/30/polyglot-scientific-computing-with- xtensor.html for reference...
This article exemplifies manipulation of numerical arrays. Now I ask you: Given an interactive language $L of the one you cite above, can xtensor provide objects with custom type and operators for manipulation in *that* language, like in e.g. the pyrsb case: a=rsb_matrix((4,4)) print(a+a) # + operator and 'print' interfacing ?
Also, one quick question: is the LGPL license a deliberate choice or is it not important to you? Most projects in the Python scientific stack are BSD licensed. So the LGPL choice makes it unlikely that a higher-level project adopts it as a dependency. If you are the only copyright holder, you would still have the possibility to license it under a more permissive license such as BSD or MIT...
No important choice. No problems relicensing the PyRSB prototype to BSD or MIT.
Congratulations on the release!
Thanks for the interest and welcome constructive feedback :-)
Sylvain
On Wed, Jun 21, 2017 at 11:51 AM, Michele Martone <michelemartone@users. sourceforge.net> wrote:
Hi.
I'm the author of the high performance multithreaded sparse matrix library `librsb' (mostly C, LGPLv3): http://librsb.sourceforge.net/
I'm *not* a user of SciPy/NumPy/Python, but using Cython I have written a proof-of-concept interface to librsb, named `PyRSB': https://github.com/michelemartone/pyrsb
PyRSB is in a prototypal state; e.g. still lacks good error handling. Its interface is trivial, as it mimicks that of SciPy's 'csr_matrix'. Advantages over csr_matrix are in fast multithreaded multiplication of huge sparse matrices. Intended application area is iterative solution of linear systems; particularly fast if with symmetric matrices and many rhs.
With this email I am looking for prospective:
- users/testers
- developers (any interest to collaborate/adopt/include the project?)
Looking forward for your feedback, Michele
SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
SciPy-Dev mailing list SciPy-Dev@python.org https://mail.python.org/mailman/listinfo/scipy-dev
participants (6)
-
Carl Kleffner
-
josef.pktd@gmail.com
-
Michele Martone
-
Nathaniel Smith
-
Sebastian Berg
-
Sylvain Corlay