If what you really want is sparse matrices, you should use those: https://docs.scipy.org/doc/scipy/reference/sparse.html.

Or maybe from the experimental Dask offshoot that I contributed a few lines to: https://github.com/mrocklin/sparse.

Either of those will be about two orders of magnitude faster than working with Python lists for numeric data.

The reason, I think, that there's no RLE module in Python (standard library, there's probably something on PyPI) is that it's so easy to roll your own with the building blocks in itertools.  The `zipfile` and `gzip` modules are written in hundreds or thousands of lines of C code, and more importantly they are for dealing with *files* mostly, not generic sequences... that's the domain of `itertools`, but `itertools` is kept to a bare minimum collection of building blocks from which 1-10 line functions can be built efficiently.

On Sat, Jun 10, 2017 at 8:14 PM, Neal Fultz <nfultz@gmail.com> wrote:
Thanks, that's cool.   Maybe the root problem is that the docs aren't using the right words when I google. Run-length-encoding is particularly relevant for spare matrices, but there's probably a library for those as well.  On the data science side of things, there's a few hundred R packages that use it there[1].

Can you explicate the guiding principle a bit? I'm perplexed that python would come with zip and gzip but not rle.


On Sat, Jun 10, 2017 at 7:59 PM, David Mertz <mertz@gnosis.cx> wrote:
Here's a one-line version:

from itertools import groupby
rle_encode = lambda it: (
    (l[0],len(l)) for g in groupby(it) for l in [list(g[1])])

Since "not every one line function needs to be in the standard library" is a guiding principle of Python, and even moreso of `itertools`, probably this is a recipe in the documentation at most.  Or maybe it would have a home in `more_itertools`.


On Sat, Jun 10, 2017 at 7:20 PM, Neal Fultz <nfultz@gmail.com> wrote:
Hello python-ideas,

I am very new to this, but on a different  forum and after a couple conversations, I really wished Python came with run-length encoding built-in; after all, it ships with zip, which is much more complicated :)

The general idea is to be able to go back and forth between two representations of a sequence:

[1,1,1,1,2,3,4,4,3,3,3]

and

[(1, 4), (2, 1), (3, 1), (4, 2), (3, 3)]

where the first element is the data element, and the second is how many times it is repeated.

I wrote an encoder/decoder in about 20 lines ( https://github.com/nfultz/rle.py/blob/master/rle.py ) and would like to offer it for the next version; I think it might fit in nicely in the itertools module, for example. I am curious about your thoughts.

Best,  

-Neal


_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/




--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.




--
Keeping medicines from the bloodstreams of the sick; food
from the bellies of the hungry; books from the hands of the
uneducated; technology from the underdeveloped; and putting
advocates of freedom in prisons.  Intellectual property is
to the 21st century what the slave trade was to the 16th.