<div dir="ltr"><br><div class="gmail_quote"><div dir="ltr">On Wed, Nov 28, 2018 at 10:43 AM Gregory P. Smith <<a href="mailto:greg@krypto.org">greg@krypto.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><br><div class="gmail_quote"><div dir="ltr">On Wed, Nov 28, 2018 at 9:52 AM Brett Cannon <<a href="mailto:brett@python.org" target="_blank">brett@python.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Are we getting to the point that we want a compresslib like hashlib if we are going to be adding more compression algorithms?<br></div></blockquote><div><br></div><div>Lets avoid the lib suffix when unnecessary. I used the name hashlib because the name hash was already taken by a builtin that people normally shouldn't be using. zlib gets a lib suffix because a one letter name is evil and it matches the project name. ;) "compress" sounds nicer.</div><div><br></div><div>... looking on PyPI to see if that name is taken: <a href="https://pypi.org/project/compress/" target="_blank">https://pypi.org/project/compress/</a> exists and is already effectively what you are describing. (never used it or seen it used, no idea about quality)</div><div><br></div><div>I don't think adding lz4 to the stdlib is worthwhile. It isn't required for core functionality as zlib is (lowest common denominator zip support). I'd argue that bz2 doesn't even belong in the stdlib, but we shouldn't go removing things. PyPI makes getting more algorithms easy.</div><div><br></div><div>If anything, it'd be nice to standardize on some stdlib namespaces that others could plug their modules into. Create a compress in the stdlib with zlib and bz2 in it, and a way for extension modules to add themselves in a managed manner instead of requiring a top level name? Opening up a designated namespace to third party modules is not something we've done as a project in the past though. It requires care. I haven't thought that through.</div><div><br></div><div>-gps</div></div></div></div></blockquote><div><br></div><div>While my gut reaction was to say "no" to adding lz4 to the stdlib above... </div><div><br></div><div>I'm finding myself reconsidering and not against adding lz4 to the stdlib.</div><div><br></div><div>I just want us to have a good reason if we do. This type of extension module tends to be very easy to maintain (and you are volunteering). A good reason in the past has been the algorithm being widely used. Obviously the case with zlib (gzip and zipfile), bz2, and lzma (.xz). Those are all slower and tighter though. lz4 is extremely fast, especially for decompression. It could make a nice addition as that is an area our standard library offers nothing.</div><div><br></div><div>So change my -1 to a +0.5.</div><div><br></div><div>Q: Are there other popular alternatives to fill that niche that we should strongly consider instead or as well?</div><div><br></div><div>5 years ago the answer would've been Snappy. 15 years ago the answer would've been LZO.</div><div><br></div><div>I suggest not rabbit-holing this on whether we should adopt a top level namespace for these such as "compress". A good question to ask, but we can resolve that larger topic on its own without blocking anything.</div><div><br></div><div>lz4 has claimed the global pypi lz4 module namespace today so moving it to the stdlib under that name is normal - A pretty transparent transition. If we do that, the PyPI version of lz4 should remain for use on older CPython versions, but effectively be frozen, never to gain new features once lz4 has landed in its first actual CPython release.</div><div><br></div><div>-gps</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"></div><br><div class="gmail_quote"><div dir="ltr">On Wed, 28 Nov 2018 at 08:44, Antoine Pitrou <<a href="mailto:solipsis@pitrou.net" target="_blank">solipsis@pitrou.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Wed, 28 Nov 2018 10:28:19 +0000<br>
Jonathan Underwood <<a href="mailto:jonathan.underwood@gmail.com" target="_blank">jonathan.underwood@gmail.com</a>> wrote:<br>
> Hi,<br>
> <br>
> I have for sometime maintained the Python bindings to the LZ4<br>
> compression library[0, 1]:<br>
> <br>
> I am wondering if there is interest in having these bindings move to<br>
> the standard library to sit alongside the gzip, lzma etc bindings?<br>
> Obviously the code would need to be modified to fit the coding<br>
> guidelines etc.<br>
<br>
Personally I would find it useful indeed. LZ4 is very attractive<br>
when (de)compression speed is a primary factor, for example when<br>
sending data over a fast network link or a fast local SSD.<br>
<br>
Another compressor worth including is Zstandard (by the same author as<br>
LZ4). Actually, Zstandard and LZ4 cover most of the (speed /<br>
compression ratio) range quite well. Informative graphs below:<br>
<a href="https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/" rel="noreferrer" target="_blank">https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstandard/</a><br>
<br>
Regards<br>
<br>
Antoine.<br>
<br>
<br>
_______________________________________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org" target="_blank">Python-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-dev" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/brett%40python.org" rel="noreferrer" target="_blank">https://mail.python.org/mailman/options/python-dev/brett%40python.org</a><br>
</blockquote></div>
_______________________________________________<br>
Python-Dev mailing list<br>
<a href="mailto:Python-Dev@python.org" target="_blank">Python-Dev@python.org</a><br>
<a href="https://mail.python.org/mailman/listinfo/python-dev" rel="noreferrer" target="_blank">https://mail.python.org/mailman/listinfo/python-dev</a><br>
Unsubscribe: <a href="https://mail.python.org/mailman/options/python-dev/greg%40krypto.org" rel="noreferrer" target="_blank">https://mail.python.org/mailman/options/python-dev/greg%40krypto.org</a><br>
</blockquote></div></div></div>
</blockquote></div></div>