[issue20405] Add io.BinaryTransformWrapper and a "transform" parameter to open()

[Nick Coghlan]

Nick Coghlan added the comment:

I only used hex as the example because it was trivial to generate test data for.

The stackable streaming IO model is an extremely powerful one - the approach we already use in the io module has some similarities to the one Texas Instruments use in DSP/BIOS (http://www.ti.com/tool/dspbios) and I know from experience how convenient that is. The model means you can push a lot of your data manipulation into your stream definitions, and keep all that data transformation logic out of your main application. (In my case, it let us mostly ignore the differences in a-law, u-law and ADPCM encoded audio, since we just built the IO streams differently depending on which one we were dealing with).

However, relative to DSP/BIOS, our stream model is currently missing the "stackable" piece - it's difficult to plug additional wrappers into the stream, because we don't have either the "binary in, binary out" or the "text in, text out" component.

A well designed streaming codec should be able to sit in the pipeline providing transparent encryption whether you're piping to a file, to another process or to a socket. If you're handling audio or video data, then you would also be able to place your codecs directly in the stream pipeline, rather than needing to come up with your own custom data pipeline model.

This isn't a novel design overall - it's the way the signal processing world has been doing things for decades (I first learned this model when using DSP/BIOS more than a decade ago, and Linux STREAMS, which includes some similar concepts, is substantially older than that). The only novel concept here is the idea of offering this feature as part of Python 3's native io model.

DSP/BIOS and STREAMS also have some solid design concepts around using gather/scatter devices for stream multiplexing, but that's not related to codec handling improvements.

----------

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue20405>
_______________________________________