File-like filter pattern?

[Oren Tirosh]

On Tue, Dec 17, 2002 at 05:55:37PM -0800, Erik Max Francis wrote:
> Is there a commonly accepted pattern for implementing file-like filters
> in Python?  That is, a generally accepted approach for file-like objects
> that can act as filters and point to other file-like objects (which may
> themselves also be filters).
> 
> For EmPy (my Python templating system), I support filtering of processed
> text, and to do this I have a base filter class which looks like this
> (I've trimmed some bells and whistles out, as well as some
> implementation details):
...

Your filters are of the "push" type: a data source pushes data to the 
first filter which feeds it to the next filter, until it reaches the 
destinatio.

Another approach to implementing filter chains is "pull" filters where the
sink pulls data from the last filter in the chain, which reads from the 
filter above it and eventually reads data from a source.

These two approaches are more-or-less equivalent but they don't mix very 
well. You can switch from pull to push by writing a loop that reads data 
from an upstream chain and feeds it to a downstream chain, but switching 
from push to pull requires the use of threads.

Personally, I prefer the "pull" type. I use the iterator protocol to
implement sources and transformations. Generator functions are a
particularly convenient way to implement stateful filters that read more 
or less than one data item from their upstream source for each item they 
yield.

	Oren