Frustrating circular bytes issue

[Stefan Behnel]

J, 26.06.2012 18:30:
> def _reader(self, file, size=4096, delimiter=r"\n{2,}"):
>         buffer_old = ""
>         while True:
>             buffer_new = file.read()
>             print(type(buffer_new))
>             if not buffer_new:
>                 break
>             lines = re.split(delimiter, buffer_old + buffer_new)

"delimiter" is a Unicode string, which makes the regular expression a
Unicode regex that can't work on a byte string.


>             buffer_old = lines.pop(-1)
> 
>             for line in lines:
>                 yield line
> 
>         yield buffer_old
> 
> 
> (the print statement is something I put in to verify the problem.
> 
> So stepping through this, when _reader executes, it executes read() on
> the opened filehandle.  Originally, it read in 4096 byte chunks, I
> removed that to test a theory.  It creates buffer_new with the output
> of the read.
> 
> Running type() on buffer_new tells me that it's a bytes object.

Stefan