On Sun, Jan 24, 2021 at 9:53 AM <2QdxY4RzWzUUiLuE@potatochowder.com> wrote:
On 2021-01-25 at 00:29:41 +1100,
Steven D'Aprano <steve@pearwood.info> wrote:

> On Sat, Jan 23, 2021 at 03:24:12PM +0000, Barry Scott wrote:
> > First problem I see is that the file may be a pipe and then you will block
> > until you have enough data to do the auto detect.
>
> Can you use `open('filename')` to read a pipe?

Yes.  Named pipes are files, at least on POSIX.

And no.  Unnamed pipes are identified by OS-level file descriptors, so
you can't open them with open('filename'),

The `open` function takes either a file path as a string, or a file descriptor as an integer. So you can use `open` to read an unnamed pipe or a socket.

> Is blocking a problem in practice? If you try to open a network file,
> that could block too, if there are network issues. And since you're
> likely to follow the open with a read, the read is likely to block. So
> over all I don't think that blocking is an issue.

If open blocks too many bytes, then my application never gets to respond
unless enough data comes through the pipe.

It's possible to do a `f.read(1)` on a file opened in text mode. If the first two bytes of the file are 0xC2 0x99, that's either ™ if the file is UTF-8, or 슙 if the file is UTF-16BE, or 駂 if the file is UTF-16LE. And `f.read(1)` needs to pick one of those and return it immediately. It can't wait for more information. The contract of `read` is "Read from underlying buffer until we have n characters or we hit EOF." A call to `read(1)` cannot keep blocking after the first character was received to decide what encoding to decode it as; that would be backwards incompatible, and it might block forever if the sender only sends one character before waiting for a response.

~Matt