On Wed, Jul 31, 2013 at 10:40:03AM -0500, Ryan wrote:

Here's something more interesting than my shlex idea.

os.path is, pretty much, the Python FS toolbox, along with shutil. But, there's one feature missing: check if a file is binary. It isn't hard, see http://code.activestate.com/recipes/173220/. But, writing 50 lines of code for a more common task isn't really Python-ish.

So...

What if os.path had a binary checker that works just like isfile: os.path.isbinary('/nothingness/is/eternal') # Returns boolean

What is a binary file? Would Russian text in koi8-r encoding be considered binary? What about utf-16? UTF16-encoded files have many zero characters. UTF32-encoded have even more. Oleg. -- Oleg Broytman http://phdru.name/ phd@phdru.name Programmers don't die, they just GOSUB without RETURN.

On 31 juil. 2013, at 18:02, Oleg Broytman wrote:

On Wed, Jul 31, 2013 at 10:40:03AM -0500, Ryan wrote:

...

Here's something more interesting than my shlex idea.

os.path is, pretty much, the Python FS toolbox, along with shutil. But, there's one feature missing: check if a file is binary. It isn't hard, see http://code.activestate.com/recipes/173220/. But, writing 50 lines of code for a more common task isn't really Python-ish.

So...

What if os.path had a binary checker that works just like isfile: os.path.isbinary('/nothingness/is/eternal') # Returns boolean

What is a binary file? Would Russian text in koi8-r encoding be considered binary? What about utf-16? UTF16-encoded files have many zero characters. UTF32-encoded have even more.

And the recipe linked is worse than that: even with no nul byte, if more than 30% of the files's bytes aren't ASCII it considers the file binary. Files in iso-8859 parts 5 to 8 (Cyrillic, Arabic, Greek and Hebrew) are pretty much guaranteed to be inferred as binary. Part 11 (Thai) as well. UTF-8 for any non-Latin script will also be considered binary as the high bit is always set when encoding codepoints outside the ASCII range.

On Jul 31, 2013 12:22 PM, "Eli Bendersky" wrote:

On Wed, Jul 31, 2013 at 8:40 AM, Ryan wrote:

...

Here's something more interesting than my shlex idea.

os.path is, pretty much, the Python FS toolbox, along with shutil. But,

...

So...

What if os.path had a binary checker that works just like isfile: os.path.isbinary('/nothingness/is/eternal') # Returns boolean

Besides the high chance of false positives, what makes this method (and the

there's one feature missing: check if a file is binary. It isn't hard, see http://code.activestate.com/recipes/173220/. But, writing 50 lines of code for a more common task isn't really Python-ish. problem it tries to solve) so so difficult is that binary files may contain what is considered to be large amounts of text, and text files may contain pieces of binary data. For example, consider a windows executable file - Much of the data in such a file is considered binary data, but there are defined sections where strings and text resources are stored. Any heuristic algorithm like the one mentioned will be insufficient in such cases. Although I can't think of a situation off hand where the opposite may be true (binary data embedded in what is considered to be a text file) I'm pretty sure such a situation exists.

Some time ago I put on a gas mask and dove into the Perl source code to

figure out how its "is binary" and "is text" operators work: http://eli.thegreenplace.net/2011/10/19/perls-guess-if-file-is-text-or-binar...

I would recommend against including such a simplistic heuristic in the

Python stdlib.

Eli

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

On Jul 31, 2013, at 12:03, Ryan wrote:

So...although that is a nice piece of code, an os.path implementation would probably be more complete and foolproof.

And because there is no foolproof, or even remotely close to foolproof, way to do it, there can be no os.path implementation.

On 7/31/2013 3:03 PM, Ryan wrote:

1.The link I provided wasn't how I wanted it to be.

And there is no 'one way' that will satisfy everyone, or every most people, as they will have different use cases for 'istext'.

I was using it as an example to show it wasn't impossible.

It is obviously possible to apply any arbitrary predicate to any object within its input domain. No one has claimed otherwise that I know of.

2.You yourself stated it doesn't work on UTF-8 files. If you wanted one that worked on all text files, it wouldn't work right.

The problem is that the problem is ill-defined. Every file is (or can be viewed as) a sequence of binary bytes. Every file can be interpreted as a text file encoded with any of the encodings (like at least some latin-1 encodings, and the IBM PC Graphics encoding) that give a character meaning to every byte. So, to be strict, every file is both binary and text. Python allows us to open any file as either binary or text (with some encoding, with latin-1 one of the possible choices). The pragmatic question is 'Is this file 'likely' *intended* to be interpreted as text, given that the creator is a member of our *local culture*. For the function you referenced, the 'local culture' is 'closed Western European'. For 'closed American', the threshold of allowed non-ascii text and control chars should be more like 0 or 1%. For many cultures, the referenced function is nonsensical. For an open global context, istext would have to try all standard text encodings and for those that worked, apply the grammar rules of the languages that normally are encoded with that encoding. -- Terry Jan Reedy

I just realized I misexpressed myself...again. I meant ASCII or binary, not text or binary. Kind of like the old FTP programs. The implementation would determine if it was ASCII or binary. And, the '/nothingness/is/eternal' is a quote from Xemnas in Kingdom Hearts. I was hoping someone would pick it up. Terry Reedy wrote:

On 7/31/2013 3:03 PM, Ryan wrote:

...

1.The link I provided wasn't how I wanted it to be.

And there is no 'one way' that will satisfy everyone, or every most people, as they will have different use cases for 'istext'.

...

I was using it as an example to show it wasn't impossible.

It is obviously possible to apply any arbitrary predicate to any object

within its input domain. No one has claimed otherwise that I know of.

...

2.You yourself stated it doesn't work on UTF-8 files. If you wanted one that worked on all text files, it wouldn't work right.

The problem is that the problem is ill-defined. Every file is (or can be viewed as) a sequence of binary bytes. Every file can be interpreted as

a text file encoded with any of the encodings (like at least some latin-1 encodings, and the IBM PC Graphics encoding) that give a character meaning to every byte. So, to be strict, every file is both binary and text. Python allows us to open any file as either binary or text (with some encoding, with latin-1 one of the possible choices).

The pragmatic question is 'Is this file 'likely' *intended* to be interpreted as text, given that the creator is a member of our *local culture*. For the function you referenced, the 'local culture' is 'closed Western European'. For 'closed American', the threshold of allowed non-ascii text and control chars should be more like 0 or 1%. For many cultures, the referenced function is nonsensical.

For an open global context, istext would have to try all standard text encodings and for those that worked, apply the grammar rules of the languages that normally are encoded with that encoding.

-- Terry Jan Reedy

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

On Wed, Jul 31, 2013 at 4:40 PM, Ryan wrote:

Here's something more interesting than my shlex idea.

os.path is, pretty much, the Python FS toolbox, along with shutil. But, there's one feature missing: check if a file is binary. It isn't hard, see http://code.activestate.com/recipes/173220/. But, writing 50 lines of code for a more common task isn't really Python-ish.

So...

What if os.path had a binary checker that works just like isfile: os.path.isbinary('/nothingness/is/eternal') # Returns boolean

Going right back to the beginning here. Suppose this were deemed useful. Why should it be in os.path? Nothing else there, as far as I know, looks at the *contents* of a file. Everything's looking at directory entries, sometimes not even that (eg os.path.basename is pure string manipulation). I should be able to getctime() on a file even without permission to read it. I can't see whether it's binary or text without read permission. This sounds more like a job for a file-like object, maybe a subclass of file that reads (and buffers) the first 512 bytes, guesses whether it's text or binary, and then watches everything that goes through after that and revises its guess later on. And then the question becomes: How useful would that be? But mainly, I think it's only going to cause problems to have a potentially expensive operation stuck away with the very cheap operations in os.path. ChrisA

On 01/08/13 10:25, Ryan wrote:

I just realized I misexpressed myself...again. I meant ASCII or binary, not text or binary. Kind of like the old FTP programs. The implementation would determine if it was ASCII or binary.

Still can't be done reliably, but even if it could, what's so special about ASCII? Should we have dozens of such functions? isascii isbig5 iskoi8u iskoi8r and so on? The concept of "isbinary" is fundamentally flawed. The concept of "try to guess what sort of data a file might plausibly contain" is not flawed, but is a much, much bigger problem than is suitable for a simple os.path function. -- Steven

On Wed, Jul 31, 2013 at 10:11 PM, Steven D'Aprano wrote:

Still can't be done reliably, but even if it could, what's so special about ASCII?

Lots of things are special about ASCII. It is a 7-bit subset of pretty much every modern encoding scheme. Being 7-bit, it can be fairly reliably distinguished from most binary formats. Same is true about UTF-8. It is very unlikely that a binary dump of a double array make a valid UTF-8 text and vice versa - UTF-8 text interpreted as a list of doubles is unlikely to produce numbers that are in a reasonable range. I would not mind seeing an "istext()" function somewhere in the stdlib that would only recognize ASCII and UTF-8 as text.

From: Ryan Sent: Wednesday, July 31, 2013 7:45 PM

What about something like this:

https://github.com/ahupp/python-magic

That particular wrapper is just a ctypes wrapper libmagic.so/.dylib, and not a very portable one (e.g., it's got the path /usr/local/Cellar/libmagic/5.10/ hardcoded into it… which will only work for Mac Homebrew users who are 4 versions/19 months out of date…). Also, note that libmagic already comes with very similar Python ctypes-based bindings. However, there are a half-dozen other wrappers around libmagic on PyPI, and it's pretty trivial to create a new one. The tricky bit is where to get the libmagic code and data files from. If you want to make it usable on most platforms, you'd need to add the file source distribution to the Python source, build it with Python, and statically link it into a module, ala zlib or sqlite. And, unlike those modules, you'd also need to include a data file (magic.mgc) with the binary distribution. I slapped together a quick&dirty wrapper to see what the costs are. It adds 640KB to the 14MB source distribution, 300KB to the 91MB binary (64-bit Mac framework build), and under 10 seconds to the build process. There'd be a bit of an extra maintenance burden in tracking updates (the most recent two updates were 21 Mar 2013 and 22 Feb 2013). The code and data are BSD-licensed, which shouldn't be a problem. The library is very portable: "./configure --enable-static; make" worked even on Windows. So, is it worth adding to Python? I don't know. But it seems at least feasible.

Andrew Barnert writes:

Plenty of files in popular charsets are actually perfectly valid UTF-8,

FVO "popular charset" in {ASCII} or "plenty of files" in "len(file) < 1KB", yes. Otherwise, see below.

but garbage when read that way. This and its converse are

The converse *is* a problem, because the ISO 8859 family (and even more so the Windows 125x family) basically use up all the bytes.

probably the most common cause of mojibake problems people have today.

Actually the most common cause in my experience is Apache or MUA configuration of a default charset and/or fallback to Latin-1 for files actually written in UTF-8, combined with conformant browsers and MUAs that respect transport-level defaults or protocol defaults rather than try to detect the charset. Viz:

(I don't know if you can search Stack Overflow for problems with "Ã" in the description, but if you can, it'll be illuminating.)

But:

… you're probably better off following EAFP and just doing this:

    try:         dotextstuff(b)     except UnicodeDecodeError:         dobinstuff(b)

Yes, indeedy! Just because those algorithms exist doesn't mean it's a good idea to use them (outside of some interactive applications like text editors where the user can look at the mojibake and tell the editor either the right encoding or to try another guess).

That's a pretty good idea. Or it could be like this: if fh.printable(): It would have an optional argument: the number of bytes to read in. Default is 512. So, if we wanted 1024 bytes instead of 512: if fh.printable(1024): David Mertz wrote:

On Wed, Jul 31, 2013 at 5:42 PM, Chris Angelico wrote:

...

This sounds more like a job for a file-like object, maybe a subclass of file that reads (and buffers) the first 512 bytes, guesses whether it's text or binary, and then watches everything that goes through after that and revises its guess later on.

Something like:

if fh.read(512).isprintable(): do_the_ascii_stuff(fh) else: do_the_bin_stuff(fh)

-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

------------------------------------------------------------------------

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

On Thu, Aug 1, 2013 at 2:47 PM, MRAB wrote:

...

Something like:

if fh.read(512).isprintable(): do_the_ascii_stuff(fh) else: do_the_bin_stuff(fh)

Except that:

...

...

...

"\n".isprintable() False

Doesn't that seem like a bug: ----- Help on method_descriptor: isprintable(...) S.isprintable() -> bool Return True if all characters in S are considered printable in repr() or S is empty, False otherwise. ----- In what sense is "\n" "not printable in repr()"?! -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

I see... This works: import re with open('test.xml', 'r') as f: print re.sub(r'(^(\'|\")|(\'|\")$)', '', repr(f.read()).replace('\\n', '\n')) But, repr can still print binary characters. Opening libexpat.so shows all sorts of crazy characters like \x00. random832@fastmail.us wrote:

On Thu, Aug 1, 2013, at 16:05, David Mertz wrote:

...

In what sense is "\n" "not printable in repr()"?!

Because it prints as \n instead of as itself, doesn't it? Or is that only in Python 2? _______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

I actually got something simple working. It's only been tested on Android: import re def isbinary(fpath): with open(fpath, 'r') as f: data = re.sub(r'(^(\'|\")|(\'|\")$)', '', repr(f.read()).replace('\\n', '\n')) binchars = re.findall(r'\\x[0123456789abcdef]{2}', data) per = (float(len(binchars)) / float(len(data))) * 100 if int(per) == 0: return True else: return False Ryan wrote:

That's a pretty good idea. Or it could be like this:

if fh.printable():

It would have an optional argument: the number of bytes to read in. Default is 512. So, if we wanted 1024 bytes instead of 512:

if fh.printable(1024):

David Mertz wrote:

...

On Wed, Jul 31, 2013 at 5:42 PM, Chris Angelico wrote:

...

This sounds more like a job for a file-like object, maybe a subclass of file that reads (and buffers) the first 512 bytes, guesses whether it's text or binary, and then watches everything that goes through after that and revises its guess later on.

Something like:

if fh.read(512).isprintable(): do_the_ascii_stuff(fh) else: do_the_bin_stuff(fh)

-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

------------------------------------------------------------------------

_______________________________________________ Python-ideas mailing list Python-ideas@python.org http://mail.python.org/mailman/listinfo/python-ideas

-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

-- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

On 2013-08-12, at 15:42 , Philipp A. wrote:

well, the only remotely valid thing to do is to test if the input data is decodable with any of the encodings python knows.

Most iso-8859 parts can decode any byte (and thus any byte sequence). Parts 3, 6, 7, 8 and 11 are the only ones not to be defined across all of the [128, 255] range (they're ascii extensions so the [0, 127] range is identical to ascii in all iso-8859 parts)

On 08/12/2013 03:52 PM, Masklinn wrote:

On 2013-08-12, at 15:42 , Philipp A. wrote:

...

well, the only remotely valid thing to do is to test if the input data is decodable with any of the encodings python knows.

Most iso-8859 parts can decode any byte (and thus any byte sequence).

Are you sure about the null byte? '\0' But yes, just looking if there is a '\0' in the file isn't a good heuristic either.

Parts 3, 6, 7, 8 and 11 are the only ones not to be defined across all of the [128, 255] range (they're ascii extensions so the [0, 127] range is identical to ascii in all iso-8859 parts)

On Mon, Aug 12, 2013 at 3:32 PM, Mathias Panzenböck wrote:

On 08/12/2013 03:52 PM, Masklinn wrote:

...

On 2013-08-12, at 15:42 , Philipp A. wrote:

...

well, the only remotely valid thing to do is to test if the input data is decodable with any of the encodings python knows.

Most iso-8859 parts can decode any byte (and thus any byte sequence).

Are you sure about the null byte? '\0' But yes, just looking if there is a '\0' in the file isn't a good heuristic either.

I've often used the presence of a NUL in the data as a simple heuristic for "binary file", though only in places where it won't matter (for instance, showing file size in bytes rather than line count - if a binary file happens to have no \0 and its number of \n gets counted, big deal). Otherwise, not worth the hassle of finding out. ChrisA