On Mon, May 1, 2017 at 9:38 AM, Nick Coghlan wrote:

just support two keyword arguments to hex(): "delimiter" (as you suggest) and "chunk_size" (defaulting to 1, so you get per-byte chunking by default)

I'd expect "chunk_size" to mean the number of hex digits (not bytes) per chunk. Cheers, -- Juancarlo *Añez*

On 2017-05-01 01:34 PM, Ethan Furman wrote:

On 05/01/2017 07:04 AM, Juancarlo Añez wrote:

...

On Mon, May 1, 2017 at 9:38 AM, Nick Coghlan wrote:

...

just support two keyword arguments to hex(): "delimiter" (as you suggest) and "chunk_size" (defaulting to 1, so you get per-byte chunking by default)

I'd expect "chunk_size" to mean the number of hex digits (not bytes) per chunk.

I was also surprised by that. Also, should Python be used on a machine with, say, 24-bit words then a chunk size of three makes more sense that one of 1.5. ;)

-- ~Ethan~ A hex digit is 4 bits long. To separate into words, the 24-bit word Python would use 3 (counting in bytes as initially proposed), or 6 (counting in hex digits). Neither option would result in a 1.5 chunk_size for 24-bit chunks.

Counting chunk_size either in nibbles or bytes seem equally intuitive to me (as long as it's documented).

On 5/1/2017 1:41 PM, Alexandre Brault wrote:

On 2017-05-01 01:34 PM, Ethan Furman wrote:

...

On 05/01/2017 07:04 AM, Juancarlo Añez wrote:

...

On Mon, May 1, 2017 at 9:38 AM, Nick Coghlan wrote:

...

just support two keyword arguments to hex(): "delimiter" (as you suggest) and "chunk_size" (defaulting to 1, so you get per-byte chunking by default)

I'd expect "chunk_size" to mean the number of hex digits (not bytes) per chunk.

I was also surprised by that. Also, should Python be used on a machine with, say, 24-bit words then a chunk size of three makes more sense that one of 1.5. ;)

-- ~Ethan~ A hex digit is 4 bits long. To separate into words, the 24-bit word Python would use 3 (counting in bytes as initially proposed), or 6 (counting in hex digits). Neither option would result in a 1.5 chunk_size for 24-bit chunks.

Counting chunk_size either in nibbles or bytes seem equally intuitive to me (as long as it's documented).

Call the paramater 'octets' and it should be clear that it means 8 bit chunks. Do any machine now use anything else? -- Terry Jan Reedy

On 1 May 2017 at 11:04, Juancarlo Añez wrote:

On Mon, May 1, 2017 at 9:38 AM, Nick Coghlan wrote:

...

just support two keyword arguments to hex(): "delimiter" (as you suggest) and "chunk_size" (defaulting to 1, so you get per-byte chunking by default)

I'd expect "chunk_size" to mean the number of hex digits (not bytes) per chunk.

So do I. Moreover, if "1" is for two digits, there is no way to specify single digits - for little use we can perceive for that. Maybe it does not need to be named "chunk_size" - "digits_per_block" is too big, but is precise. Also, whatever we think is good for "hex" could also be done to "bin" .

Cheers,

-- Juancarlo Añez

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

Sorry. I meant to be terse, but wasn't clear enough. I meant the method name. If it takes a `delimiter` karg, it would be consistent to call the operation `delimit`. On Tue, 2 May 2017 13:06 Carl Smith, wrote:

Couldn't it just be named `str.delimit`? I totally agree with Steve for what it's worth. Thanks for everything guys. Best,

On Tue, 2 May 2017 13:02 Joao S. O. Bueno, wrote:

...

On 1 May 2017 at 11:04, Juancarlo Añez wrote:

...

On Mon, May 1, 2017 at 9:38 AM, Nick Coghlan

...

...

just support two keyword arguments to hex(): "delimiter" (as you suggest) and "chunk_size" (defaulting to 1, so you get per-byte chunking by default)

I'd expect "chunk_size" to mean the number of hex digits (not bytes)

wrote: per

...

chunk. So do I. Moreover, if "1" is for two digits, there is no way to specify single digits - for little use we can perceive for that.

Maybe it does not need to be named "chunk_size" - "digits_per_block" is too big, but is precise.

Also, whatever we think is good for "hex" could also be done to "bin" .

...

Cheers,

-- Juancarlo Añez

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

For a name, I think "group" would be better than "chunk". We talk about grouping the digits of a number, not chunking them. -- Greg

On Mon, May 01, 2017 at 11:38:20PM +1000, Nick Coghlan wrote:

We're definitely open to offering better formatting options for bytes.hex().

My proposal in https://bugs.python.org/issue22385 was to define a new formatting mini-language (akin to the way strftime works, but with a much simpler formatting mini-language): http://bugs.python.org/issue22385#msg292663

However, a much simpler alternative would be to just support two keyword arguments to hex(): "delimiter" (as you suggest) and "chunk_size" (defaulting to 1, so you get per-byte chunking by default)

I disagree with this approach. There's nothing special about bytes.hex() here, perhaps we want to format the output of hex() or bin() or oct(), or for that matter "%x" and any of the other string templates? In fact, this is a string operation that could apply to any character string, including decimal digits. Rather than duplicate the API and logic everywhere, I suggest we add a new string method. My suggestion is str.chunk(size, delimiter=' ') and str.rchunk() with the same arguments: "1234ABCDEF".chunk(4) => returns "1234 ABCD EF" rchunk will be useful for money or other situations where we group from the right rather than from the left: "$" + str(10**6).rchunk(3, ',') => returns "$1,000,000" And if we want to add bells and whistles, we could accept a tuple for the size argument: # Format mobile phone number in the Australian style "04123456".rchunk((4, 3)) => returns "0412 345 678" # Format an integer in the Indian style str(123456789).rchunk((3, 2), ",") => returns "12,34,56,789" In the OP's use-case: bytes("abcde", "ascii").hex().chunk(2) => returns '61 62 63 64 65' bytes("abcde", "ascii").hex().chunk(4) => returns '6162 6364 65' I don't see any advantage to adding this to bytes.hex(), hex(), oct(), bin(), and I really don't think it is helpful to be grouping the characters by the number of bits. Its a string formatting operation, not a bit operation. -- Steve

On Tue, May 02, 2017 at 11:45:35PM +1000, Nick Coghlan wrote:

Attempting to align the terminology with existing string methods and other stdlib APIs: [...] 1. we don't have any current APIs or documentation that use "chunk" in combination with any kind of delimiter 2. we don't have any current APIs or documentation that use "chunk" as a verb - they all use it as a noun

English has a long and glorious tradition of verbing nouns, and nouning verbs. Group can mean the action of putting things into a group, join likewise refers to both the action of attaching two things and the seam or joint where they have been joined. Likewise for chunking: https://duckduckgo.com/html/?q=chunking "Chunk" has used as a verb since at least 1890 (albeit with a different meaning). None of my dictionaries give a date for the use of chunking to mean dividing something up into chunks, so that could be quite recent, but it's well-established in education (chunking as a technique for doing long division), psychology, linguistics and more. I remember using "chunking" as a verb to describe Hyperscript's text handling back in the mid 1980s, e.g. "word 2 of line 6 of text". The nltk library handles chunk as both a noun and verb in a similar sense: http://www.nltk.org/howto/chunk.html

So if we went with this approach, then Carl Smith's suggestion of "str.delimit()" likely makes sense.

The problem with "delimit" is that in many contexts it refers to marking both the start and end boundaries, e.g. people often refer to string delimiters '...' and list delimiters [...]. That doesn't apply here, where we're adding separators between chunks/groups. The term delimiter can be used in various ways, and some of them do not match the behaviour we want here: http://stackoverflow.com/questions/9118769/when-to-use-the-terms-delimiter-t... In this case, we are not adding delimiters, we're adding separators. We're chunking (or grouping) characters by counting them, then separating the groups. The test here is what happens if the string is shorter than the group size? "xyz".chunk(5, '*') If we're delimiting the boundaries of the group, then I expect that we should get "*xyz*", but if we're separating groups, I expect that we should get "xyz" unchanged.

However, the other question worth asking is whether we might want a "string slice splitting" operation rather than a string delimiting option: once you have the slices, then combining them again with str.join is straightforward, but extracting the slices in the first place is currently a little fiddly (especially for the reversed case):

Let me think about that :-) -- Steve

The main reason for naming it `delimit` was to be consistent with the karg `delimiter`, so `str.delimit(index, delimiter)`. You could call it `chop` I guess, but I'm just bikeshedding, so will leave it while you guys figure out the important stuff. -- Carl Smith carl.input@gmail.com On 2 May 2017 at 18:48, David Mertz wrote:

On Tue, May 2, 2017 at 4:31 AM, Steven D'Aprano wrote:

...

Rather than duplicate the API and logic everywhere, I suggest we add a new string method. My suggestion is str.chunk(size, delimiter=' ') and str.rchunk() with the same arguments:

"1234ABCDEF".chunk(4) => returns "1234 ABCD EF"

rchunk will be useful for money or other situations where we group from the right rather than from the left:

"$" + str(10**6).rchunk(3, ',') => returns "$1,000,000"

# Format mobile phone number in the Australian style "04123456".rchunk((4, 3)) => returns "0412 345 678"

# Format an integer in the Indian style str(123456789).rchunk((3, 2), ",") => returns "12,34,56,789"

I like this general idea very much. Dealing with lakh and crore is a very nice feature (and one that the `.format()` mini-language sadly fails to handle; it assumes numeric delimiters can only be commas, and only ever three positions).

But I'm not sure the semantics you propose is flexible enough. I take it that the tuple means (<first-delimiter>, <other-delimiters>) from your examples. But I don't think that suffices for every common format. It would be fine to get a USA phone number like:

str(4135559414 <(413)%20555-9414>).rchunk((4,3),'-') # -> 413-555-9414 <(413)%20555-9414>

But for example, looking somewhat at random at an international call ( https://en.wikipedia.org/wiki/Telephone_numbers_in_Belgium)

*Dialing from New York to Brussel**011-32-2-555-12-12 <+32%202%20555%2012%2012>* - Omitting the leading "0".

Maybe your API is for any length tuple, with the final element repeated. So I guess maybe this example could be:

"0113225551212 <+32%202%20555%2012%2012>".rchunk((2,2,3,1,2,3),'-')

I don't care about this method being called .chunk() vs. .delimit() vs. something else.

-- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

_______________________________________________ Python-ideas mailing list Python-ideas@python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

On 02/05/17 12:31, Steven D'Aprano wrote:

I disagree with this approach. There's nothing special about bytes.hex() here, perhaps we want to format the output of hex() or bin() or oct(), or for that matter "%x" and any of the other string templates?

In fact, this is a string operation that could apply to any character string, including decimal digits.

Rather than duplicate the API and logic everywhere, I suggest we add a new string method. My suggestion is str.chunk(size, delimiter=' ') and str.rchunk() with the same arguments:

"1234ABCDEF".chunk(4) => returns "1234 ABCD EF"

FWIW, I implemented a version of something similar as a fixed-length "chunk" method in itertoolsmodule.c (it was similar to izip_longest - it had a "fill" keyword to pad the final chunk). It was ~100 LOC including the structure definitions. The chunk method was an iterator (so it returned a sequence of "chunks" as defined by the API). Then I read that "itertools" should consist of primitives only and that we should defer to "moreitertools" for anything that is of a higher level (which this is - it can be done in terms of itertools functions). So I didn't propose it, although the processing of my WAV files (in which the sample data are groups of bytes - frames - of a fixed length) was significantly faster with it :( I also looked at implementing itertools.chunk as a function that would make use of a "__chunk__" method on the source object if it existed (which allowed a class to support an even more efficient version of chunking - things like range() etc).

I don't see any advantage to adding this to bytes.hex(), hex(), oct(), bin(), and I really don't think it is helpful to be grouping the characters by the number of bits. Its a string formatting operation, not a bit operation.

Why do you want to limit it to strings? Isn't something like this potentially useful for all sequences (where the result is a tuple of objects that are the same as the source sequence - be that strings or lists or lazy ranges or whatever?). Why aren't the chunks returned via an iterator? E.

On Tue, May 2, 2017 at 8:43 PM, Steven D'Aprano wrote:

String methods should return strings.

...

...

"A-B-C".split("-") ['A', 'B', 'C']

If chunk() worked for all iterables:

...

...

" ".join("1234ABCDEF".chunk(4)) "1234 ABCD EF"

Cheers, -- Juancarlo *Añez*

On 03/05/17 01:43, Steven D'Aprano wrote:

On Tue, May 02, 2017 at 11:39:48PM +0100, Erik wrote:

...

On 02/05/17 12:31, Steven D'Aprano wrote:

...

...

Rather than duplicate the API and logic everywhere, I suggest we add a new string method. My suggestion is str.chunk(size, delimiter=' ') and str.rchunk() with the same arguments:

For the record, I now think the second argument should be called "sep", for separator, and I'm okay with Greg's suggestion we call the method "group".

...

...

"1234ABCDEF".chunk(4) => returns "1234 ABCD EF" [...]

...

Why do you want to limit it to strings?

I'm not stopping anyone from proposing a generalisation of this that works with other sequence types. As somebody did :-)

Who? I didn't spot that in the thread - please give a reference. Thanks. Anyway, I know you can't stop anyone from *proposing* something like this, but as soon as they do you may decide to quote the recipe from "https://docs.python.org/3/library/functions.html#zip" and try to block their proposition. There are already threads on fora that do that. That was my sticking point at the time when I implemented a general solution. Why bother to propose something that (although it made my code significantly faster) had already been blocked as being something that should be a python-level operation and not something to be included in a built-in?

String methods should return strings.

In that case, we need to fix this ASAP ;) :

...

...

'foobarbaz'.split('o') ['f', '', 'barbaz']

Where the result is reasonably a sequence, a method should return a sequence (but I would agree that it should generally be a sequence of objects of the source type - which I think is what I effectively said: "Isn't something like this potentially useful for all sequences (where the result is a [sequence] of objects that are the same [type] as the source sequence)"

That's not to argue against a generic iterator solution, but the barrier to use of an iterator solution is higher than just calling a method.

Knowing which sequence classes have a "chunk" method and which don't is a higher barrier than knowing that all sequences can be "chunked" by a single imported function. E.

On 3 May 2017 at 02:48, Erik wrote:

...

Anyway, I know you can't stop anyone from *proposing* something like

Hi Paul, On 03/05/17 08:57, Paul Moore wrote: this,

...

but as soon as they do you may decide to quote the recipe from "https://docs.python.org/3/library/functions.html#zip" and try to block their proposition. There are already threads on fora that do that.

That was my sticking point at the time when I implemented a general solution. Why bother to propose something that (although it made my code significantly faster) had already been blocked as being something that should be a python-level operation and not something to be included in a built-in?

It sounds like you have a reasonable response to the suggestion of using zip- that you have a use case where performance matters, and your proposed solution is of value in that case.

I don't think so, though. I had a use-case where splitting an iterable into a sequence of same-sized chunks efficiently improved the performance of my code significantly (processing a LOT of 24-bit, multi-channel - 16 to 32 - PCM streams from a WAV file). Having thought "I need to split this stream by a fixed number of bytes" and then found more_itertools.chunked() (and the zip_longest(*([iter(foo)] * num)) trick) it turned out they were not quick enough so I implemented itertools.chunked() in C. That worked well for me, so when I was done I did a search in case it was worth proposing as an enhancement to feed it back to the community. Then I came across things such as the following: http://bugs.python.org/issue6021 I am specifically referring to the "It has been rejected before" comment, also mentioned here: https://mail.python.org/pipermail/python-dev/2012-July/120885.html See this entire thread, too: https://mail.python.org/pipermail/python-ideas/2012-July/015671.html This is the reason why I really just didn't care enough to go through the process of proposing it in the end (even though the more_itertools.chunked function was one of the first 3 implemented in V1.0 and seems to _still_ be cropping up all the time in different guises - so is perhaps more fundamental than people recognise). The strong implication of the discussions linked to above is that if it had been mentioned before it would be immediately rejected, and that was supported by several members of the community in good standing. So I didn't propose it. I have no idea now what I spent my saved hours doing, but I imagine that it was fun

Whether it's a *sufficient* response remains to be seen, but unless you present the argument we won't know.

Summary: I didn't present the argument because I'm not a masochist Regards, E.

On 04/05/17 01:24, Steven D'Aprano wrote:

On Thu, May 04, 2017 at 12:13:25AM +0100, Erik wrote:

...

I had a use-case where splitting an iterable into a sequence of same-sized chunks efficiently improved the performance of my code [...] So I didn't propose it. I have no idea now what I spent my saved hours doing, but I imagine that it was fun

...

Summary: I didn't present the argument because I'm not a masochist

I'm not sure what the point of that anecdote was, unless it was "I wrote some useful code, and you missed out".

Then you have misunderstood me. Paul suggested that my use-case (chunking could be faster) was perhaps enough to propose that my patch may be considered. I responded with historical/empirical evidence that perhaps that would actually not be the case. I was responding, honestly, to the questions raised by Paul's email.

Your comments come across as a passive-aggressive chastisment of the core devs and the Python-Ideas community for being too quick to reject useful code: we missed out on something good, because you don't have the time or energy to deal with our negativity and knee-jerk rejection of everything good. That's the way your series of posts come across to me.

I apologise if my words or my turn of phrase do not appeal to you. I am trying to be constructive with everything I post. If you choose to interpret my messages in a different way then I'm not sure what I can do about that. Back to the important stuff though:

- you could have offered it to the moreitertools project;

A more efficient version of moreitertools.chunked() is what we're talking about.

- you could have published it on PyPy;

Does PyPy support C extension modules? If so, that's a possibility.

- you could have proposed it on Python-Ideas with an explicit statement

I may well do that - my current patch (because of when I did it) is against a Py2 codebase, but I could port it to Py3. I still have a nagging doubt that I'd be wasting my time though ;)

If you care so little that you can't be bothered even to propose it, why do you care if it is rejected?

You are mistaking not caring enough about the functionality with not caring enough to enter into an argument about including that functionality ... I didn't propose it at the time because of the reasons I mentioned. But when I saw something being discussed yet again that I had a general solution for already written I thought I mention it in case it was useful. As I said, I'm _trying_ to be constructive. E.

Steven D'Aprano wrote:

I've also been thinking about generalisations such as grouping lines into paragraphs, words into lines, etc.

You're probably going to want considerably more complicated algorithms for that kind of thing, though. Let's keep it simple. -- Greg