I don't think it makes sense to add any more ideas to PEP 467. That needed to be a PEP because it proposed breaking backwards compatibility in a couple of areas, and because of the complex history of Python 3's "bytes-as-tuple-of-ints" and Python 2's "bytes-as-str" semantics. Other enhancements to the binary data handling APIs in Python 3 can be considered on their own merits. On 12 October 2016 at 14:08, INADA Naoki wrote:

Memoryview problem =================

To avoid redundant copy of `line = bytes(buf)[:n]`, current solution is using memoryview.

First code I wrote is: `line = bytes(memoryview(buf)[:n])`.

On CPython, it works fine. But `del buff[:n+2]` in next line may fail on other Python implementations. Changing bytearray size is inhibited while memoryview is alive.

So right code is:

with memoryview(buf) as m: line = bytes(m[:n])

The problem of memoryview approach is:

* Overhead: creating temporary memoryview, __enter__, and __exit__. (see below)

* It isn't "one obvious way": Developers including me may forget to use context manager. And since it works on CPython, it's hard to point it out.

To add to the confusion, there's also https://docs.python.org/3/library/stdtypes.html#memoryview.tobytes giving: line = memoryview(buf)[:n].tobytes() However, folks *do* need to learn that many mutable data types will lock themselves against modification while you have a live memory view on them, so it's important to release views promptly and reliably when we don't need them any more.

Quick benchmark:

(temporary bytes) $ python3 -m perf timeit -s 'buf = bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- 'bytes(buf)[:3]' .................... Median +- std dev: 652 ns +- 19 ns

(temporary memoryview without "with" $ python3 -m perf timeit -s 'buf = bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- 'bytes(memoryview(buf)[:3])' .................... Median +- std dev: 886 ns +- 26 ns

(temporary memoryview with "with") $ python3 -m perf timeit -s 'buf = bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- ' with memoryview(buf) as m: bytes(m[:3]) ' .................... Median +- std dev: 1.11 us +- 0.03 us

This is normal though, as memory views trade lower O(N) costs (reduced data copying) for higher O(1) setup costs (creating and managing the view, indirection for data access).

Proposed solution ===============

Adding one more constructor to bytes:

# when length=-1 (default), use until end of *byteslike*. bytes.frombuffer(byteslike, length=-1, offset=0)

With ths API

with memoryview(buf) as m: line = bytes(m[:n])

becomes

line = bytes.frombuffer(buf, n)

Does that need to be a method on the builtin rather than a separate helper function, though? Once you define: def snapshot(buf, length=None, offset=0): with memoryview(buf) as m: return m[offset:length].tobytes() then that can be replaced by a more optimised C implementation without users needing to care about the internal details. That is, getting back to a variant on one of Serhiy's suggestions in the last PEP 467 discussion, it may make sense for us to offer a "buffertools" library that's specifically aimed at supporting efficient buffer manipulation operations that minimise data copying. The pure Python implementations would work entirely through memoryview, but we could also have selected C accelerated operations if that showed a noticeable improvement on asyncio's benchmarks. Regards, Nick. P.S. The length/offset API design is also problematic due to the way it differs from range() & slice(), but I don't think it makes sense to get into that kind of detail before discussing the larger question of adding a new helper module for working efficiently with memory buffers vs further widening the method API for the builtin bytes type -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 12.10.16 08:03, Nathaniel Smith wrote:

On Tue, Oct 11, 2016 at 9:08 PM, INADA Naoki wrote:

...

From Python 3.4, bytearray is good solution for I/O buffer, thanks to #19087 [1]. Actually, asyncio uses bytearray as I/O buffer often.

Whoa what?! This is awesome, I had no idea that bytearray had O(1) deletes at the front. I literally reimplemented this myself on type of bytearray for some 3.5-only code recently because I assumed bytearray had the same asymptotics as list, and AFAICT this is totally undocumented. Shouldn't we write this down somewhere? Maybe here? -> https://docs.python.org/3/library/functions.html#bytearray

I afraid this is CPython implementation detail (like string concatenation optimization). Other implementations can have O(N) deletes at the front of bytearray.

On Wed, Oct 12, 2016 at 2:32 PM, Serhiy Storchaka wrote:

On 12.10.16 07:08, INADA Naoki wrote:

...

Sample code:

def read_line(buf: bytearray) -> bytes: try: n = buf.index(b'\r\n') except ValueError: return b''

line = bytes(buf)[:n] # bytearray -> bytes -> bytes

Wouldn't be more correct to write this as bytes(buf[:n])?

Yes, you're right! I shouldn't copy whole data only for cast from bytearray to byte.

...

Adding one more constructor to bytes:

# when length=-1 (default), use until end of *byteslike*. bytes.frombuffer(byteslike, length=-1, offset=0)

This interface looks unusual. Would not be better to support the interface of buffer in Python 2: buffer(object [, offset[, size]])?

It looks better. (Actually speaking, I love deprecated old buffer for simplicity. memoryview supports non bytes-like complex data types.) Thanks, -- INADA Naoki

On Oct 11 2016, Nathaniel Smith wrote:

On Tue, Oct 11, 2016 at 9:08 PM, INADA Naoki wrote:

...

From Python 3.4, bytearray is good solution for I/O buffer, thanks to #19087 [1]. Actually, asyncio uses bytearray as I/O buffer often.

Whoa what?! This is awesome, I had no idea that bytearray had O(1) deletes at the front. I literally reimplemented this myself on type of bytearray for some 3.5-only code recently because I assumed bytearray had the same asymptotics as list, and AFAICT this is totally undocumented.

Indeed, same here. Best, -Nikolaus -- GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F »Time flies like an arrow, fruit flies like a Banana.«

Oops, right, I wanted to write "I don't think that adding a single method deserves its own PEP." Victor 2016-10-12 18:37 GMT+02:00 Stephen J. Turnbull :

Victor Stinner writes:

...

2016-10-12 11:34 GMT+02:00 INADA Naoki :

...

...

I see. My proposal should be another PEP (if PEP is required).

I don't think that adding a single method deserves its own method.

You mean "deserves own PEP", right? I interpreted Nick to say that "the reasons that applied to PEP 367 don't apply here, so you can Just Do It" (subject to the usual criteria for review, but omit the PEP).

I'm not sure whether he was channeling Guido or that should be qualified with an IMO or IMHO.

On 13 October 2016 at 02:37, Stephen J. Turnbull wrote:

Victor Stinner writes:

...

2016-10-12 11:34 GMT+02:00 INADA Naoki :

...

...

I see. My proposal should be another PEP (if PEP is required).

I don't think that adding a single method deserves its own method.

You mean "deserves own PEP", right? I interpreted Nick to say that "the reasons that applied to PEP 367 don't apply here, so you can Just Do It" (subject to the usual criteria for review, but omit the PEP).

Sort of. Adding this to PEP 467 doesn't make sense (as it's not related to easing migration from Python 2 or addressing the mutable->immutable design legacy), but I don't have an opinion yet on whether this should be a PEP or not - that really depends on whether we tackle it as an implementation detail of asyncio, or as a public API in its own right. Method proliferation on builtins is a Big Deal(TM), and efficient buffer management for IO protocol development is a relatively arcane speciality (as well as one where there are dedicated OS level capabilities we may want to exploit some day), which is why I think a dedicated helper module is likely a better way to go. For example: - add `asyncio._iobuffers` as a pure Python memoryview based implementation of the desired buffer management semantics - add `_iobuffers` as an optional asyncio independent accelerator module for `asyncio._iobuffers` If that works out satisfactorily, *then* consider a PEP to either make `iobuffers` a public module in its own right (ala the `selectors` module from the original asyncio implementation), or to expose some of its features directly via the builtin binary data types. The logical leap I strongly disagree with is going straight from "asyncio needs some better low level IO buffer manipulation primitives" to "we should turn the builtin types into low level IO buffer manipulation primitives that are sufficient for asyncio's needs". The notion of "we shouldn't need to define our own domain specific helper libraries" isn't a given for standard library modules any more than it is for 3rd party ones. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 13 October 2016 at 12:54, Nick Coghlan wrote:

Method proliferation on builtins is a Big Deal(TM)

I wanted to quantify this concept, so here's a quick metric that helps convey how every time we add a new builtin method we're immediately making Python harder to comprehend: >>> def get_builtin_types(): ... import builtins ... return {name:obj for name, obj in vars(builtins).items() if isinstance(obj, type) and not (name.startswith("__") or issubclass(obj, BaseException))} ... >>> len(get_builtin_types()) 26 >>> def get_builtin_methods(): ... return [(name, method_name) for name, obj in get_builtin_types().items() for method_name, method in vars(obj).items() if not method_name.startswith("__")] ... >>> len(get_builtin_methods()) 230 Putting special purpose functionality behind an import gate helps to provide a more explicit context of use (in this case, IO buffer manipulation) vs the relatively domain independent namespace that is the builtins. Cheers, Nick. P.S. Since I was poking around in the builtins anyway, here are some other simple language complexity metrics: >>> len(vars(builtins)) 151 >>> def get_interpreter_builtins(): ... import builtins ... return {name:obj for name, obj in vars(builtins).items() if name.startswith("__")} ... >>> len(get_interpreter_builtins()) 8 >>> def get_builtin_exceptions(): ... import builtins ... return {name:obj for name, obj in vars(builtins).items() if isinstance(obj, type) and issubclass(obj, BaseException)} ... >>> len(get_builtin_exceptions()) 65 >>> def get_builtin_functions(): ... import builtins ... return {name:obj for name, obj in vars(builtins).items() if isinstance(obj, type(repr))} ... >>> len(get_builtin_functions()) 42 >>> def get_other_builtins(): ... import builtins ... return {name:obj for name, obj in vars(builtins).items() if not name.startswith("__") and not isinstance(obj, (type, type(repr)))} ... >>> len(get_other_builtins()) 12 The "other" builtins are the builtin constants (None, True, False, Ellipsis, NotImplemented) and various artifacts from doing this at the interactive prompt (license, credits, copyright, quit, exit, help, "_") -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 19 October 2016 at 01:28, Chris Barker - NOAA Federal wrote:

...

...

...

...

def get_builtin_methods(): ... return [(name, method_name) for name, obj in get_builtin_types().items() for method_name, method in vars(obj).items() if not method_name.startswith("__")] ... len(get_builtin_methods()) 230

So what? No one looks in all the methods of builtins at once.

Yes, Python implementation developers do, which is why it's a useful part of defining the overall "size" of Python and how that is growing over time. When we define a new standard library module (particularly pure Python ones) rather than new methods on builtin types, we create substantially less additional work for other implementations, and we make it easier for educators to decide whether or not they should be introducing their students to the new capabilities. That latter aspect is important, as providing functionality as separate modules means we also gain an enhanced ability to explain "What is this *for*?", which is something we regularly struggle with when making changes to the core language to better support relatively advanced domain specific use cases (see http://learning-python.com/books/python-changes-2014-plus.html for one generalist author's perspective on the vast gulf that can arise between "What professional programmers want" and "What's relevant to new programmers")

If we have anything like an OO System (and python builtins only sort of do...) then folks look for a built in that they need, and only then look at its methods.

If you need to work with bytes, you'll look at the bytes object and bytarray object. Having to go find some helper function module to know to efficiently do something with bytes is VERY non-discoverable!

Which is more comprehensible and discoverable, dict.setdefault(), or collections.defaultdict()? Micro-optimisations like dict.setdefault() typically don't make sense in isolation - they only make sense in the context of a particular pattern of thought. Now, one approach to such patterns is to say "We just need to do a better job of teaching people to recognise and use the pattern!". This approach tends not to work very well - you're often better off extracting the entire pattern out to a higher level construct, giving that construct a name, and teaching that, and letting people worry about how it works internally later. (For a slightly different example, consider the rationale for adding the `secrets` module, even though it's mostly just a collection of relatively thin wrappers around `os.urandom()`)

bytes and bytarray are already low-level objects -- adding low-level functionality to them makes perfect sense.

They're not really that low level. They're *relatively* low level (especially for Python), but they're still a long way away from the kind of raw control over memory layout that a language like C or Rust can give you.

And no, this is not just for asycio at all -- it's potentially useful for any byte manipulation.

Yes, which is why I think the end goal should be a public `iobuffers` module in the standard library. Doing IO buffer manipulation efficiently is a complex topic, but it's also one where there are: - many repeatable patterns for managing IO buffers efficiently that aren't necessarily applicable to manipulating arbitrary binary data (ring buffers, ropes, etc) - many operating system level utilities available to make it even more efficient that we currently don't use (since we only have general purpose "bytes" and "bytearray" objects with no "iobuffer" specific abstraction that could take advantage of those use case specific features)

+1 on a frombuffer() method.

Still -1 in the absence of evidence that a good IO buffer abstraction for asyncio and the standard library can't be written without it (where the evidence I'll accept is "We already wrote the abstraction layer, and not having this builtin feature necessarily introduces inefficiencies or a lack of portability beyond CPython into our implementation").

...

Putting special purpose functionality behind an import gate helps to provide a more explicit context of use

This is a fine argument for putting bytearray in a separate module -- but that ship has sailed. The method to construct a bytearray from a buffer belongs with the bytearray object.

The bytearray constructor already accepts arbitrary bytes-like objects. What this proposal is about is a way to *more efficiently* snapshot a slice of a bytearray object for use in asyncio buffer manipulation in cases where all of the following constraints apply: - we don't want to copy the data twice - we don't want to let a memoryview be cleaned up lazily - we don't want to incur the readability penalty of explicitly managing the memoryview For a great many use cases, we simply don't care about those constraints (especially the last one), so adding `bytes.frombuffer` is just confusing: we can readily predict that after adding it, a future Stack Overflow question will be "When should I use bytes.frombuffer() in Python instead of the normal bytes constructor?" By contrast, if we instead say "We want Python to natively support efficient. readily discoverable, IO buffer manipulation", then folks can ask "What's preventing us from providing an `iobuffers` module today?" and start working towards that end goal (just as the"selectors" module was added as an asyncio-independent abstraction layer over select, epoll and kqueue, but probably wouldn't have been without the asyncio use case to drive its design and implementation as a standard library module) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 22 October 2016 at 07:57, Chris Barker wrote:

I'm still confused about the "io" in "iobuffers" -- I've used buffers a lot -- for passing data around between various C libs -- numpy, image processing, etc... I never really thought of it as IO though. which is why a simple frombuffer() seems to make a lot of sense to me, without any other stuff. (to be honest, I reach for Cyton these days for that sort of thing though)

From that perspective, adding "[bytes/bytearray].frombuffer" is adding complexity to the core language for the sake of giving people one small additional piece of incremental performance improvement that

That's the essence of my point though: if you care enough about the performance of a piece of code for the hidden copy in "bytes(mydata[start:stop])" to be deemed unacceptable, and also can't afford the lazy cleanup of the view in "bytes(memoryview(mydata)[start:stop])", then it seems likely that you're writing specialist, high performance, low overhead, data manipulation code, that probably shouldn't be written in Python In such cases, an extension module written in something like Cython, C or Rust would be a better fit, as using the more appropriate tool will give you a range of additional performance improvements (near) automatically, such as getting to avoid the runtime overhead of Python's dynamic type system. At that point, having to write the lowest-available-overhead version explicitly in Python as: with memoryview(mydata) as view: return bytes(mydata[start:stop] is a sign that someone is insisting on staying in pure Python code when they're do sufficiently low level bit bashing that it probably isn't the best idea to continue down that path. they can eke out before they admit to themselves "OK, I'm probably not using the right language for this part of my application". By contrast, a library that provided better low level data buffer manipulation that was suitable for asyncio's needs is *much* easier to emulate on older versions, and provides more scope for extracting efficient data manipulation patterns beyond this one very specific case of more efficiently snapshotting a subset of an existing buffer. Cheers, Nick. P.S. I bring up Rust and the runtime overhead of the type system specifically here, as Armin Ronacher recently wrote an excellent post about that in relation to some performance improvement work they were doing at Sentry: https://blog.sentry.io/2016/10/19/fixing-python-performance-with-rust.html -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 25 October 2016 at 02:53, Chris Barker wrote:

IS this na either-or? IF someone is proposing a nice lib for "low level data buffer manipulation", then yes, putting frombuffer() in there would be a fine idea.

But if there is no such proposal on the table, then I think adding a frombuffer method to the bytes object is a small improvement that we can do now.

The suggestion came from folks working on asyncio performance improvements, and we already got the entire ``selectors`` abstraction from the original asyncio implementation work, as well as Yury's new libuv-based ``uvloop`` asyncio event loop implementation. Given that "make OpenStack/SDN/NFV run faster" is a key point of interest for folks like Intel and Red Hat, I'm *absolutely* suggesting that they put some paid time and energy into a lower level buffer manipulation library between now and the Python 3.7 feature freeze in 12+ months, as I think that will have longer term pay-offs well beyond the scope of the original use cases :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Tue, Oct 25, 2016 at 12:25 AM, Nick Coghlan wrote:

The suggestion came from folks working on asyncio performance improvements,

I'm *absolutely* suggesting that they put some paid time and energy into a lower level buffer manipulation library between now and the Python 3.7 feature freeze in 12+ months, as I think that will have longer term pay-offs well beyond the scope of the original use cases :)

Sounds good -- let's hope something comes of that. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov