Mailman 3 July 2013 - Python-ideas

Remove tty module
by anatoly techtonik Aug. 2, 2013

Aug. 2, 2013

http://hg.python.org/cpython/file/74fd1959cf44/Lib/tty.py tty module is cryptic. The low level interface it is exposed can be covered by appropriate recipes in termios module documentation. You can't understand tty without understanding termios. You also can't understand tty without being a Unix guru - http://en.wikipedia.org/wiki/TTY I doubt that raw mode function (which is 50% of this stuff) is really used by anyone. In ideal world the tty should be substituted with interface with less cryptic terminology replacing cbreak mode, cooked mode, raw mode with user oriented concepts. Otherwise this functionality is already covered by termios interface. -- anatoly t.

6 15

Support Unicode code point notation
by Steven D'Aprano Aug. 2, 2013

Aug. 2, 2013

Unicode's standard notation for code points is U+ followed by a 4, 5 or 6 hex digit string, such as π = U+03C0. This notation is found throughout the Unicode Consortium's website, e.g.: http://www.unicode.org/versions/corrigendum2.html as well as in third party sites that have reason to discuss Unicode code points, e.g.: https://en.wikipedia.org/wiki/Eth#Computer_input I propose that Python strings support this as the preferred escape notation for Unicode code points: '\U+03C0' => 'π' The existing \U and \u variants must be kept for backwards compatibility, but should be (mildly) discouraged in new code. Doesn't this violate "Only One Way To Do It"? --------------------------------------------- That's not what the Zen says. The Zen says there should be One Obvious Way to do it, not Only One. It is my hope that we can agree that the One Obvious Way to refer to a Unicode character by its code point is by using the same notation that the Unicode Consortium uses: d <=> U+0064 and leave legacy escape sequences as the not-so-obvious ways to do it: \x64 \144 \u0064 \U00000064 Why do we need yet another way of writing escape sequences? ----------------------------------------------------------- We don't need another one, we need a better one. U+xxxx is the standard Unicode notation, while existing Python escapes have various problems. One-byte hex and oct escapes are a throwback to the old one-byte ASCII days, and reflect an obsolete idea of strings being equivalent to bytes. Backwards compatibility requires that we continue to support them, but they shouldn't be encouraged in strings. Two-byte \u escapes are harmless, so long as you imagine that Unicode is a 16-bit character set. Unfortunately, it is not. \u does not support code points in the Supplementary Multilingual Planes (those with ordinal value greater than 0xFFFF), and can silently give the wrong result if you make a mistake in counting digits: # I want EGYPTIAN HIEROGLYPH D010 (Eye of Horus) s = '\u13080' => oops, I get 'ገ0' (ETHIOPIC SYLLABLE GA, ZERO) Four-byte \U escape sequences support the entire Unicode character set, but they are terribly verbose, and the first three digits are *always* zero. Python doesn't (and shouldn't) support \U escapes beyond 10FFFF, so the first three digits of the eight digit hex value are pointless. What is the U+ escape specification? ------------------------------------ http://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-l… lists the escape sequences, including: \uxxxx Character with 16-bit hex value xxxx \Uxxxxxxxx Character with 32-bit hex value xxxxxxxx To this should be added: \U+xxxx Character at code point xxxx (hex) with the note: Exactly 4, 5 or 6 hexadecimal digits are required. Upper or lower case? -------------------- Uppercase should be preferred, as the Unicode Consortium uses it, but both should be accepted. Variable number of digits? Isn't that a bad thing? -------------------------------------------------- It's neither good nor bad. Octal escapes already support from 1 to 3 oct digits. In some languages (but not Python), hex escapes support from 1 to an unlimited number of hex digits. Is this backwards compatible? ----------------------------- I believe it is. As of Python 3.3, strings using \U+ give a syntax error: py> '\U+13080' File "<stdin>", line 1 SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in position 0-7: end of string in escape sequence What deprecation schedule are you proposing? -------------------------------------------- I'm not. At least, the existing features should not be considered for removal before Python 4000. In the meantime, the U+ form should be noted as the preferred way, and perhaps blessed in PEP 8. Should string reprs use the U+ form? ------------------------------------ \u escapes are sometimes used in string reprs, e.g. for private-use characters: py> chr(0xE034) '\ue034' Should this change to '\U+E034'? My personal preference is that it should, but I fear backwards compatibility may prevent it. Even if the exact form of str.__repr__ is not guaranteed, changing the repr would break (e.g.) some doctests. This proposal defers any discussion of changing the repr of strings to use U+ escapes. -- Steven

15 54

Python stdlib for hackers
by anatoly techtonik July 30, 2013

July 30, 2013

stdlib.py check https://code.google.com/p/rainforce/wiki/PythonLibrarySplit -- anatoly t.

1 0

shlex escapes without Posix mode
by Ryan July 27, 2013

July 27, 2013

Note: This is my first post to the mailing list, so I'm not sure if I'm doing something wrong or something. I've been playing around with shlex.lately, and I mostly like it, but I have an idea. Have an option with the ability to enable certain Posix mode features selectively, most particularly character escapes. It could be something like, if Posix mode is disabled, the string of escape characters is set to empty or None, and assigning a value to it enables that feature in non-Posix mode. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity.

3 6

sys.args
by anatoly techtonik July 27, 2013

July 27, 2013

sys.argv is an atavism with not intuitive name, how about sys.args for program arguments only? if not sys.args: print("usage: ...") -- anatoly t.

11 14

namedlist() or Record type
by anatoly techtonik July 26, 2013

July 26, 2013

Rationale: There are many 2D data with table layout where order is important and it is hard to work with them in Python. The use case: 1. Get data row from a table 2. Change row column by name 3. Save data back For example, termios.tcgetattr() returns list [iflag, oflag, cflag, lflag, ispeed, ospeed, cc]. I need to modify some bits in lflag value and save this list back. In my code it looks like: newattr[3] &= ~termios.ICANON I want to get rid of magic number 3 without defining additional variables just for names. I thought that namedtuple can help here, but it is read-only and seems not serializeable. What I want is: >>> data = [1, 'john', 23434] >>> named = Record(data, ['id', 'name', 'number']) >>> named.id = 3 >>> named.name 'john' >>> named [3, 'john', 23434] >>> named.naem AttributeError(...) >>> named.dict() {'id': 3, 'name': john, 'number': 23434} >>> named.json() ... I hacked OrderedDict to accept list as param and allow attribute access. It doesn't behave as named list - you still need to call .values() to get list back, and indexed access doesn't work as well. http://bugs.python.org/file31026/DictRecord.py -- anatoly t.

6 8

Arbitrary constants in ASTs
by Jonathan Brandvein July 24, 2013

July 24, 2013

In the deprecated (removed in Python 3) compiler library, it was possible to construct a "Const" AST node. This node took in an arbitrary Python object, and got compiled to a LOAD_CONST opcode. This functionality is not available with the modern ast library. I wonder if there are any objections to adding it. Embedding objects in the AST can be handy. In a program transformation, you can avoid having to create code to define and assign to a new variable name. In code that is meant for exec or eval, you don't need to add the object to the global or local namespaces that you pass in. You also avoid the runtime overhead of a global namespace lookup, although your semantics are slightly different in that the object cannot be replaced at runtime. For a program transformation that introduces its own fixed objects that are not meant to be hot-swapped or monkey patched, this is a good tradeoff. I bring this up because I just ported a project from Python 2.4 to 3.3, and this was one feature where there was no good modern substitute available. The one drawback I can think of is that there is no general canonical source representation for an arbitrary constant. I will note that there are already ASTs that don't have a unique source representation: think of two ast.Name nodes that differ in their Store/Load context. Are there any limitations in CPython concerning what values can be placed in co_consts and accessed by LOAD_CONST? Jon

5 5

Another attempt at a sum() alternative: the concatenation protocol
by Nick Coghlan July 23, 2013

July 23, 2013

I haven't been following the sum() threads fully, but something Ron suggested gave me an idea for a concatenation API and protocol. I think we may also be able to use a keyword-only argument to solve the old string.join vs str.join problem in a more intuitive way. def concat(start, iterable, *, interleave=None): try: build = start.__concat__ except AttributeError: result = start if interleave is None: for x in iterable: result += x else: for x in iterable: result += interleave result += x else: result = build(iterable, interleave=interleave) If implementing this as a third party API you'd use a tool like functools.singledispatch (which has a backport available on PyPI) rather than defining a new protocol. Registering implementations for the immutable builtin types like str, bytes and tuple would then allow those to be handled efficiently, just as if they provided appropriate __concat__ implementations. A simple "use sum for numbers, concat for containers" approach is simpler and clearer than trying to coerce sum into being fast for both when its assumptions are thoroughly grounded in manipulating numbers rather than containers. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

13 25

list.copy() for consistency with dict.copy()
by anatoly techtonik July 23, 2013

July 23, 2013

Almost everybody knows that you need to copy structures in Python if you need to fork and modify them without consequences. >>> x = [1,2,3] >>> d = x >>> d[1] = 3 >>> x [1, 3, 3] For dict() the fastest way is to use dict.copy() method - http://stackoverflow.com/questions/5861498/fast-way-to-copy-dictionary-in-p… For list() it appears that the fastest is to use non-obvious slicing operator: http://stackoverflow.com/questions/2612802/how-to-clone-a-list-in-python The idea is to add list.copy() and make it as fast as slicing. -- anatoly t.

2 2

Adding __getter__ to compliment __iter__.
by Ron Adam July 23, 2013

July 23, 2013

[May be a duplicate... access to news server acting up.] On 07/16/2013 08:01 AM, Nick Coghlan wrote: > On 16 July 2013 22:28, Joshua Landau <joshua(a)landau.ws> wrote: >> On 16 July 2013 11:21, Oscar Benjamin <oscar.j.benjamin(a)gmail.com> wrote: >>> On 16 July 2013 07:50, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >>> >>> If people are using sum() to concatenate lists then this should be >>> taken not as evidence that a new solution needs to be found but as >>> evidence that chain is not sufficiently well-known. The obvious >>> solution to that is not to implement a new protocol but to make the >>> existing solution more well known i.e. move chain.from_iterable to >>> builtins and rename it (the obvious choice being concat). >> >> You could wait for PEP 448, which will let you use [*sublist for >> sublist in list_to_be_flattened]. > > Ah, true, I forgot about that. Too many interesting things going on > for me to keep track of everything :) > > In effect, PEP 448 goes further than making chain a builtin: it gives > it syntax! With PEP 448, the generator expression: > > (*itr for itr in iterables) > > would be equivalent to either of the current: > > itertools.chain(*iterables) > itertools.chain.from_iterable(iterables) > > That's pretty cool. It also means I can go back to happily ignoring > the sum threads :) > > Cheers, > Nick. > > P.S. Something about this should probably be added to the rationale > section of PEP 448 I played around with trying to find something that would work like the example Nick put up and found out that the different python types are not similar enough in how they do things to make a function that takes a method or other operator work well. What happens is you either end up with widely varying results depending on how the methods are implemented on each type, or an error because only a few methods are very common on all types. Mostly introspection methods. I believe this to be stronger underlying reason why functions like reduce and map were removed. And it's also a good reason not to recommend functions like sum() for things other than numbers. To use functions similar to that, you really have to think about what will happen in each case because the gears of the functions and methods are not visible in the same way a comprehension or generator expression is. It's too late to change how a lot of those methods work and I'm not sure it will still work very well. One of the most consistent protocols python has is the iterator and generator protocols. The reason they work so well is that they need to interface with for-loops and nearly all containers support that. examples... >>> a = [1,2,3] >>> iter(a) <list_iterator object at 0x7f3bc9306e90> >>> b = (1,2,3) >>> iter(b) <tuple_iterator object at 0x7f3bc9306f10> >>> c = {1:2, 3:4} >>> iter(c) <dict_keyiterator object at 0x7f3bc93cf6d8> >>> d = {1, 2, 3} >>> iter(d) <set_iterator object at 0x7f3bc9380500> >>> e = "123" >>> iter(e) <str_iterator object at 0x7f3bc9306e90> And is why chain is the recommended method of joining multiple containers. This really only addresses getting stuff OUT of containers. PEP 448's * unpacking in comprehensions helps with the problem of putting things into containers. But that isn't the PEP's main point. What I'm thinking of is the inverse operation of an iter. Lets call it a "getter". You would get a getter the same way you get an iter. g = getter(obj) But instead of __next__ or send() methods, it would have an iter_send(), or isend() method. The isend method would takes an iter object, or an object that iter() can be called on. The getter would return either the object it came from, or a new object depending on weather or not it was created from a mutable or immutable obj. Mutable objects... g = getter(A) # A needs a __getter__ method. A = g.isend(B) A += B # extend Mutable objects... g = getter(A) C = g.isend(B) C = A + B # join The point, is to have something that works on many types and is as consistent in how it's defined as the iter protocol. Having a strict and clear definition is a very important! The internal implementation of a getter could do a direct copy to make it faster, like slicing does, but that would be a private implementation detail. They don't replace generator expressions or comprehensions. Those generally will do something with each item. Functions like extend() and concat() could be implemented with *getter-iters*, and work with a larger variety of objects with much less work and special handling. def extend(A, B): return getter(A).isend(B) def concat(A, B): """ Extend A with multiple containers from B. """ g = getter(A) if g.isend() is not A: raise TypeError("can't concat immutable arg, use merge()") for x in B: g.isend(x) return A def merge(A, B): """ Combine A with containers in B, return new container. """ a = list(A) g = getter(a) for x in B: g.isend(x) return type(A)(a) Expecting many holes to be punched in this idea ... But hope not too many. ;-) Ron

12 35