Mailman 3 June 2013 - Python-ideas

Add "namereplace" error handler
by Serhiy Storchaka June 12, 2013

June 12, 2013

I propose to add "namereplace" error handler which is similar to "backslashreplace" error handler but use \N escapes instead of \x/\u/\U escapes. >>> '−1'.encode('ascii', 'backslashreplace') b'\\u22121' >>> '−1'.encode('ascii', 'namereplace') b'\\N{MINUS SIGN}1' In some cases such representation is more readable. What are you think about this? Are there suggestions for better name?

10 17

Python 3 open() text files: make encoding parameter optional for cross-platform scripts
by anatoly techtonik June 11, 2013

June 11, 2013

Without reading subject of this letter, what is your idea about which encoding Python 3 uses with open() calls on a text file? Please write in reply and then scroll down. Without cheating my opinion was cp1252 (latin-1), because it was the way Python 2 assumed all text files are. Or Python 2 uses ISO-8859-1? But it appeared to be different way - http://docs.python.org/3/library/functions.html#open. No, it appeared here - https://bitbucket.org/techtonik/hexdump/pull-request/1/ and after a small lecture I realized how things are bad. open() in Python uses system encoding to read files by default. So, if Python script writes text file with some Cyrillic character on my Russian Windows, another Python script on English Windows or Greek Windows will not be able to read it. This is just what happened. The solution proposed is to specify encoding explicitly. That means I have to know it. Luckily, in this case the text file is my .py where I knew the encoding beforehand. In real world you can never know the encoding beforehand. So, what should Python do if it doesn't know the encoding of text file it opens: 1. Assume that encoding of text file is the encoding of your operating system 2. Assume that encoding of text file is ASCII 3. Assume that encoding of text file is UTF-8 Please write in reply and then scroll down. I propose three, because ASCII is a binary compatible subset of UTF-8. Choice one is the current behaviour, and it is very bad. Troubleshooting this issue, which should be very common, requires a lot of prior knowledge about encodings and awareness of difference system defaults. For cross-platform work with text files this fact implicitly requires you to always use 'encoding' parameter for open(). Is it enough for a PEP? This stuff is rather critical IMO, so even if it will be rejected there will be a documented design decision. -- anatoly t.

15 38

Hooks into super()'s __getattribute__
by Ronald Oussoren June 11, 2013

June 11, 2013

Hi, Super() currently does not have a way to hook into the behavior of attribute lookup, the __getattribute__ method of super peeks in the __dict__ of types along the MRO until it finds what it is looking for. That can be a problem when a class implements __getattribute__ and doesn't necessarily store (all) attributes in the type __dict__. PyObjC is an example where the current behavior causes problems: PyObjC defines proxy classes for classes in the Objective-C runtime. The __dict__ of those classes is filled on demand, whenever a method is called that isn't in the __dict__ yet PyObjC looks in the Objective-C runtime datastructures for the method and adds it to __dict__. This works fine for normal method resolution, but can fail with super: super will only return the correct value when the superclass method happens to be in __dict__ already. My current solution for this is a custom subclass of super that must be used with Cocoa subclasses, and that's something I'd like to get rid off. I'd therefore like to propose adding a slot to PyTypeObject that is called by super's __getattribute__ when present, instead of peeking in the type's tp_dict slot. Does this look like a sane solution? I've filed an issue about this that includes a proof of concept patch: http://bugs.python.org/issue18181. That patch is incomplete, but does make it possible to use the builtin super for Cocoa classes (with a suitably patched version of PyObjC). An earlier issue mentions that the current behavior of super can be inconsistent with the behavior of __getattribute__, see http://bugs.python.org/issue783528 (that issue is closed, but IMHO for the wrong reason). I'm appearently not the only one running into this problem ;-) Ronald

1 0

Mixed script numbers. Was: Unicode minus sign in numeric conversions
by Alexander Belopolsky June 11, 2013

June 11, 2013

I am changing the subject because the issue of mixing digits from different scripts is quite different from the issue of accepting MINUS SIGN. I left a comment on the original subject at the issue tracker: < http://bugs.python.org/issue6632#msg190881>. On Mon, Jun 10, 2013 at 8:53 PM, Steven D'Aprano <steve(a)pearwood.info> wrote: > > On 10/06/13 05:35, Alexander Belopolsky wrote: >> >> ... Consider this case: >> >>>>> float('123٠95') >> 123095.0 >> >> Depending on your font, '123٠95' may be indistinguishable from '123.95'. > > > Indistinguishable *by eye* maybe, but the same applies to ASCII, > 365lO98, and there are plenty of ways to distinguish them other > than by a careless glance at the screen. > > [Aside: I have seen users type I or O for digits, based on the fact > that it works fine when using a typewriter, and I've read books from > the 1970s that recommended that number parsers accept I, L and O > for just that reason.] I am not sure why your example is relevant. There is little harm from accepting O for zero, but accepting it for say 8 would be a different story. > > It would be a pretty awful font that made ٠ look like . But even if it did, what is the concern here? > If somebody enters a mixed script number, presumably they have some reason for it. Sure, say someone who wants to sell you a $23.95 gadget for $23,095. :-)

1 0

namedtuple is not as good as it should be
by Christian Tismer June 11, 2013

June 11, 2013

Sorry Raymond if this offends you. But after some extensive use of namedtuple I think it needs a re-design. The pros: _________ - a namedtuple can be easily used in place of a tuple. - it provides names for its fields and behaves like a tuple, otherwise. Furthermorre: - a named tuple is the natural choice for modelling the records of a database. The cons: - All of the above becomes wrong if you use namedtuple as a real replacement for tuple. - especially for databases it makes little sense as it is. Reason: Pickling ________ - pickling of tuples: always possible if it contains built-in types. tuples are simply tuples. There is just _one_ type. - pickling of namedtuple: sometimes possible, if your definition is static enough. namedtuple has a subtype per tuple layout, and you need to cope with that. Just to be clear about that: Sure, it is possible to pickle named tuples, but you have to think about it! And to have to thing about it trashes a lot of the fun of having those names for free. And typically this happens after you did your analysis of 20 GB of data: You cannot pickle your nicely formatted namedtuple instances after the fact. Actually, to save all the computation, you do a hack that turns all your alive namedtuple instances back into ordinary tuples. Silent implications introduced by namedtuple: _____________________________________________ Without being very explicit, namedtuple makes you use it happily instead of tuples. But instead of using a native, simple type, you now use a not-so-simple, user-defined type. This type - is not built in - has a class definition - needs a global, constant definition _somewhrere_ Typically, you run a script interactively, and typically you need to pickle some __main__.somename namedtuple class instances. This is exactly what you don't want! You want to have some anonymous data in the pickle and don't want to make anything fixed in stone. namedtuple() Factory Function for Tuples with Named Fields __________________________________________________________ It would be great if namedtuple were just this, as the doku says. But instead, it - creates a named class, i.e. forces me to name it - you create instances of that specific class and not just tuple. Usability for databases ______________________ For simple databases which enumerate (employee, salary, ...) or (shoesize, height, married) as example "database"s, namedtuple is ok. As soon as you write a real database implementation with no fixed layout, you get into trouble. Easy database approach: You define a dbtable as a collection of tuples, maybe as a dict with fast index keys. Not a problem with tuples, which are of type tuple. With named tuple, you suddenly see yourself creating namedtuple instead. But those namedtuple records cannot be pickled when used as a replacement of regular tuples, because they now have a dynamically created type, and extra actions are necessary to make it possible to pickle those. From the documentation: EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, title, department, paygrade') This implicitly suggests that namedtuple is the tool of choice to support databases in Python. Wrong! It supports simple, fixed data structures in Python. For instance, you can use it to define a fixed structural record to define the general description of a database field and its attributes. For a database itself instead, this is very wrong. Nobody uses a fixed class definition to define the structure of some database tables. Instead, you use a generic approach with a row class that describes what a row is, but dynamically. So why all this rant? ___________________ What I'm trying to explain is that namedtuple should be closer to tuple! A namedtuple should not be an explicit class with instances, but a generic subclass of tuple, for all namedtuples. Then, if the user decides to use a namedtuple to build his own class upon that, fine. He then might want to do everything needed to support and pickle his class. But the namedtuple should be a singe (maybe builtin) class that is just a tuple with field names, nothing more. Implementation idea (roughly) _____________________________ Whatever a namedtuple does, it should behave as closely as possible to a tuple, just providing attribute names. Pickling support should be so that the user does not need to know that a namedtuple has a special class. Actually, there should be only a generic class, and the namedtuple "class" is a template instance that just holds the names. Those names could go into some registry or whatever. The only interesting thing about a namedtuple is the set of names used. This set of names is not eligible to enforce the whole import machinery, the associated problems etc. The set of attribute names defines the namedtuple, and that's it. If it is necessary to have class instances like today, ok. But there is no need to search that class in a pickle! Instead, the defining set of attributes could be pickled (uniquely stored by tuple comparison), and the class could be re-created on-the-fly at unpickling time. Conclusion ___________ I love namedtuple, and I hate it. I want to get rid of the second half of this sentence. Let us invent one that does not enforce class behavior. I am thinking of a prototype... cheers - chris p.s.: there is a lot about database design not mentioned here. -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

4 9

Line continuations with comments
by Ron Adam June 10, 2013

June 10, 2013

There were a few people who liked the idea of having comments after a line continuation. I was able to make a small patch that removed the some of the restrictions on the '\' for testing some ideas which does the following. * Allow a line to continue on the same line. * Skips comments before checking for the new line after a back slash. Here are some examples... These are technically the same. >>> 'aaa' \ ... 'bbb' \ ... 'ccc' 'aaabbbccc' >>> 'aaa' \ 'bbb' \ 'ccc' 'aaabbbccc' Yes there is't much need for this, but I wanted to see if it would work and if the test suit passes. It does. ;-) You can put a comment after a line continuation. >>> 'aaa' \# one ... 'bbb' \# two ... 'ccc' # three 'aaabbbccc' Works with expressions too. >>> result = \ ... + 111 \# A ... + 222 \# B ... + 333 \# C ... + 444 # D >>> result 1110 But if it has a space between the \ and the #, the line is continued on the same line instead of the following line. >>> 'aaa' \ #comment 'aaa' The reason \# works, but not \ #, is when the comment comes directly after the back slash, it's removed and leaves a (backslash + new-line) pair. Removing the white space before the new line check caused some errors in the test suite. I haven't figured out why yet. So this doesn't do that for now. Currently you get this if you try any of these examples. >>> 'abc' \#comment File "<stdin>", line 1 'abc' \#comment ^ SyntaxError: unexpected character after line continuation character Only one of pythons tests fail, and I don't think it's related. test test_urllib2_localnet failed See the diff below if you want to play with it. It's not big. Cheers, Ron diff -r 155e6fb309f5 Parser/tokenizer.c --- a/Parser/tokenizer.c Tue May 21 21:02:04 2013 +0200 +++ b/Parser/tokenizer.c Tue May 21 22:10:31 2013 -0500 @@ -1391,18 +1391,31 @@ again: tok->start = NULL; + + c = tok_nextc(tok); + + /* Check if continuing line */ + if (tok->cont_line == 1 && c == '\n') { + tok->cont_line = 0; + c = tok_nextc(tok); + } + /* Skip spaces */ - do { + while (c == ' ' || c == '\t' || c == '\014') { c = tok_nextc(tok); - } while (c == ' ' || c == '\t' || c == '\014'); + tok->cont_line = 0; + } /* Set start of current token */ tok->start = tok->cur - 1; /* Skip comment */ - if (c == '#') + if (c == '#') { while (c != EOF && c != '\n') c = tok_nextc(tok); + tok_backup(tok, c); + goto again; + } /* Check for EOF and errors now */ if (c == EOF) { @@ -1641,12 +1654,6 @@ /* Line continuation */ if (c == '\\') { - c = tok_nextc(tok); - if (c != '\n') { - tok->done = E_LINECONT; - tok->cur = tok->inp; - return ERRORTOKEN; - } tok->cont_line = 1; goto again; /* Read next line */ }

9 12

Re: [Python-ideas] Ordering keyword dicts
by Nick Coghlan June 10, 2013

June 10, 2013

(migrating thread from python-dev to python-ideas) On 10 June 2013 13:13, Alexander Belopolsky <alexander.belopolsky(a)gmail.com> wrote: > > On Sun, May 19, 2013 at 1:47 AM, Guido van Rossum <guido(a)python.org> wrote: >> >> I'm slow at warming up to the idea. My main concern is speed -- since >> most code doesn't need it and function calls are already slow (and >> obviously very common :-) it would be a shame if this slowed down >> function calls that don't need it noticeably. > > > Here is an idea that will not affect functions that don't need to know the > order of keywords: a special __kworder__ local variable. The use of this > variable inside the function will signal compiler to generate additional > bytecode to copy keyword names from the stack to a tuple and save it in > __kworder__. With that feature, an OrderedDict constructor, for example > can be written as > > def odict(**kwargs): > return OrderedDict([(key, kwargs[key]) for key in __kworder__]) The problem is that this is too late to help. To help folks understand the *technical* (rather than conceptual) limitations that are at issue in only *sometimes* ordering the keyword arguments here's an overview of the way the binding of arguments to parameters currently works: 1. In the calling bytecode, the arguments are collected together on the stack as positional arguments and keyword arguments: >>> def f(): ... return call(1, 2, *(3, 4), x=1, y=2, **{'z':3}) ... >>> dis(f) 2 0 LOAD_GLOBAL 0 (call) 3 LOAD_CONST 1 (1) 6 LOAD_CONST 2 (2) 9 LOAD_CONST 3 ('x') 12 LOAD_CONST 1 (1) 15 LOAD_CONST 4 ('y') 18 LOAD_CONST 2 (2) 21 LOAD_CONST 8 ((3, 4)) 24 BUILD_MAP 1 27 LOAD_CONST 5 (3) 30 LOAD_CONST 7 ('z') 33 STORE_MAP 34 CALL_FUNCTION_VAR_KW 514 (2 positional, 2 keyword pair) 37 RETURN_VALUE The various "CALL_FUNCTION*" opcodes in CPython almost always end up passing through a snippet like the following in ceval.c [1,2] (there are a couple of exceptions related to optimisation of calls that only involve positional arguments and parameters): if (PyCFunction_Check(func)) { PyThreadState *tstate = PyThreadState_GET(); C_TRACE(result, PyCFunction_Call(func, callargs, kwdict)); } else result = PyObject_Call(func, callargs, kwdict); [1] http://hg.python.org/cpython/file/default/Python/ceval.c#l4389 [2] http://hg.python.org/cpython/file/default/Python/ceval.c#l4484 You can see a couple of things here: * That *any* collection of arguments, regardless of syntax, is reduced to a positional argument tuple and a keyword argument dictionary before handing it over to the callable to handle the binding of arguments to parameters. * That this applies even to the optimised fast path that lets functions implemented in C avoid some of the overhead associated with the method dispatch machinery. 2. In the called object, the supplied arguments are bound appropriately as parameters. There's a lot more variability in how this happens. C functions usually use PyArg_ParseTuple or PyArg_ParseTupleAndKeywords. Python functions handle it as an implicit part of the frame initialization based on the function metadata. Guido's speed concern is specifically with any approach which requires the calling code to *always* check callables to see whether they want an ordered dictionary or not. Before we proceed to considering more exotic design ideas that require an "ordered or not" decision at the call sites, this concern should be validated by someone trying it out and checking the macro benchmark suite for the consequences. This can be done without actually making it possible to define functions that require order preservation - since we're mostly interested in the performance consequences when such a flag *isn't* set on the callables, an extra redundant check against "PyCFunction_GET_FLAGS(func) & 0x8000" (for PyCFunction instances), falling back to something like "PyObject_GetAttrId(__name__)" (for everything else), for all code paths in ceval.c that lead to creation of a kwdict instance should suffice. It may be that the cost of the flag check is swamped by the cost of actually creating the keyword dictionary, in which case the runtime check would be a preferable design choice, since function *invocation* wouldn't need to change, only function declarations. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

3 5

Re: [Python-ideas] A Simpler Enum For Python 3
by Eli Bendersky June 8, 2013

June 8, 2013

On Sat, Jun 8, 2013 at 11:04 AM, andrew cooke <andrew(a)acooke.org> wrote: > > > or, you know, you could have documented the design in the pep? isn't that > what's supposed to happen? > Did you read the PEP before you started spreading your FUD? http://www.python.org/dev/peps/pep-0435/#not-having-to-specify-values-for-e… > > if you want a secretay, pay them. > > I have absolutely no idea what you're talking about here. Eli > andrew > > > On Sat, Jun 08, 2013 at 09:30:33AM -0700, Eli Bendersky wrote: > > > Not sure if this is appropriate or useful, so won't repeat, but I > wrote an > > > alternative Enum implementation for Python 3, which is available at > > > https://github.com/andrewcooke/simple-enum > > > > > > > > It may have been appropriate and useful if you gave us the benefit of the > > doubt and wouldn't assume we are oblivious to the possibility of such an > > approach. Hint: it had seen probably at least 100 emails of discussion in > > this list (months back). Yes, it's way more fun to write flamy rants and > > new code than digging in mailing list archives; yet, language design > often > > requires much of the latter as well. > > > > Other than that, it's perfectly fine to have alternative enum proposals > out > > in the wild. I sincerely wish you well in the sense that if someone > doesn't > > like the stdlib enum and wants to use something like your bnum, they will > > find it online and will happily use it. > > > > Eli >

1 0

A Simpler Enum For Python 3
by andrew cooke June 8, 2013

June 8, 2013

Hi, Not sure if this is appropriate or useful, so won't repeat, but I wrote an alternative Enum implementation for Python 3, which is available at https://github.com/andrewcooke/simple-enum There's a discussion of how it differes from PEP 0435 at https://github.com/andrewcooke/simple-enum#discussion and some examples of teh differences at https://github.com/andrewcooke/simple-enum#things-you-can-do-with-the-simpl… Also, I have an almost-complete codebase that extends Ethan's "standard" implementation with some of the ideas in this design, available at https://github.com/andrewcooke/bnum - I have abandoned that project in favour of the one mentioned first, above, but it may be of interest if you want to consider modifying the PEP design slightly (in other words: bnum includes inheritance, simple-enum does not). Cheers, Andrew

3 2

Allow using ** twice
by Ram Rachum June 8, 2013

June 8, 2013

I'd like to be able to use ** twice (using 2 different dicts) when calling a function, and have them both feed arguments into the function.

25 59