Mailman 3 June 2013 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 21, 2020

Feb. 21, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

51 165

while conditional in list comprehension ??
by Wolfgang Maier Feb. 21, 2014

Feb. 21, 2014

Dear all, I guess this is so obvious that someone must have suggested it before: in list comprehensions you can currently exclude items based on the if conditional, e.g.: [n for n in range(1,1000) if n % 4 == 0] Why not extend this filtering by allowing a while statement in addition to if, as in: [n for n in range(1,1000) while n < 400] Trivial effect, I agree, in this example since you could achieve the same by using range(1,400), but I hope you get the point. This intuitively understandable extension would provide a big speed-up for sorted lists where processing all the input is unnecessary. Consider this: some_names=["Adam", "Andrew", "Arthur", "Bob", "Caroline","Lancelot"] # a sorted list of names [n for n in some_names if n.startswith("A")] # certainly gives a list of all names starting with A, but . [n for n in some_names while n.startswith("A")] # would have saved two comparisons Best, Wolfgang

19 70

A nice __repr__ for the ast.* classes?
by Haoyi Li Nov. 19, 2013

Nov. 19, 2013

Wouldn't it be nice if this >>> import ast >>> print repr(ast.parse("(1 + 1)").body[0].value) <_ast.BinOp object at 0x0000000001E94B38> printed something more useful? >>> print repr(ast.parse("(1 + 1)").body[0].value) BinOp(left=Num(n=1), op=Add(), right=Num(n=1)) I've been doing some work on macropy <https://github.com/lihaoyi/macropy>, which uses the ast.* classes extensively, and it's annoying that we have to resort to dirty-tricks like monkey-patching the AST classes (for CPython 2.7) or even monkey-patching __builtin__.repr (to get it working on PyPy) just to get eval(repr(my_ast)) == my_ast to hold true. And a perfectly good solution already exists in the ast.dump() method, too! (It would also be nice if "==" did a structural comparison on the ast.* classes too, but that's a different issue). -Haoyi

5 5

Gzip and zip extra field
by Serhiy Storchaka Nov. 17, 2013

Nov. 17, 2013

Gzip files can contains an extra field [1] and some applications use this for extending gzip format. The current GzipFile implementation ignores this field on input and doesn't allow to create a new file with an extra field. ZIP file entries also can contains an extra field [2]. Currently it just saved as bytes in the `extra` attribute of ZipInfo. I propose to save an extra field for gzip file and provide structural access to subfields. f = gzip.GzipFile('somefile.gz', 'rb') f.extra_bytes # A raw extra field as bytes # iterating over all subfields for xid, data in f.extra_map.items(): ... # get Apollo file type information f.extra_map[b'AP'] # (or f.extra_map['AP']?) # creating gzip file with extra field f = gzip.GzipFile('somefile.gz', 'wb', extra=extrabytes) f = gzip.GzipFile('somefile.gz', 'wb', extra=[(b'AP', apollodata)]) f = gzip.GzipFile('somefile.gz', 'wb', extra={b'AP': apollodata}) # change Apollo file type information f.extra_map[b'AP'] = ... Issue #17681 [3] has preliminary patches. There is some open doubt about interface. Is not it over-engineered? Currently GzipFile supports seamless reading a sequence of separately compressed gzip files. Every such chunk can have own extra field (this is used in dictzip for example). It would be desirable to be able to read only until the end of current chunk in order not to miss an extra field. [1] http://www.gzip.org/format.txt [2] http://www.pkware.com/documents/casestudies/APPNOTE.TXT [3] http://bugs.python.org/issue17681

3 4

Stdlib YAML evolution (Was: PEP 426, YAML in the stdlib and implementation discovery)
by anatoly techtonik Nov. 13, 2013

Nov. 13, 2013

FWIW, I am +1 on for the ability to read YAML based configs Python without dependencies, but waiting for several years is hard. Maybe try an alternative data driven development process as opposed to traditional PEP based all-or-nothing style to speed up the process? It is possible to make users happy incrementally and keep development fun without sacrificing too much on the Zen side. If open source is about scratching your own itches, then the most effective way to implement a spec would be to allow people add support for their own flavor without disrupting works of the others. For some reason I think most people don't need full YAML speccy, especially if final implementation will be slow and heavy. So instead of: import yaml I'd start with something more generic and primitive: from datatrans import yamlish Where `datatrans` is data transformation framework taking care of usual text parsing (data partitioning), partition mapping (structure transformation) and conversion (binary to string etc.) trying to be as fast and lightweight as possible, bringing vast field for future optimizations on algorithmic level. `yamlish` is an implementation which is not vastly optimized (datatrans to yamlish is like RPython to PyPy) and can be easily extended to cover more YAML (hopefully). Therefore the name - it doesn't pretend to parse YAML - it parses some supported subset, which is improved over time by different parties (if datatrans is done right to provide readable (maintainable + extensible) code for implementation). There is an exisiting package called `yamlish` on PyPI - I am not talking about it - it is PyYAML based, which is not an option for now as I see it. So I stole its name. Sorry. This PyPI package was used to parse TAP format, which is again, a subset. Subset.. It appears that YAML is good for humans for its subsets. It leaves an impression (maybe it's just an illusion) that development work for subset support can also be partitioned. If `datatrans` "done right" is possible, it will allow incremental addition of new YAML features as the need for them arises (new data examples are added). Or it can help to build parsers for YAML subsets that are intentionally limited to make them performance efficient. Because `datatrans` is a package isolating parsing, mapping and conversion parts of the process to make it modular and extensible, it can serve as a reference point for various kinds of (scientific) papers including the ones that prove that such data transformation framework is impossible. As for `yamlish` submodule, the first important paper covering it will be a reference table matrix of supported features. While it all sounds too complicated, driving development by data and real short term user needs (as opposed to designing everything upfront) will make the process more attractive. In data driven development, there are not many things that can break - you either continue parsing previous data or not. The output from the parsing process may change over time, but it may be controlled by configuring the last step of data transformation phase. `Parsing AppEngine config file` or `reading package meta data` are good starting points. Once package meta data subset is parsed, it is done and won't break. The implementation for meta data parsing may mature in distutils package, for AppEngine in its SDK, and merged in either of those, sending patches for `datastrans` to stdlib. The question is only to design output format for the parse stage. I am not sure everything should be convertible into Python objects using the "best fit" technique. I will be pretty comfortable if target format will not be native Python objects at all. More than that - I will even insist to avoid converting to native Python object from the start. The ideal output for the first version should be generic tree structure with defined names for YAML elements. The tree that can be represented as XML where these names are tags. The tree can be therefore traversed and selected using XPath/JQuery syntax. It will take several years for implementation to mature and in the end there will be a plenty of backward compatibility matters with the API, formatting and serializing. So the first thing I'd do is [abandon serialization]. From the point of view of my self-proclaimed data transformation theory, the input and output formats are data. If output format is not human readable - as some linked Python data structures in memory - it wastes time and hinders development. Serializing Python is a problem of different level, which is an example of binary, abstract, memory-only output format - a lot of properties that you don't want to deal with while working with data. To summarize: 1. full spec support is no goal 2. data driven (using real world examples/stories) 3. incremental (one example/story at a time) 4. research based (beautiful ideas vs ugly real world limitations) 5. maintainable (code is easy to read and understand the structure) 6. extensible (easy to find out the place to be modified) 7. core "generic tree" data structure as an intermediate format and "yaml tree" data structure as a final format from parsing process P.S. I am willing to wok on this "data transformation theory" stuff and prototype implementation, because it is generally useful in many areas. But I need support. -- anatoly t.

9 12

Idea: Compressing the stack on the fly
by Ram Rachum Sept. 12, 2013

Sept. 12, 2013

Hi everybody, Here's an idea I had a while ago. Now, I'm an ignoramus when it comes to how programming languages are implemented, so this idea will most likely be either (a) completely impossible or (b) trivial knowledge. I was thinking about the implementation of the factorial in Python. I was comparing in my mind 2 different solutions: The recursive one, and the one that uses a loop. Here are example implementations for them: def factorial_recursive(n): if n == 1: return 1 return n * factorial_recursive(n - 1) def factorial_loop(n): result = 1 for i in range(1, n + 1): result *= i return result I know that the recursive one is problematic, because it's putting a lot of items on the stack. In fact it's using the stack as if it was a loop variable. The stack wasn't meant to be used like that. Then the question came to me, why? Maybe the stack could be built to handle this kind of (ab)use? I read about tail-call optimization on Wikipedia. If I understand correctly, the gist of it is that the interpreter tries to recognize, on a frame-by-frame basis, which frames could be completely eliminated, and then it eliminates those. Then I read Guido's blog post explaining why he doesn't want it in Python. In that post he outlined 4 different reasons why TCO shouldn't be implemented in Python. But then I thought, maybe you could do something smarter than eliminating individual stack frames. Maybe we could create something that is to the current implementation of the stack what `xrange` is to the old-style `range`. A smart object that allows access to any of a long list of items in it, without actually having to store those items. This would solve the first argument that Guido raises in his post, which I found to be the most substantial one. What I'm saying is: Imagine the stack of the interpreter when it runs the factorial example above for n=1000. It has around 1000 items in it and it's just about to explode. But then, if you'd look at the contents of that stack, you'd see it's embarrassingly regular, a compression algorithm's wet dream. It's just the same code location over and over again, with a different value for `n`. So what I'm suggesting is an algorithm to compress that stack on the fly. An algorithm that would detect regularities in the stack and instead of saving each individual frame, save just the pattern. Then, there wouldn't be any problem with showing informative stack trace: Despite not storing every individual frame, each individual frame could still be *accessed*, similarly to how `xrange` allow access to each individual member without having to store each of them. Then, the stack could store a lot more items, and tasks that currently require recursion (like pickling using the standard library) will be able to handle much deeper recursions. What do you think? Ram.

13 20

format specifier for "not bytes"
by Daniel Holth July 5, 2013

July 5, 2013

While I was implementing JSON-JWS (JSON web signatures), a format which in Python 3 has to go from bytes > unicode > bytes > unicode several times in its construction, I notice I wrote a lot of bugs: "sha256=b'abcdef1234'" When I meant to say: "sha256=abcdef1234" Everything worked perfectly on Python 3 because the verifying code also generated the sha256=b'abcdef1234' as a comparison. I would have never noticed at all unless I had tried to verify the Python 3 output with Python 2. I know I'm a bad person for not having unit tests capable enough to catch this bug, a bug I wrote repeatedly in each layer of the bytes > unicode > bytes > unicode dance, and that there is no excuse for being confused at any time about the type of a variable, but I'm not willing to reform. Instead, I would like a new string formatting operator tentatively called 'notbytes': "sha256=%notbytes" % (b'abcdef1234'). It gives the same error as 'sha256='+b'abc1234' would: TypeError: Can't convert 'bytes' object to str implictly

8 17

Re: [Python-ideas] Is this PEP-able? for X in ListY while conditionZ:
by Wolfgang Maier July 3, 2013

July 3, 2013

On Jun 27, 2013, at 5:19, Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> If you're willing, I'm actually thinking this may be one of those >> discussions that's worth summarising in a PEP, even if it's just to >> immediately mark it Rejected. Similar to PEP 315 and a few other PEPs, >> it can help to have a document that clearly spells out the objective >> (which I think you summarised nicely as "trying to find a syntactic >> replacement for itertools.takewhile, just as comprehensions replaced >> many uses of map and filter"), even if no acceptable solution has been >> found. > >The ideas are pretty broad-ranging. In particular, #1 got two somewhat >supportive responses that had nothing to do with the comprehension termination >idea. Do side issues like that need to be discussed first/separately before >referencing them in a while clause PEP? > >Also, we seem pretty far from a consensus on what the actual tradeoffs are for >most of the options. For example, is the definition of comps in terms of loops >the core idea behind the abstraction, or a minor triviality that's only useful >in understanding bad code? Are the differences between comps and genexps a bug >or a feature? > >Finally, how does this connect up to the original idea of this thread, which was >to draft a PEP for while clauses in loop statements rather than in >comprehensions? Obviously if that existed, it would change the options for comp >syntax. Do we need to include that idea in the discussion? (I completely forgot >about it while digging up all of the ideas that spun off from it...) I don't think while clauses in loop statements are a big gain for the language. There's break and despite all the programming-style discussions going on in this part of the thread, it has been working well for many years and most people find it intuitive to use. >> That phrasing of the objective also highlights a counter argument I >> hadn't considered before: if we don't consider takewhile a common >> enough use case to make it a builtin, why on *earth* are we even >> discussing the possibility of giving it dedicated syntax? > >Besides early termination for comps, the only good use case I know of is in code >that already makes heavy use of itertools, and making one of the functions a >builtin wouldn't change anything. > >And if early termination for comps is a special case, it doesn't seem too >unreasonable to consider other ways to handle it. > >But you're right that "make takewhile a builtin" should probably be considered >among the alternatives. But a builtin takewhile would still not come with nicer and easier to read syntax! I guess the use cases are not that rare, it's just that right now people switch to explicit loops when they need early termination because it keeps things readable. I'm encountering this situation quite regularly (reading a small header from large files is the most common example, but there are others). Let me suggest one more solution although it requires a new keyword: introduce *breakif* condition and define its translation as if condition: break . You can now write (x for x in iterable breakif x < 0) and I don't see a way how that could possibly be misread by anyone. Also it would translate unambiguously to the explicit: for x in iterable: breakif x<0 # itself translating to if x<0: break yield x It would work with genexps, comprehensions and explicit loops alike (with very little benefit for the later, though maybe it increases readability even there by making it clear from the start of the line what the purpose of the condition test is). Best, Wolfgang

19 40

Re: [Python-ideas] "Iteration stopping" syntax [Was: Is this PEP-able? for X in ListY while conditionZ:]
by Shane Green June 30, 2013

June 30, 2013

Yes, but it only works for generator expressions and not comprehensions. My opinion of that workaround is that it’s also a step backward in terms of readability. I suspect if i < 50 else stop() would probably also work, since it throws an exception. That’s better, IMHO. On Jun 28, 2013, at 6:38 PM, Andrew Carter <acarter(a)g.hmc.edu> wrote: > Digging through the archives (with a quick google search) http://mail.python.org/pipermail/python-ideas/2013-January/019051.html, if you really want an expression it seems you can just do > > def stop(): > raise StopIteration > list(i for i in range(100) if i < 50 or stop()) > > it seems to me that this would provide syntax that doesn't require lambdas. > > > > > > > On Fri, Jun 28, 2013 at 4:50 PM, Alexander Belopolsky <alexander.belopolsky(a)gmail.com> wrote: > > > > On Fri, Jun 28, 2013 at 7:38 PM, Shane Green <shane(a)umbrellacode.com> wrote: > .. > [x until condition for x in l ...] or > [x for x in l until condition] > > Just to throw in one more variation: > > [expr for item in iterable break if condition] > > (inversion of "if" and "break"reinforces the idea that we are dealing with an expression rather than a statement - compare with "a if cond else b") > > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > http://mail.python.org/mailman/listinfo/python-ideas

8 21

Short form for keyword arguments and dicts
by Anders Hovmöller June 29, 2013

June 29, 2013

Keyword arguments are great for increasing readability and making code more robust but in my opinion they are underused compared to the gains they can provide. You often end up with code like: foo(bar=bar, baz=baz, foobaz=foobaz) which is less readable than the ordered argument version when the names of the variables and the keywords match. ( Here's another guy pointing out the same thing: http://stackoverflow.com/questions/7041752/any-reason-not-to-always-use-key… ) I have a suggestion that I believe can enable more usage of keyword arguments while still retaining almost all the brevity of ordered arguments: if the variable name to the right of the equal sign equals the keyword argument ("foo=foo") make it optional to just specify the name once ("=foo"). For completeness I suggest also make the same change for dictionaries: {'foo': foo} -> {:foo}. This change would turn the following code: a = 1 b = 2 c = 3 d = {'a':a, 'b':b, 'c':c} foo(a=a, b=b, c=c) into: a = 1 b = 2 c = 3 d = {:a, :b, :c} foo(=a, =b, =c) This should be compatible with existing code bases since the new forms are syntax errors in current python. What do you think? / Anders Hovmöller

19 96

PEP 315: do-while
by Łukasz Langa June 29, 2013

June 29, 2013

The PEP is deferred because it seems that while True: <code> if condition: break is good enough. I agree. We should reject the PEP and summarise the status. Alternatively, the only way I think we can improve on the syntax above is something like this: do: <code> if condition: break or without a new keyword: while: <code> if condition: break The empty-predicate variant would let Python check whether there's actually a break in the body of the loop. -- Best regards, Łukasz Langa WWW: http://lukasz.langa.pl/ Twitter: @llanga IRC: ambv on #python-dev

12 27

Re: [Python-ideas] Is this PEP-able? for X in ListY while conditionZ:
by jimjhb＠aol.com June 28, 2013

June 28, 2013

You shouldn't have to invoke takewhile and a lambda just to break out of for loop. >http://docs.python.org/2/library/itertools.html#itertools.takewhile > >for item in takewhile(lambda x: x < 5, range(10)): > pass >> >> [People who avoid the 'break' by functionalizing an inner portion of the >> loop are just kidding themselves and making their own code worse, IMO. >> Takewhile from itertools also works, but that's clumsy and wordy as well.] >>

18 42

Re: [Python-ideas] Is this PEP-able? for X in ListY while conditionZ:
by jimjhb＠aol.com June 25, 2013

June 25, 2013

Syntax: for X in ListY while conditionZ: The 'for' loop would proceed as long as conditionZ remains true. The motivation is to be able to make use of all the great aspects of the python 'for' (no indexing or explicit end condition check, etc.) and at the same time avoiding a 'break' from the 'for'. (NOTE: Many people are being taught to avoid 'break' and 'continue' at all costs, so they instead convert the clean 'for' into a less-clean 'while'. Or they just let the 'for' run out. You can argue against this teaching practice (at least for Python) but that doesn't mean it's not prevalent and prevailing.) [People who avoid the 'break' by functionalizing an inner portion of the loop are just kidding themselves and making their own code worse, IMO. Takewhile from itertools also works, but that's clumsy and wordy as well.] I'm not super familiar with CPython, but I'm pretty sure I could get this up and working without too much effort. Please note that I don't feel the answer to this is 'just use break'. Programmers are now being taught to avoid 'break' and 'continue' as if they were 'goto's. The result (now) is that people are avoiding the 'for' (with its GREAT properties) because they can't break out of it. Comments and Questions welcome. Thanks.

9 19

interactive sqlite3 module
by ZeD June 25, 2013

June 25, 2013

Hi. some python modules contains a "console" usage (for example the module timeit). It would be useful (for the times when you have python installed but not a sqlite executable... think windows...) to add a minimal interactive sqlite interpreter... a (really) basic implementation is something like the source below... teoretically the idea is to to $ python -msqlite3 'my_data.db' or $ python -msqlite3 ':memory:' and have a simple sqlite interpreter do you think it whould be useful? ------------------------- #!/usr/bin/env python # ATM I'm on python2... from __future__ import print_function try: input = raw_input except: pass # ideally this file is part of sqlite from sqlite3 import * def main(): from sys import argv if len(argv) < 2: raise SystemExit("usage: %s DATABASE" % (argv[0], )) with connect(argv[1]) as conn: while True: command = input("> ") if command == '.exit': break try: c = conn.cursor() for row in c.execute(command): print(row) conn.commit() except Exception as e: print(e) if __name__ == '__main__': main() -- Vito 'ZeD' De Tullio

3 2

Catching of multiple exceptions without halting execution
by Samuel Littley June 24, 2013

June 24, 2013

I find I often have to run several tests on data input, making a list of which tests succeed, which fail, and somehow determine where to go after running all the tests. Obviously, the standard way of showing that a test as failed is to raise an exception, caught by try/except/finally, however this would only allow one test to be flagged as failing, requiring multiple runs to correct every fault that may exist. I propose an alternative to try/except, as follows: validate: // Code to run tests, raising exceptions if tests fail accept: // Code to run if all tests pass (i.e. no exceptions) reject es: // Code to handle each failed test except: // Code to handle non-test related exceptions finally: // Code to be always executed The difference to a normal try/except is a different type of exception, which, rather than halting execution, is added to the list `es`, which the reject block could then loop through to display error messages, require re-entry, etc. Standard exceptions could be raised and caught by the except block. Alternatively, the except block could not be a part of this, and instead all exceptions are caught in the reject block, which could then raise exceptions itself to be caught by a try/except/finally around the validate/accept/reject/finally The use case I thought of is validating data entry (from web forms for example), where each exception creates an error message displayed on the form, however I'm pretty sure there would be other uses for this. Samuel Littley

6 6

Fwd: Iterable unpacking within containers
by SpaghettiToastBook . June 24, 2013

June 24, 2013

I think it would be very convenient if the * syntax was extended to allow unpacking iterables inside containers. For example: >>> x = range(1, 4) >>> y = [-2, *(-1, 0), *x, 4, 5] >>> print(y) [-2, -1, 0, 1, 2, 3, 4, 5] This would be especially convenient if two lists needed to be combined, but one list's contents needed to be the the middle of the other one. I don't think any code would break because the suggested syntax currently raises a SyntaxError.

5 4

Get the class that *defines* a method
by anatoly techtonik June 24, 2013

June 24, 2013

Currently, the only way to get the name of the class that *defines* a method is to get chain of parent classes from inspect.getmro() and look into every class's dict until a given name is found. Knowing that dict contains not only methods, and knowing that it can be modified at run-time, this doesn't seem too reliable for me (unless Python itself does the same). At first I wanted to propose an enhancement to runtime skeleton of Python, which is 2D lookup tree for objects and containers that define them. But then I realized that it will may not reflect the model I need. For example, classes need to provide their parent classes, but in my model classes need to provide module name (or function name, or method name) in which they are defined. And while writing this I realized that *definition* scope may be different from *run-time* scope, and Python doesn't make it clear: >>> def lll(): ... class A(object): ... pass ... a=A() ... return a ... >>> lll() <__main__.A object at 0x948252c> >>> __main__.A Traceback (most recent call last): File "<stdin>", line 1, in <module> NameError: name '__main__' is not defined The A object is said to be in __main__ namespace, but it seems to be a run-time namespace local to function and it seems like Python loses this information. There is no information about the function that defined the class (owner, parent or .?.), and hence no info about container of the function, which makes it hard to assume the scope of variables for this class at run-time. So, the above is a generalization of a simple idea - store the "structure reference" of the class that *defines* a method inside this method. "structure reference" here is the address in the nested scopes formed by Python definitions. The specific action items for you here are: 1. is that stuff will be useful (for me it brings some much needed consistency into the chaos of run-time Python object space) 2. what is the best way to define/cache the reference to the class defining the method? 3. what is the best way to define/cache the reference to the scope defining the method? 4. what is the best way to organize storing of this static scope structure information at run-time? See method.im_class note at http://docs.python.org/2/library/inspect.html#types-and-members "Namespaces are one honking great idea -- let's do more of those!" (c) import this -- anatoly t.

4 3

Python execution progress counter
by anatoly techtonik June 23, 2013

June 23, 2013

In x86 assembly there is a concept of instruction pointer (IP) that indicates where a computer is in its program sequence (c) http://en.wikipedia.org/wiki/Program_counter I wonder if it is possible to implement a similar thing in Python? Not the instruction pointer, which points to the linear memory address space (and therefore is a 1D structure), but a 2D Execution Progress Counter, which counts position in program sequence for every stack level? -- anatoly t.

3 2

argparse: --help-explain [options] ...
by anatoly techtonik June 23, 2013

June 23, 2013

Inspired by QEMU: qemu -L . -m 128 -hda ReactOS.vmdk -net nic,model=ne2k_pci -net user -serial file:CON Can anybody tell what -L option does? QEMU help output is 6+ pages, and while it is not impossible to search through the content, I guess the first try for everybody is to try to lookup options visually. Going with every option one by one is rather tedious. So, the idea is that for such applications with extensive command line API, optparse (argparse or docopt) could provide --help-explain (or just --explain) option that parses command line and explains what it found. --help-explain [options] ... [options] ... --help-explain The option is position-independent to append it to existing command line. The explanation logic could be extensible, but that's a topic about state machine of valid option combinations. I am not sure argparse allows to manage that. However, docopt may be the saviour. -- anatoly t.

2 2

Adding zope.testrunner test selections to unittest
by Barry Warsaw June 22, 2013

June 22, 2013

(Maybe this should better go to the TIP mailing list?) One of the last things from zope.testrunner that I really rely on is its highly flexible support for test selection: https://pypi.python.org/pypi/zope.testrunner/4.4.0#test-selection Michael and I talked briefly about adding something like this to unittest's discover, where --pattern is a pale ghost of related functionality. Has anybody looked into this before? Is it something that we could feasibly add to Python 3.4? (and/or unittest2) -Barry

5 6

unittest and warnings
by Giampaolo Rodola' June 18, 2013

June 18, 2013

One of the features I like the most in the new unittest2 module is the possibility to skip tests and how they are included in the final result (e.g. FAILED (errors=2, failures=3, *skipped=5*)). After http://bugs.python.org/issue10093 it is not rare that different ResourceWarnings appear while running tests. Personally I consider these warnings as something which needs to be fixed so after I run tests I often scroll my console window back up to see whether there were warnings. Would it make sense for unittest module to keep track of them so that they get included in the final test result as it currently happens for skipped tests? Side note: unittest provides some "skip-related" APIs such unittest.skip*, TestResult.skipped and others but I don't think something similar would be necessary except maybe a TestResult.warnings list similar to TestResult.skipped. Regards, - Giampaolo https://code.google.com/p/pyftpdlib https://code.google.com/p/psutil https://code.google.com/p/pysendfile

4 4

Re: [Python-ideas] unittest and warnings
by Vernon D. Cole June 18, 2013

June 18, 2013

+1 for being able to get warnings counted, and optionally included with traceback. I am working on the django-mssql project, but I don't know django itself all that well. Two warnings are produced during the run of 4900 tests. (No, I did not key that incorrectly -- the test suite runs for hours on Windows.) I want to repair the code which causes the warnings -- but it is pretty hard to identify what caused them. Being able to get a full traceback, rather than a printout in the midst of the ...... would be really nice. -- Vernon Cole

1 0

Elixir inspired pipe to apply a series of functions
by Jan Wrobel June 17, 2013

June 17, 2013

Hello, I've recently stumbled upon a Joe Armstrong's (of Erlang) blog post that praises an Elixir pipe operator: http://joearms.github.io/2013/05/31/a-week-with-elixir.html The operator allows to nicely structure code that applies a series of functions to transform an input value to some output. I often end up writing code like: pkcs7_unpad( reduce(lambda result, block: result.append(block), map(decrypt_block, pairwise([iv] + secret_blocks)))) Which is dense, and needs to be read backwards (last operation is written first), but as Joe notes, the alternative is also not very compelling: decrypted_blocks = map(decrypt_block, pairwise([iv] + secret_blocks)) combined_blocks = reduce(lambda result, block: result.append(block)) return pkcs7_unpad(combined_blocks) The pipe operator nicely separates subsequent operations and allows to read them in a natural order without the need for temporary variables. Something like: [iv] + secret_blocks |> pairwise |> map, decrypt_block |> \ reduce, lambda result, block: result.append(block) |> \ pkcs7_unpad I'm not sure introducing pipes like this at the Python level would be a good idea. Is there already a library level support for such constructs? If not, what would be a good way to express them? I've tried a bit an figured out a following API (https://gist.github.com/wrr/5775808): Pipe([iv] + secret_blocks)\ (pairwise)\ (map, decrypt_block)\ (reduce, lambda result, block: result.append(block))\ (pkcs7_unpad)\ () The API is more verbose than the language level operator. I initially tried to overload `>>`, but it doesn't allow for additional arguments. It is also somehow smelly, because `()` returns a different type of value if it is invoked without an argument. Any suggestions how this could be improved? Best regards, Jan

4 5

duck typing for io write methods
by Wolfgang Maier June 16, 2013

June 16, 2013

Dear all, currently - and referring to Python 3 - the write methods of the different io module objects work on bytes and str objects only. The built-in functions print() and bytes(), on the other hand, use an arbitrary object's __str__ and __bytes__ methods to compute the str and bytes they should work with. Wouldn't it be more consistent and pythonic if the io write methods behaved the same way? Best, Wolfgang

16 41

Add "htmlcharrefreplace" error handler
by Serhiy Storchaka June 14, 2013

June 14, 2013

I propose to add "htmlcharrefreplace" error handler which is similar to "xmlcharrefreplace" error handler but use html entity names if possible. >>> '∀ x∈ℜ'.encode('ascii', 'xmlcharrefreplace') b'∀ x∈ℜ' >>> '∀ x∈ℜ'.encode('ascii', 'htmlcharrefreplace') b'∀ x∈&real;' Possible implementation: import codecs from html.entities import codepoint2name def htmlcharrefreplace_errors(exc): if not isinstance(exc, UnicodeEncodeError): raise exc try: replace = r'&%s;' % codepoint2name[ord(exc.object[exc.start])] except KeyError: return codecs.xmlcharrefreplace_errors(exc) return replace, exc.start + 1 codecs.register_error('htmlcharrefreplace', htmlcharrefreplace_errors) Even if do not register this handler from the start, it may be worth to provide htmlcharrefreplace_errors() in the html or html.entities module.

12 47

Suprocess functionality partitioning
by anatoly techtonik June 13, 2013

June 13, 2013

Something needs to be done about subprocess API, which got overly complicated. The idea is to have: shutil.run() - run command through shell, unsafe sys.run() - run command directly in operating system, ?safe Both should be API compatible (unblocking stdin/stdout read etc.). Currently, subprocess calls are unreadable in any Python code - many conditions makes its behaviour hard to predict and memorize. By partitioning shell and out-of-shell execution, the documentation will be easier to grasp and reference to. Maybe it will be even possible to add some 2D table of various subprocess states that affect behavior. -- anatoly t.

7 12

Re: [Python-ideas] [Python-Dev] Unicode minus sign in numeric conversions
by Guido van Rossum June 13, 2013

June 13, 2013

[Diverting to python-ideas, since this isn't as clear-cut as you think.] Why exactly is that expected behavior? What's the use case? (Surely you don't have a keyboard that generates \u2212 when you hit the minus key? :-) Is there a Unicode standard for parsing numbers? IIRC there are a variety of other things marked as "digits" in the Unicode standard -- do we do anything with those? If we do anything we should be consistent. For now, I think we *are* consistent -- we only support the ASCII representation of numbers. (And that's the only representation we generate as output as well -- think about symmetry too.) This page scares me: http://en.wikipedia.org/wiki/Numerals_in_Unicode --Guido On Sat, Jun 8, 2013 at 2:49 PM, Łukasz Langa <lukasz(a)langa.pl> wrote: > Expected behaviour: >>>> float('\N{MINUS SIGN}12.34') > -12.34 > > > Current behaviour: > Traceback (most recent call last): > ... > ValueError: could not convert string to float: '−12.34' > > > Please note: '\N{MINUS SIGN}' == '\u2212' > > -- > Best regards, > Łukasz Langa > > WWW: http://lukasz.langa.pl/ > Twitter: @llanga > IRC: ambv on #python-dev > > _______________________________________________ > Python-Dev mailing list > Python-Dev(a)python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org -- --Guido van Rossum (python.org/~guido)

11 42

Add \e escape code
by Steven D'Aprano June 12, 2013

June 12, 2013

Should Python support \e in strings for the ESC control character, ASCII 0x1B (27)? \e is supported by Perl, PHP, Ruby, and although it is not standard C, gcc: http://gcc.gnu.org/onlinedocs/gcc-3.1/gcc/Character-Escapes.html I have the Python Pocket Reference, by Mark Lutz, 1st Edition from 1998, which lists \e as a string escape code. I don't know if that was a mistake, or if \e used to be supported prior to 1.5 but was then removed. -- Steven

9 14

Add "namereplace" error handler
by Serhiy Storchaka June 11, 2013

June 11, 2013

I propose to add "namereplace" error handler which is similar to "backslashreplace" error handler but use \N escapes instead of \x/\u/\U escapes. >>> '−1'.encode('ascii', 'backslashreplace') b'\\u22121' >>> '−1'.encode('ascii', 'namereplace') b'\\N{MINUS SIGN}1' In some cases such representation is more readable. What are you think about this? Are there suggestions for better name?

10 17

Python 3 open() text files: make encoding parameter optional for cross-platform scripts
by anatoly techtonik June 11, 2013

June 11, 2013

Without reading subject of this letter, what is your idea about which encoding Python 3 uses with open() calls on a text file? Please write in reply and then scroll down. Without cheating my opinion was cp1252 (latin-1), because it was the way Python 2 assumed all text files are. Or Python 2 uses ISO-8859-1? But it appeared to be different way - http://docs.python.org/3/library/functions.html#open. No, it appeared here - https://bitbucket.org/techtonik/hexdump/pull-request/1/ and after a small lecture I realized how things are bad. open() in Python uses system encoding to read files by default. So, if Python script writes text file with some Cyrillic character on my Russian Windows, another Python script on English Windows or Greek Windows will not be able to read it. This is just what happened. The solution proposed is to specify encoding explicitly. That means I have to know it. Luckily, in this case the text file is my .py where I knew the encoding beforehand. In real world you can never know the encoding beforehand. So, what should Python do if it doesn't know the encoding of text file it opens: 1. Assume that encoding of text file is the encoding of your operating system 2. Assume that encoding of text file is ASCII 3. Assume that encoding of text file is UTF-8 Please write in reply and then scroll down. I propose three, because ASCII is a binary compatible subset of UTF-8. Choice one is the current behaviour, and it is very bad. Troubleshooting this issue, which should be very common, requires a lot of prior knowledge about encodings and awareness of difference system defaults. For cross-platform work with text files this fact implicitly requires you to always use 'encoding' parameter for open(). Is it enough for a PEP? This stuff is rather critical IMO, so even if it will be rejected there will be a documented design decision. -- anatoly t.

15 38

Hooks into super()'s __getattribute__
by Ronald Oussoren June 11, 2013

June 11, 2013

Hi, Super() currently does not have a way to hook into the behavior of attribute lookup, the __getattribute__ method of super peeks in the __dict__ of types along the MRO until it finds what it is looking for. That can be a problem when a class implements __getattribute__ and doesn't necessarily store (all) attributes in the type __dict__. PyObjC is an example where the current behavior causes problems: PyObjC defines proxy classes for classes in the Objective-C runtime. The __dict__ of those classes is filled on demand, whenever a method is called that isn't in the __dict__ yet PyObjC looks in the Objective-C runtime datastructures for the method and adds it to __dict__. This works fine for normal method resolution, but can fail with super: super will only return the correct value when the superclass method happens to be in __dict__ already. My current solution for this is a custom subclass of super that must be used with Cocoa subclasses, and that's something I'd like to get rid off. I'd therefore like to propose adding a slot to PyTypeObject that is called by super's __getattribute__ when present, instead of peeking in the type's tp_dict slot. Does this look like a sane solution? I've filed an issue about this that includes a proof of concept patch: http://bugs.python.org/issue18181. That patch is incomplete, but does make it possible to use the builtin super for Cocoa classes (with a suitably patched version of PyObjC). An earlier issue mentions that the current behavior of super can be inconsistent with the behavior of __getattribute__, see http://bugs.python.org/issue783528 (that issue is closed, but IMHO for the wrong reason). I'm appearently not the only one running into this problem ;-) Ronald

1 0

Mixed script numbers. Was: Unicode minus sign in numeric conversions
by Alexander Belopolsky June 10, 2013

June 10, 2013

I am changing the subject because the issue of mixing digits from different scripts is quite different from the issue of accepting MINUS SIGN. I left a comment on the original subject at the issue tracker: < http://bugs.python.org/issue6632#msg190881>. On Mon, Jun 10, 2013 at 8:53 PM, Steven D'Aprano <steve(a)pearwood.info> wrote: > > On 10/06/13 05:35, Alexander Belopolsky wrote: >> >> ... Consider this case: >> >>>>> float('123٠95') >> 123095.0 >> >> Depending on your font, '123٠95' may be indistinguishable from '123.95'. > > > Indistinguishable *by eye* maybe, but the same applies to ASCII, > 365lO98, and there are plenty of ways to distinguish them other > than by a careless glance at the screen. > > [Aside: I have seen users type I or O for digits, based on the fact > that it works fine when using a typewriter, and I've read books from > the 1970s that recommended that number parsers accept I, L and O > for just that reason.] I am not sure why your example is relevant. There is little harm from accepting O for zero, but accepting it for say 8 would be a different story. > > It would be a pretty awful font that made ٠ look like . But even if it did, what is the concern here? > If somebody enters a mixed script number, presumably they have some reason for it. Sure, say someone who wants to sell you a $23.95 gadget for $23,095. :-)

1 0

namedtuple is not as good as it should be
by Christian Tismer June 10, 2013

June 10, 2013

Sorry Raymond if this offends you. But after some extensive use of namedtuple I think it needs a re-design. The pros: _________ - a namedtuple can be easily used in place of a tuple. - it provides names for its fields and behaves like a tuple, otherwise. Furthermorre: - a named tuple is the natural choice for modelling the records of a database. The cons: - All of the above becomes wrong if you use namedtuple as a real replacement for tuple. - especially for databases it makes little sense as it is. Reason: Pickling ________ - pickling of tuples: always possible if it contains built-in types. tuples are simply tuples. There is just _one_ type. - pickling of namedtuple: sometimes possible, if your definition is static enough. namedtuple has a subtype per tuple layout, and you need to cope with that. Just to be clear about that: Sure, it is possible to pickle named tuples, but you have to think about it! And to have to thing about it trashes a lot of the fun of having those names for free. And typically this happens after you did your analysis of 20 GB of data: You cannot pickle your nicely formatted namedtuple instances after the fact. Actually, to save all the computation, you do a hack that turns all your alive namedtuple instances back into ordinary tuples. Silent implications introduced by namedtuple: _____________________________________________ Without being very explicit, namedtuple makes you use it happily instead of tuples. But instead of using a native, simple type, you now use a not-so-simple, user-defined type. This type - is not built in - has a class definition - needs a global, constant definition _somewhrere_ Typically, you run a script interactively, and typically you need to pickle some __main__.somename namedtuple class instances. This is exactly what you don't want! You want to have some anonymous data in the pickle and don't want to make anything fixed in stone. namedtuple() Factory Function for Tuples with Named Fields __________________________________________________________ It would be great if namedtuple were just this, as the doku says. But instead, it - creates a named class, i.e. forces me to name it - you create instances of that specific class and not just tuple. Usability for databases ______________________ For simple databases which enumerate (employee, salary, ...) or (shoesize, height, married) as example "database"s, namedtuple is ok. As soon as you write a real database implementation with no fixed layout, you get into trouble. Easy database approach: You define a dbtable as a collection of tuples, maybe as a dict with fast index keys. Not a problem with tuples, which are of type tuple. With named tuple, you suddenly see yourself creating namedtuple instead. But those namedtuple records cannot be pickled when used as a replacement of regular tuples, because they now have a dynamically created type, and extra actions are necessary to make it possible to pickle those. From the documentation: EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, title, department, paygrade') This implicitly suggests that namedtuple is the tool of choice to support databases in Python. Wrong! It supports simple, fixed data structures in Python. For instance, you can use it to define a fixed structural record to define the general description of a database field and its attributes. For a database itself instead, this is very wrong. Nobody uses a fixed class definition to define the structure of some database tables. Instead, you use a generic approach with a row class that describes what a row is, but dynamically. So why all this rant? ___________________ What I'm trying to explain is that namedtuple should be closer to tuple! A namedtuple should not be an explicit class with instances, but a generic subclass of tuple, for all namedtuples. Then, if the user decides to use a namedtuple to build his own class upon that, fine. He then might want to do everything needed to support and pickle his class. But the namedtuple should be a singe (maybe builtin) class that is just a tuple with field names, nothing more. Implementation idea (roughly) _____________________________ Whatever a namedtuple does, it should behave as closely as possible to a tuple, just providing attribute names. Pickling support should be so that the user does not need to know that a namedtuple has a special class. Actually, there should be only a generic class, and the namedtuple "class" is a template instance that just holds the names. Those names could go into some registry or whatever. The only interesting thing about a namedtuple is the set of names used. This set of names is not eligible to enforce the whole import machinery, the associated problems etc. The set of attribute names defines the namedtuple, and that's it. If it is necessary to have class instances like today, ok. But there is no need to search that class in a pickle! Instead, the defining set of attributes could be pickled (uniquely stored by tuple comparison), and the class could be re-created on-the-fly at unpickling time. Conclusion ___________ I love namedtuple, and I hate it. I want to get rid of the second half of this sentence. Let us invent one that does not enforce class behavior. I am thinking of a prototype... cheers - chris p.s.: there is a lot about database design not mentioned here. -- Christian Tismer :^) <mailto:tismer@stackless.com> Software Consulting : Have a break! Take a ride on Python's Karl-Liebknecht-Str. 121 : *Starship* http://starship.python.net/ 14482 Potsdam : PGP key -> http://pgp.uni-mainz.de phone +49 173 24 18 776 fax +49 (30) 700143-0023 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/

4 9

Line continuations with comments
by Ron Adam June 10, 2013

June 10, 2013

There were a few people who liked the idea of having comments after a line continuation. I was able to make a small patch that removed the some of the restrictions on the '\' for testing some ideas which does the following. * Allow a line to continue on the same line. * Skips comments before checking for the new line after a back slash. Here are some examples... These are technically the same. >>> 'aaa' \ ... 'bbb' \ ... 'ccc' 'aaabbbccc' >>> 'aaa' \ 'bbb' \ 'ccc' 'aaabbbccc' Yes there is't much need for this, but I wanted to see if it would work and if the test suit passes. It does. ;-) You can put a comment after a line continuation. >>> 'aaa' \# one ... 'bbb' \# two ... 'ccc' # three 'aaabbbccc' Works with expressions too. >>> result = \ ... + 111 \# A ... + 222 \# B ... + 333 \# C ... + 444 # D >>> result 1110 But if it has a space between the \ and the #, the line is continued on the same line instead of the following line. >>> 'aaa' \ #comment 'aaa' The reason \# works, but not \ #, is when the comment comes directly after the back slash, it's removed and leaves a (backslash + new-line) pair. Removing the white space before the new line check caused some errors in the test suite. I haven't figured out why yet. So this doesn't do that for now. Currently you get this if you try any of these examples. >>> 'abc' \#comment File "<stdin>", line 1 'abc' \#comment ^ SyntaxError: unexpected character after line continuation character Only one of pythons tests fail, and I don't think it's related. test test_urllib2_localnet failed See the diff below if you want to play with it. It's not big. Cheers, Ron diff -r 155e6fb309f5 Parser/tokenizer.c --- a/Parser/tokenizer.c Tue May 21 21:02:04 2013 +0200 +++ b/Parser/tokenizer.c Tue May 21 22:10:31 2013 -0500 @@ -1391,18 +1391,31 @@ again: tok->start = NULL; + + c = tok_nextc(tok); + + /* Check if continuing line */ + if (tok->cont_line == 1 && c == '\n') { + tok->cont_line = 0; + c = tok_nextc(tok); + } + /* Skip spaces */ - do { + while (c == ' ' || c == '\t' || c == '\014') { c = tok_nextc(tok); - } while (c == ' ' || c == '\t' || c == '\014'); + tok->cont_line = 0; + } /* Set start of current token */ tok->start = tok->cur - 1; /* Skip comment */ - if (c == '#') + if (c == '#') { while (c != EOF && c != '\n') c = tok_nextc(tok); + tok_backup(tok, c); + goto again; + } /* Check for EOF and errors now */ if (c == EOF) { @@ -1641,12 +1654,6 @@ /* Line continuation */ if (c == '\\') { - c = tok_nextc(tok); - if (c != '\n') { - tok->done = E_LINECONT; - tok->cur = tok->inp; - return ERRORTOKEN; - } tok->cont_line = 1; goto again; /* Read next line */ }

9 12

Re: [Python-ideas] Ordering keyword dicts
by Nick Coghlan June 10, 2013

June 10, 2013

(migrating thread from python-dev to python-ideas) On 10 June 2013 13:13, Alexander Belopolsky <alexander.belopolsky(a)gmail.com> wrote: > > On Sun, May 19, 2013 at 1:47 AM, Guido van Rossum <guido(a)python.org> wrote: >> >> I'm slow at warming up to the idea. My main concern is speed -- since >> most code doesn't need it and function calls are already slow (and >> obviously very common :-) it would be a shame if this slowed down >> function calls that don't need it noticeably. > > > Here is an idea that will not affect functions that don't need to know the > order of keywords: a special __kworder__ local variable. The use of this > variable inside the function will signal compiler to generate additional > bytecode to copy keyword names from the stack to a tuple and save it in > __kworder__. With that feature, an OrderedDict constructor, for example > can be written as > > def odict(**kwargs): > return OrderedDict([(key, kwargs[key]) for key in __kworder__]) The problem is that this is too late to help. To help folks understand the *technical* (rather than conceptual) limitations that are at issue in only *sometimes* ordering the keyword arguments here's an overview of the way the binding of arguments to parameters currently works: 1. In the calling bytecode, the arguments are collected together on the stack as positional arguments and keyword arguments: >>> def f(): ... return call(1, 2, *(3, 4), x=1, y=2, **{'z':3}) ... >>> dis(f) 2 0 LOAD_GLOBAL 0 (call) 3 LOAD_CONST 1 (1) 6 LOAD_CONST 2 (2) 9 LOAD_CONST 3 ('x') 12 LOAD_CONST 1 (1) 15 LOAD_CONST 4 ('y') 18 LOAD_CONST 2 (2) 21 LOAD_CONST 8 ((3, 4)) 24 BUILD_MAP 1 27 LOAD_CONST 5 (3) 30 LOAD_CONST 7 ('z') 33 STORE_MAP 34 CALL_FUNCTION_VAR_KW 514 (2 positional, 2 keyword pair) 37 RETURN_VALUE The various "CALL_FUNCTION*" opcodes in CPython almost always end up passing through a snippet like the following in ceval.c [1,2] (there are a couple of exceptions related to optimisation of calls that only involve positional arguments and parameters): if (PyCFunction_Check(func)) { PyThreadState *tstate = PyThreadState_GET(); C_TRACE(result, PyCFunction_Call(func, callargs, kwdict)); } else result = PyObject_Call(func, callargs, kwdict); [1] http://hg.python.org/cpython/file/default/Python/ceval.c#l4389 [2] http://hg.python.org/cpython/file/default/Python/ceval.c#l4484 You can see a couple of things here: * That *any* collection of arguments, regardless of syntax, is reduced to a positional argument tuple and a keyword argument dictionary before handing it over to the callable to handle the binding of arguments to parameters. * That this applies even to the optimised fast path that lets functions implemented in C avoid some of the overhead associated with the method dispatch machinery. 2. In the called object, the supplied arguments are bound appropriately as parameters. There's a lot more variability in how this happens. C functions usually use PyArg_ParseTuple or PyArg_ParseTupleAndKeywords. Python functions handle it as an implicit part of the frame initialization based on the function metadata. Guido's speed concern is specifically with any approach which requires the calling code to *always* check callables to see whether they want an ordered dictionary or not. Before we proceed to considering more exotic design ideas that require an "ordered or not" decision at the call sites, this concern should be validated by someone trying it out and checking the macro benchmark suite for the consequences. This can be done without actually making it possible to define functions that require order preservation - since we're mostly interested in the performance consequences when such a flag *isn't* set on the callables, an extra redundant check against "PyCFunction_GET_FLAGS(func) & 0x8000" (for PyCFunction instances), falling back to something like "PyObject_GetAttrId(__name__)" (for everything else), for all code paths in ceval.c that lead to creation of a kwdict instance should suffice. It may be that the cost of the flag check is swamped by the cost of actually creating the keyword dictionary, in which case the runtime check would be a preferable design choice, since function *invocation* wouldn't need to change, only function declarations. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

3 5

Re: [Python-ideas] A Simpler Enum For Python 3
by Eli Bendersky June 8, 2013

June 8, 2013

On Sat, Jun 8, 2013 at 11:04 AM, andrew cooke <andrew(a)acooke.org> wrote: > > > or, you know, you could have documented the design in the pep? isn't that > what's supposed to happen? > Did you read the PEP before you started spreading your FUD? http://www.python.org/dev/peps/pep-0435/#not-having-to-specify-values-for-e… > > if you want a secretay, pay them. > > I have absolutely no idea what you're talking about here. Eli > andrew > > > On Sat, Jun 08, 2013 at 09:30:33AM -0700, Eli Bendersky wrote: > > > Not sure if this is appropriate or useful, so won't repeat, but I > wrote an > > > alternative Enum implementation for Python 3, which is available at > > > https://github.com/andrewcooke/simple-enum > > > > > > > > It may have been appropriate and useful if you gave us the benefit of the > > doubt and wouldn't assume we are oblivious to the possibility of such an > > approach. Hint: it had seen probably at least 100 emails of discussion in > > this list (months back). Yes, it's way more fun to write flamy rants and > > new code than digging in mailing list archives; yet, language design > often > > requires much of the latter as well. > > > > Other than that, it's perfectly fine to have alternative enum proposals > out > > in the wild. I sincerely wish you well in the sense that if someone > doesn't > > like the stdlib enum and wants to use something like your bnum, they will > > find it online and will happily use it. > > > > Eli >

1 0

A Simpler Enum For Python 3
by andrew cooke June 8, 2013

June 8, 2013

Hi, Not sure if this is appropriate or useful, so won't repeat, but I wrote an alternative Enum implementation for Python 3, which is available at https://github.com/andrewcooke/simple-enum There's a discussion of how it differes from PEP 0435 at https://github.com/andrewcooke/simple-enum#discussion and some examples of teh differences at https://github.com/andrewcooke/simple-enum#things-you-can-do-with-the-simpl… Also, I have an almost-complete codebase that extends Ethan's "standard" implementation with some of the ideas in this design, available at https://github.com/andrewcooke/bnum - I have abandoned that project in favour of the one mentioned first, above, but it may be of interest if you want to consider modifying the PEP design slightly (in other words: bnum includes inheritance, simple-enum does not). Cheers, Andrew

3 2

Allow using ** twice
by Ram Rachum June 7, 2013

June 7, 2013

I'd like to be able to use ** twice (using 2 different dicts) when calling a function, and have them both feed arguments into the function.

25 59

Re: [Python-ideas] Allow using ** twice
by Vernon D. Cole June 6, 2013

June 6, 2013

On Thu, Jun 6, 2013 at 12:47 PM, <python-ideas-request(a)python.org> wrote: > > Does it? All the languages which only allow += on immutable values (e.g. > Java, C, Javascript) obviously don't mutate anything. Eh? It's been twenty years or so since I programmed C for a living, but I seem to remember that a C variable is actually only a handy label for a memory location, and that therefore *everything* is mutable -- even when it shouldn't be. And += was invariably implemented in machine language by a two-argument add straight to that memory location. Some of the early C compilers would even allow you to use a literal on the left side, like "2 += 3" which would change the value of the literal '2' to become five. Made for some interesting bugs! Ignoring all of that, if + worked on dictionaries, I would expect that duplicate keys would receive the value from the right-hand argument, and that the operation would create a new dictionary. (and I think it would be a great +1 idea.) -- Vernon Cole

3 3

applying the PSF Code of Conduct to this mailing list
by Brett Cannon June 6, 2013

June 6, 2013

Titus and I have discussed it and we have decided to apply the PSF's Code of Conduct (http://hg.python.org/coc) to this mailing list. In a nutshell everyone should be open, respectful, and considerate to each other, i.e. don't be rude. If you think someone is not following the CoC, either publicly let them know you think they are not (obviously do this politely; publicly notifying someone is to help others on the list understand what may be considered inappropriate behaviour, not to chastise them in front of a crowd) or email python-ideas-owner@ privately with a link to the email in question from the archives and we will handle it w/o disclosing who brought forward the complaint. Anyone found repeatedly not following the CoC will be banned from the list. You can always ask to be allowed back on based on the discretion of the list owners. Because this is a new policy, no one is starting with any strikes against them. Everyone is assumed to have been polite and on good terms up to this point. In the 6.5 year history of this list there is possibly only one instance where this would have really come into play in removing someone from this list so this isn't expected to be used very often. But we felt we wanted something in place to remind folks that we expect all subscribers to be respectful of one another and to have something to use as a guideline for removals if it came to that.

11 22

without(str(bytes))
by Daniel Holth June 6, 2013

June 6, 2013

Can I have a context manager that disables str(bytes) just for part of my code, the same as python -bb? with bb: serialize_something()

6 7

PEP 443 - multiple types registered
by João Bernardo June 5, 2013

June 5, 2013

I was reading the recently accepted PEP 443 and this is one of the examples: @fun.register(float) @fun.register(Decimal) def fun_num(arg, verbose=False): if verbose: print("Half of your number:", end=" ") print(arg / 2) Wouldn't a tuple of types be more suitable? I mean, programatically you will not apply the decorators from a list, but define the function and then write a loop: for t in some_types: fun.register(t, fun_num) which is less pleasant for the eyes. I would rather do: @fun.register((float, Decimal)) def fun_num(arg, verbose=False): ... This makes a lot of sense when you think that Python functions and constructs dealing with types usually also deal with tuples of types. E.g: isinstance(foo, (Bar, Baz)) except (Foo, Bar) as baz: Last thought: People are used with decorators returning a decorated function. This is not the case here... For some, it may be confusing at first. João Bernardo

6 7

Re: [Python-ideas] PEP 426, YAML in the stdlib and implementation discovery
by Calvin Spealman June 5, 2013

June 5, 2013

Right, "This specific idea isn't worth its specific costs in exchange for its specific benefits" is the same as "lets give up!" On Tue, Jun 4, 2013 at 7:29 AM, Philipp A. <flying-sheep(a)web.de> wrote: > 2013/6/4 Calvin Spealman <ironfroggy(a)gmail.com> > >> At this point I would argue against any new modules by default. How does >> the cost and liability of yaml in the standard library make up for such a >> ring benefit as excluding one little line from my requirements.txt? >> > how about we stop innovating completely and just give up python? there’s > so many languages out there, i can’t see the cost and liability of > maintaining python outweigh the small benefit of not using java. > > in other words: are you serious? > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

9 20

Status of PyYAML - Re: Proposal for closing ticket pyyaml/issue/11
by Kirill Simonov June 5, 2013

June 5, 2013

Hi Stefan, I'm cross-posting this to the python-ideas list. Regarding the status of PyYAML -- after 10 releases over 6 years, I consider it stable enough so that I don't need to pay much attention to it anymore. I'm migrating the project, including the issues database, from Trac to Bitbucket, but haven't finished it yet since Bitbucket added issues import/export API only a month ago and I didn't have a change to look at it. On discussion on python-ideas, I looked through it, but I'm not sure how I could contribute. Does it make sense to have YAML support in stdlib? Definitely, the format is popular enough and has some unique advantages over other options. But to the next question, how it should be done, I don't have a good answer. Would you include PyYAML+libyaml directly into the stdlib? It's a big chunk of C, Python and Pyrex/Cython code and I doubt core developers are willing to accept it. Besides, PyYAML API was designed for Python 2.3 and may feel outdated for Python 3.5. So you need to rewrite low-level parsing and emitting code to C+Python API and design a new high-level extension API, but who's going to do all this work? Thanks, Kirill On 06/05/2013 01:43 AM, Stefan Drees wrote: > Hi Kirill, > > as we have interesting discussions w.r.t. YAML and the python stdlib, I > thought you might be interested to know. > > Vinay Sajip was so nice, as to point to an open ticket at [1] as sample > and asked (in other words) "if this project is alive and well" ;-) > (he also noted that there """are 65 open issues of type "defect" > relating to PyYAML, and 15 such relating to libyaml.""" c.f. [2] > > As it seems to be unclear how and where issues are handled for PyYAML > and libyaml (TRAC on pyyaml.org or bitbucket repo?) I spent some minutes > to remove the common fairy tale that "arbitrary JSON (with unique keys) > is automatic correctly parseable YAML" and wrote[3]. > > Maybe you can use some of the wording at [3], to close the ticket and > also others in relation to this misconception, that YAML = JSON and > unique keys. > > I personally would really enjoy, if you could participate in these > discussions on python-ideas as it would certainly be beneficial, to read > your opinions and advice based on your experiences. > > Please consider it at least. It would be great! > > References: > > [1]: https://bitbucket.org/xi/pyyaml/issue/11/valid-json-not-being-loaded > [2]: http://mail.python.org/pipermail/python-ideas/2013-June/021088.html > [3]: http://mail.python.org/pipermail/python-ideas/2013-June/021095.html > > All the best, > Stefan.

2 1

Timing hefty or long-running blocks of code
by Steven D'Aprano June 5, 2013

June 5, 2013

The timeit module is great for timing small code snippets, but it is rather inconvenient for timing larger blocks of code. It's also over-kill: under normal circumstances, there is little need for the heroic measures timeit goes through to accurately time long-running code. When people want to time something that doesn't require timeit, they have to write their own timer boilerplate: from time import time t = time() do_this() do_that() print(time() - t) Although only a few lines of boilerplate, this has a few problems, the biggest two being: * the user has to decide which of the many clocks provided by Python is suitable (hint: time.time is not necessarily the best); * this is inconvenient at the interactive interpreter, since the timer is started before the code being typed has been entered. I propose a context manager in the timeit module: with Stopwatch(): do_this() do_that() => results are automatically printed, or available for programmatic access I have had a recipe for this on ActiveState for about 18 months, where it is relatively popular: http://code.activestate.com/recipes/577896-benchmark-code-with-the-with-sta… and I have extensively used it interactively, in Python 2.6 through 3.3 inclusive. The latest version can be found here: https://code.google.com/p/my-startup-file/source/browse/timer.py This solves the two problems listed above: * the Stopwatch can use the most appropriate timer by default, and allow the user to override it if they so choose; * the Stopwatch will not start until the with-block is actually entered. Is there interest in seeing this in the standard library? -- Steven

6 8

PEP 426, YAML in the stdlib and implementation discovery
by Philipp A. June 4, 2013

June 4, 2013

Hi, reading PEP 426<http://www.python.org/dev/peps/pep-0426/#switching-to-a-json-compatible-for…>, I made a connection to a (IMHO) longstanding issue: YAML not being in the stdlib. I’m no big fan of JSON, because it’s so strict and comparatively verbose compared with YAML. I just think YAML is more pythonic, and a better choice for any kind of human-written data format. So i devised 3 ideas: 1. *YAML in the stdlib* The stdlib shouldn’t get more C code; that’s what I’ve gathered. So let’s put a pure-python implementation of YAML into the stdlib. Let’s also strictly define the API and make it secure-by-naming™. What i mean is let’s use the safe load function that doesn’t instantiate user-defined classes (in PyYAML called “safe_load”) as default load function “load”, and call the unsafe one by a longer, explicit name (e.g. “unsafe_load” or “extended_load” or something) Let’s base the parser on generators, since generators are cool, easy to debug, and allow us to emit and test the token stream (other than e.g. the HTML parser we have) 2. *Implementation discovery* People want fast parsing. That’s incompatible with a pure python implementation. So let’s define (or use, if there is one I’m not aware of) a discovery mechanism that allows implementations of certain APIs to register themselves as such. Let “import yaml” use this mechanism to import a compatible 3rd party implementation in preference to the stdlib one Let’s define a property of the implementation that tells the user which implementation he’s using, and a way to select a specific implementation (Although that’s probably easily done by just not doing “import yaml”, but “import std_yaml” or “import pyyaml2”) 3. Allow YAML to be used besides JSON as metadata like in PEP 426. (so including either pymeta.yaml or pymeta.json makes a valid package) I don’t propose that we exclusively use YAML, but only because I think that PEP 426 shouldn’t be hindered from being implemented ASAP by waiting for a new std-library to be ready. What do you think? Is there a reason for not including a YAML lib that i didn’t cover? Is there a reason JSON is used other than YAML not being in the stdlib?

19 39

Re: [Python-ideas] PEP 426, YAML in the stdlib and implementation discovery
by Calvin Spealman June 4, 2013

June 4, 2013

On Tue, Jun 4, 2013 at 8:38 AM, Paul Moore <p.f.moore(a)gmail.com> wrote: > On 4 June 2013 12:26, Calvin Spealman <ironfroggy(a)gmail.com> wrote: > >> At this point I would argue against any new modules by default. How does >> the cost and liability of yaml in the standard library make up for such a >> ring benefit as excluding one little line from my requirements.txt? > > > Windows users without compilers can't "just add one little line". IIRC, > Windows users *with* compilers have to fiddle around with dependent > libraries in a messy way. Users without compilers working on prerelease > versions of Python can't find prebuilt binaries anyway. Corporate > environments may not allow 3rd party libraries. Some environments require > pure-python (non-stdlib) code. Etc, etc. > The proposal was about adding, explicitly, only a pure-python YAML library, so the compiler and binary related arguments don't apply here. Also, the fact that we don't have practical packaging solutions to this is something that, hopefully, we'll fix and so I think, until then, adding new modules should be scrutinized with the question: Are we just doing this to get around packaging limitations? New modules have a pretty high bar to adoption in any case. Let's not make > it impossibly high by assuming (wrongly) that there are no difficulties > involved in external dependencies. > > Paul. > -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://techblog.ironfroggy.com/ Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

1 0

iterator/stream-based JSON API
by Daniel Holth June 3, 2013

June 3, 2013

When will the stdlib get a decent iterator/stream-based JSON API? For example automated packaging tools may be parsing a lot of JSON but ignoring most of it, and it would be lovely to say "if key not in interesting: skip_without_parsing()". Or to go straight from parse to domain objects without putting the whole thing in an intermediate dict. http://code.google.com/p/jsonpull/ is a nice looking but woefully undocumented Java one, and https://crate.io/packages/ijson/ is one with a much different API that is for Python.

6 9

Re: [Python-ideas] Timing hefty or long-running blocks of code
by Clay Sweetser June 3, 2013

June 3, 2013

I quite like this proposal. To me, the main advantage of such a context manager/decorator over the current timeit module is that it can be used for pieces of code that either can't or shouldn't be arbitrarily repeated. For example, I am currently writing a plugin for the sublime text editor, and it would be very convenient for me to be able to get a quick summery of timing statistics for functions that make use of the editor's plugin api to modify documents. My only suggestion for the proposed function is to allow the output of statistics similar to the ones the timeit module allows, such as the mean runtime of a block of code. Sincerely, Clay Sweetser

4 5

ThreadPool worker function
by Luca Sbardella June 3, 2013

June 3, 2013

While using the ThreadPool in the multiprocessing.pool module, I needed to use a different implementation for the *worker* function. Since there is no way to overwrite it directly, I'm overwriting the Process attribute with a method: def myworker(...): ... class ThreadPool(pool.ThreadPool): def Process(self, target=None, **kwargs): return Thread(target=myworker, **kwargs) which is good enough, works for 2.6 and up. On the other hand, *worker* could be defined as a class attribute in the same way as the Process. Something like this: def default_worker(...): ... class Pool(object): Process = Process worker = default_worker ... Less verbose and less of a hack IMO.

1 0

Loaded questions
by Guido van Rossum June 2, 2013

June 2, 2013

On Sun, Jun 2, 2013 at 7:11 PM, Daniel Holth <dholth(a)gmail.com> wrote: > When will the stdlib get a decent iterator/stream-based JSON API? As a reminder, a loaded question is not a good way to suggest a new feature. If you want a feature, great. If you don't feel you have the skills to implement it, you can still play*. But if you want a feature, please say something like "I think it would be cool if the stdlib had X" and we can discuss whether we all feel the same way. To most people, "when will the stdlib get a decent X" reads as either implying "the stdlib has a lousy X" or "the stdlib is stupid (or whatever negative adjective) for not having X." And many of the folks here who together have made the stdlib what it is and never missed X will feel offended. Do you think it is a good thing to start a request by offending the person you are asking? (Don't answer. :-) It's possible you didn't mean this at all. Likely, even. So next time, think about how the words you use might be interpreted by others, and leave your loaded questions at home. Please. :-) __________ * Though if nobody can be found who wants a feature badly enough to implement it, it won't happen, and you shouldn't complain about it -- we're all volunteers here, and nobody has enough time to implement every desirable feature. -- --Guido van Rossum (python.org/~guido)

1 0