Mailman 3 April 2016 - Python-ideas

Re: [Python-ideas] Adding a pipe function to functools
by Wes Turner April 15, 2016

April 15, 2016

"[Python-ideas] The pipe protocol, a convention for extensible method chaining" https://mail.python.org/pipermail/python-ideas/2015-May/033673.html https://groups.google.com/forum/m/#!topic/python-ideas/4HgpT5yE06o On Apr 15, 2016 9:51 AM, "Ed Minnix" <egregius313(a)gmail.com> wrote: Hello, I have been looking over the toolz library and one of the functions I like the most is pipe. Since most programmers are familiar with piping (via the Unix `|` symbol), and it can help make tighter code, I think it would be nice to add it to the standard library (such as functools). - Ed Minnix _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/

1 0

Re: [Python-ideas] Module lifecycle: simple alternative to PEP 3121/PEP 489
by Nikita Nemkin April 15, 2016

April 15, 2016

On Thu, Apr 14, 2016 at 2:57 PM, Petr Viktorin <encukou(a)gmail.com> wrote: > On 04/14/2016 10:23 AM, Nikita Nemkin wrote: >> Reading PEP 3121/PEP 489 I can't stop wondering, why do extension >> modules require such specialized lifecycle APIs? Why not just let >> them subclass ModuleType? (Or any type, really, but ModuleType might >> be a good place to define some standard behavior.) > > Good question. > I'll list some assumptions; if you don't share them we can talk more > about these: > - somethings are easy in Python, but not pleasant to do correctly* in C: > - subclassing > - creating simple objects > - setting attributes > - most modules don't need special functionality > > * ("Correctly" includes things like error checking, which many > third-party modules skimp on.) > > Most of the API is in the style of "leave this NULL unless you need > something special", i.e. simple things are easy, but the complex cases > are possible. Creating custom ModuleType subclasses is possible, but I > definitely wouldn't want to require every module author to do that. > > A lot of the API is convenience features, which are important: they do > the right thing (for example, w.r.t. error checking), and they're easier > to use than alternatives. This makes the API grows into several layers; > for example: > - m_methods in the PyModuleDef > - PyModule_AddFunctions > - PyCFunction_NewEx & PyObject_SetAttrString > Usually you just set the first, but if you need more control (e.g. > you're creating modules dynamically), you can use the lower-level tools. > > Your suggestion won't really help with this kind of complexity. I agree that C API is quite difficult and dangerous to use directly (that's why I always use Cython), but I don't agree that separate module init API makes things simpler. PyModuleDef appears to be PyTypeObject surrogate with similar (or identical?) semantics. That's an extra concept to learn about, an extra bit of documentation to consult when writing new code. PyTypeObject, on the other hand, is fundamental and unavoidable, regardless of module init system. Practically speaking, PyModuleDef_Init(&spam_def) and PyType_Ready(&spam_type) differ only in the number of zeroes in their respective static structs. There's also PyType_FromSpec (stable ABI), which might appeal to some people more than PyType_Ready. > Oh, and there is a technical reason against subclassing ModuleType > unless necessary: Custom ModuleType subclasses cannot be made to work > with runpy (i.e. python -m). For ModuleType (the ones without a custom > create_module in the current API), this doesn't *currently* work, but > the PEPs were written so that it's "just" a question of spending some > development effort on runpy. I didn't even know that python -m supported extension modules. >> Module instance naturally encapsulates C level module state. >> GC/finalization happens just like for any other object. PEP 3121 >> becomes redundant. > > It doesn't become fully redundant: m_methods is still useful. ModuleType subclasses have tp_methods. A little check in PyType_Ready can make it behave like PyModuleDef.m_methods. Modules don't need normal methods anyway. > With your suggestion, I fear that someone would quickly come up with a > macro to automate creating simple ModuleType instances, and at that > point the API would be as complex as it is now, but every module > instance would now also have an extra ModuleType subclass – and I don't > think that's either simpler or more effective. One or two standard macros that help with ModuleType subsclassing are in order. Something like PyModule_HEAD_INIT, whose usage is already obvious from the name. > I personally think that your suggestion wouldn't make the API > substantially simpler, assuming you would keep it as robust and easy to > use as the current solution. And if you would want to maintain backwards > compatibility (even with only the pre-PEP 489 state), it would be even > harder. > > Many people thought about the current APIs, and (almost) all of the > unpleasant decisions we had to make do have their reasons. (And if you > ask about a more specific decision, I can give you the specific reasons.) Regarding backward compatibility, PEP 3121 and most of PEP 489 will work just fine as a facade to ModuleType subclass. I understand that existing PEPs are well researched and there's no real incentive to change. My proposal is more of a hypothetical kind.

4 8

PEP8 operator must come before line break
by SW April 15, 2016

April 15, 2016

Hi, PEP8 says that: "The preferred place to break around a binary operator is after the operator, not before it." This is ignored in the multiline if examples, and seems to generally be a bad idea as it negatively impacts clarity. For example, the following seems much clearer as the entire line does not need to be scanned to see the intent- only the start of the line is needed to see how the different properties are used for filtering: mylist = [ item for item in mylist if item['property'] == 2 and item['otherproperty'] == 'test' ] The alternative seems less clear: mylist = [ item for item in mylist if item['property'] == 2 and item['otherproperty'] == 'test' ] If this recommendation remains in force, it would be good to: 1. Follow it in the style guide. 2. Provide a rationale for it, as currently it seems arbitrary and unhelpful. Just raising this as it has bitten me a couple of times with style checkers recently. Thanks, S

16 24

Module lifecycle: simple alternative to PEP 3121/PEP 489
by Nikita Nemkin April 14, 2016

April 14, 2016

Reading PEP 3121/PEP 489 I can't stop wondering, why do extension modules require such specialized lifecycle APIs? Why not just let them subclass ModuleType? (Or any type, really, but ModuleType might be a good place to define some standard behavior.) Module instance naturally encapsulates C level module state. GC/finalization happens just like for any other object. PEP 3121 becomes redundant. Two-step initialization (PEP 489) can be achieved by defining a new kind of PyInit_XXX entry point, returning a module *type*, instead of a module *instance*. No extra API needed beyond that! Now, importer can simply instantiate this module type, passing __name__, __file__ and the rest. ModuleType.tp_new will perform attribute init, sys.modules registration etc. OR the importer can manually pull tp_new/tp_init/attribute setup, supplanting type_call. (This is closer to the current way of doing things.) Actual module initialization ("executing the module body") happens in tp_init. reload() is equivalent to calling tp_init again. Subinterpreter interaction becomes transparent: every interpreter instantiates its own module copy. "Singleton" modules with external global state should fail second instantiation (maybe by deriving from a special SingletonModuleType subclass that will handle it for them). Additionally, custom module type allows fine grained attribute access control (aka metamodules), useful to many complex modules. C synchronized module "variables" become super-easy to define (tp_members). For lazy loading and importing there's tp_getattro, tp_getset, etc. One problem not solved by this approach (nor the current approach) is module state access from methods of extension types. At least two solutions are possible: 1. Look up the module by name (sys.modules) or type (new per-interpreter cache). 2. Define a new METH_XXX calling convention (or flag) and pass both PyCFunctionObject.m_self and PyCFunctionObject.m_module to the C level method implementations. Both can be implemented, #1 being simple and #2 being proper. What do you think?

2 1

Dictionary views are not entirely 'set like'
by Joshua Morton April 14, 2016

April 14, 2016

Let's start with a quick quiz: what is the result of each of the following (on Python3.5.x)? {}.keys() | set() # 1 set() | [] # 2 {}.keys() | [] # 3 set().union(set()) # 4 set().union([]) # 5 {}.keys().union(set()) # 6 If your answer was set([]), TypeError, set([]), set([]), set([]), AttributeError, then you were correct. That, to me, is incredibly unintuitive. Next up: {}.keys() == {}.keys() # 7 {}.items() == {}.items() # 8 {}.values() == {}.values() # 9 d = {}; d.values() == d.values() # 10 True, True, False, False. Numbers 1, 2, 4, 5 are expected behavior. 3 and 6 are not, and 7-10 is up for debate.[1] First thing first, the behavior exhibited by #3 is a bug (or at least it probably should be, and one I'd be happy to fix. However, before doing that I felt it would be good to propose some questions and suggestions for how that might be done. There are, as far as I can tell, two reasons to use a MappingView, memory efficiency or auto-updating (a view will continue to mirror changes in the underlying object). I'll focus on the first because the second can conceivably be solved in other ways. Currently, if I want to union a dictionaries keys with a non-set iterable, I can do `{}.keys() | []`. This is a bug[2], the set or operator should only work on another set. That said, fixing this bug removes the ability to efficiently or keys with another iterable, `set({}.keys()).update([])` for efficiency, or `set({}.keys()).union([])` for clarity. Fixing this is simply a matter of adding a `.union` method to the KeysView (and possibly to the Set abc as well). Although that may not be something that is wanted. The issue, as far as I can tell, is whether we want people converting from MappingViews to "primitives" as soon as possible, or if we want to encourage people people to use the views for as long as possible. There are arguments on both sides: encouraging people to use the views causes these objects to become more complex, introducing more surface area for bugs, more code to be maintained, etc. Further, there's there is currently one obvious way to do things, convert to a primitive, if you're doing any complex action. On the other hand, making MappingViews more like the primitives they represent has positives for performance, simplifies user code, and would probably make testing these objects easier, since many tests could be stolen from test_set.py. My opinion is that the operators on MappingViews should be no more permissive than their primitive counterparts. A KeysView is in various ways more restrictive than a set, so having it be also occasionally less restrictive is surprising and in my opinion bad. This causes the loss of an efficient way to union a dict's keys with a list (among other methods). I'd then add .union, .intersection, etc. to remedy this. This solution would bring the existing objects more in line with their primitive counterparts, while still allowing efficient actions on large dictionaries. In short: - Is #3 intended behavior? - Should it (and the others be)? - As a related aside, should .union and other frozen ops be added to the Set interface? - If so, should the fix solely be a bugfix, should I do what I proposed, or something else entirely? - More generally, should there be a guiding principle when it comes to MappingViews and similar special case objects? [1]: There's some good conversation in this prior thread on this issue https://mail.python.org/pipermail/python-ideas/2015-December/037472.html. The consensus seemed to be that making ValuesViews comparable by value is technically infeasible (O(n^2) worst case), while making it comparable based on the underlying dictionary is a possibility. This would be for OrderedDict, although many of the same arguments apply for a normal dictionary. [2]: Well, it probably should be a bug, its explicitly tested for ( https://github.com/python/cpython/blob/master/Lib/test/test_dictviews.py#L1…), whereas sets are explicitly tested for the opposite functionality ( https://github.com/python/cpython/blob/master/Lib/test/test_set.py#L92) Thanks, I'm looking forward to the feedback, Josh

7 11

Dunder method to make object str-like
by Chris Angelico April 13, 2016

April 13, 2016

This is a spin-off from the __fspath__ discussion on python-dev, in which a few people said that a more general approach would be better. Proposal: Objects should be allowed to declare that they are "string-like" by creating a dunder method (analogously to __index__ for integers) which implies a loss-less conversion to str. This could supersede the __fspath__ "give me the string for this path" protocol, or could stand in parallel with it. Obviously str will have this dunder method, returning self. Most other core types (notably 'object') will not define it. Absence of this method implies that the object cannot be treated as a string. String methods will be defined as accepting string-like objects. For instance, "hello"+foo will succeed if foo is string-like. Downside: Two string-like objects may behave unexpectedly - foo+bar will concatenate strings if either is an actual string, but if both are other string-like objects, depends on the implementation of those objects. Bikeshedding: 1) What should the dunder method be named? __str_coerce__? __stringlike__? 2) Which standard types are sufficiently string-like to be used thus? 3) Should there be a bytes equivalent? 4) Should there be a format string "coerce to str"? "{}".format(x) is equivalent to str(x), but it might be nice to be able to assert that something's stringish already. Open season!

22 91

Modifying yield from's return value
by Bar Harel April 12, 2016

April 12, 2016

I asked a question in stackoverflow <http://stackoverflow.com/q/36582679/1658617> regarding a way of modifying yield from's return value. There doesn't seem to be any way of modifying the data yielded by the yield from expression without breaking the yield from "pipe". By "pipe" I mean the fact that .send() and .throw() pass to the inner generator. Useful cases are parsers as I've demonstrated in the question, transcoders, encoding and decoding the yielded values without interrupting .send() or .throw() and generally coroutines. I believe it would be beneficial to many aspects and libraries in Python, most notably asyncio. I couldn't think of a good syntax though, other than creating a wrapping function, lets say in itertools, that creates a class overriding send, throw and __next__ and receives the generator and associated modification functions (like suggested in one of the answers). What do you think? Is it useful? Any suggested syntax?

2 3

(no subject)
by Keith Curtis April 12, 2016

April 12, 2016

Hello again, Many FOSS communities struggle with sustainability, but Python is very rich so you've clearly figured it out. It is easy to say that bug metrics are going in the wrong direction. It is also easy to do nothing, or to criticize those who are concerned about a systemic issue. Data can be turned into action by leadership and plans. It can be helpful to have people who think 5,000 is a big number. (Automatically closing old issues doesn't take grit and isn't a valid solution.) It seems you could use at least 10 more dedicated devs who focus on the official CPython bug count. As for WebAssembly, if their security is done mostly via their virtual machine, then they won't be able to separate it. However, if Python in the browser can't enable optional access Numpy, etc., then it will be missing the best reasons to use it. Sometimes security can destroy utility. Just because there are some untrustworthy websites doesn't mean all must be. Warm regards, -Keith

8 7

A tuple of various Python suggestions
by Keith Curtis April 11, 2016

April 11, 2016

Hi all, I just discovered this alias and I thought I'd post a few ideas. I wouldn't call myself a Python master yet, but it's an amazing language and my biggest wish is that it was more widely used in the industry. Here are a few suggestions: 1. Decrease the bug count. I recently noticed that there are about 5,400 active bugs in http://bugs.python.org/. That surprised me because I almost never see anyone complain about bugs in Python (compared to the number who complain about bugs in LibreOffice, graphics drivers, Gnome / KDE, etc.) There are a lot of people on this list, and if the brainpower can be applied to practical, known, existing problems, it is a great way to improve Python while also considering more exotic ideas. I can also suggest making a pretty graph of the bug count and putting it on the front page of python.org for greater visibility. 2. Python is somewhat popular on servers, and there is a lot of potential for more. WordPress is easy to use and powerful, but lots of people don't want to program in PHP. Or Javascript, Java, Ruby, etc. Codebases like Whoosh full-text search (https://bitbucket.org/mchaput/whoosh/wiki/Home) are important, but have minimal dev resources as most people are using Lucene / ElasticSearch. The common choice is between 1.3M lines of Java: https://www.openhub.net/p/elasticsearch, containing 1100 todos and 1000 references to "deprecated", or 41K lines of Python written mostly by one person. Hadoop is another big Java project (1.9M lines), and there is even an ecosystem around it. Python interoperates with Hadoop, but it should be possible to build a radically simpler framework that provides the same functionality using Python-native functionality and without all the baggage. Hadoop has several interesting sister projects: a distributed database, scalable machine learning, a high-level data flow language, a coordination service, etc. I'm sure you'd build something smaller, cleaner, faster in many cases, more reliable, etc. 3. It was a sad mistake that Google picked Java over Python for Android. However, there is now a great program called Kivy which allows people to write apps for IOS or Android with one codebase, but it could also use more resources, as for example it doesn't fully support Python 3.x yet. There are 10s of thousands of bugs in the popular Python libraries and I would fix those before proposing more language changes. 4. I enjoy reading about the Python performance improvements, but it is mostly a perception problem with all the existing workarounds. Gnome wrote version 3 of their shell in Javascript because they didn't think Python would be fast enough. Lots of people write Node because it's compiled and "fast". I suggest taking some of the effort working on performance, and spend it on evangelizing to other programmers that Python / Cython / PyPy, etc. are already good enough! There are a lot of programmers out there who would be happier if they could work in Python. 5. It would be great to get Python in the web browsers as an alternative to Javascript. There are a number of projects which convert Python to Javascript, but this would be more direct. LibreOffice ships with a Python interpreter, why can't Firefox and Webkit? ;-) Obviously there are interoperability issues, but it would be great to just side-step all the complexity of Javascript (Here is a server-oriented article, but it gives a flavor: http://notes.ericjiang.com/posts/751) This might sound like a crazy idea, but the engineering problems aren't that hard. 6. In a few cases, there are two many codebases providing the same functionality, and none of them are really doing the job. For example, the de-facto MySQL Python interop library (https://pypi.python.org/pypi/MySQL-python) only supports Python 2.x and appears to be abandoned. There are several other libraries out there with different features, performance, compatibility, etc. and it's kind of a minefield for what should be a basic scenario. It takes leadership to jump in and figure out how to fix it and make one codebase good enough that the others can switch over. 7. Focus more on evolving the libraries rather than the language. I've recently discovered Toolz, which has a more complete set of functional language methods. I think some of them should be included in the official versions. A lot of people don't think Python is good enough for functional programming and this would help. These new routines add complexity, but a newbie doesn't need to write in a functional way, so it obeys the "only pay for what you use" rule. There are a number of under-staffed libraries and frameworks. I see people complain about the YAML parsing library being unmaintained, the default HTTP functionality being difficult and limiting, poor SOAP support, etc. There are a million ways to improve the Python ecosystem without making any language changes. You don't have a big rich company who can pay for thousands of full-time developers working on libraries, but the bug reports are a great way to prioritize. 8. I've yet to find a nice simple free IDE with debugging for Python. I use Atom, but it has primitive debugging. I tried PyCharm but it's very complicated (and not free, and Java). I use Jupyter sometimes also but I'd prefer a rich client app with watch windows, etc. 9. It would be interesting to re-imagine the spreadsheet with a more native Python interface. Pandas and matplotlib are great, but it would be cool to have it in LibreOffice Calc that supports drag and drop, copy and paste, can read and write ODS, etc. (Also, LibreOffice Base is basically unmaintained. I think if 10 Python programmers passionate about databases and GUIs showed up, it could re-invigorate this dead codebase.) 10. Being simple to learn and powerful is very hard. Fortunately, you can break compatibility every 10 years. My only suggestion is to get rid of the __self__ somehow ;-) Regards, -Keith http://keithcu.com/

9 15

simpler weakref.finalize
by Matthew Stoltenberg April 11, 2016

April 11, 2016

Currently, if I want to have weakref.finalize call an object's cleanup method, I have to do something like: class foo: _finalizer = None def __init__(self): self._finalizer = weakref.finalize(self, foo._cleanup, weakref.WeakMethod(self.cleanup)) def __del__(self): self.cleanup() @classmethod def _cleanup(cls, func): func()() def cleanup(self): if self._finalizer is not None: self._finalizer.detach() print('cleaning up') It wouldn't be difficult to have weakref.finalize automatically handle the conversion to WeakMethod and automatically attempt to dereference then call the function passed in.

1 0