Mailman 3 April 2019 - Python-ideas

Provide additional debug info for OSError and WindowsError
by Giampaolo Rodola' April 12, 2019

April 12, 2019

When dealing with C extensions from Python there are circumstances where a function is called and we get an OSError(errno) exception without knowing what exactly went wrong internally. This is especially not obvious on Windows, where multiple MSDN APIs may be invoked within the same C function and we're not sure which one of them failed. There are other times where the underlying C syscall is obvious, but it would still be useful to append some additional information. One example is socket.bind(): https://github.com/python/cpython/blob/56065d4c8ac03042cb7e29ffda9b1ac544a3… In order to work around that in psutil (on Windows) I stored the debug msg string in OSError.filename attribute: https://github.com/giampaolo/psutil/pull/1428/ As such I was thinking that perhaps it would be nice to provide 2 new cPython APIs: PyErr_SetFromErrnoWithMsg(PyObject *type, const char *msg) PyErr_SetFromWindowsErrWithMsg(int ierr, const char *msg) PyErr_SetExcFromWindowsErrWithMsg(PyObject *type, int ierr, const char *msg) With this in place also OSError and WindowsError would probably have to host a new "extramsg" attribute or something (but not necessarily). Thoughts? -- Giampaolo - http://grodola.blogspot.com

2 1

Sorted lists
by Steven D'Aprano April 12, 2019

April 12, 2019

There are quite a few important algorithms which require lists to be sorted. For example, the bisect module, and for statistics median and other quantiles. Sorting a list is potentially expensive: while Timsort is very efficient, it is still ultimately an O(N log N) algorithm which limits how efficient it can be. Checking whether a list is sorted is O(N). What if we could check that lists were sorted in constant time? Proposal: let's give lists a dunder flag, __issorted__, that tracks whether the list is definitely sorted or not: - Empty lists, or lists with a single item, are created with __issorted__ = True; lists with two or more items are created with the flag set to False. - Appending or inserting items sets the flag to False. - Deleting or popping items doesn't change the flag. - Reversing the list doesn't change the flag. - Sorting it sets the flag to True. (The sort method should NOT assume the list is sorted just because the flag is set.) Functions that require the list to be sorted can use the flag as a quick check: if not alist.__issorted__: alist.sort() ... The flag will be writable, so that functions such as those in bisect can mark that they have kept the sorted invariant: bisect.insort(alist, x) assert alist.__issorted__ Being writable, the flag is advisory, not a guarantee, and "consenting adults" applies. People can misuse the flag: alist = [1, 4, 2, 0, 5] alist.__issorted__ = True but then they have nobody to blame but themselves if they shoot themselves in the foot. That's no worse than the situation we have now, were you might pass an unsorted list to bisect. The flag doesn't guarantee that the list is sorted the way you want (e.g. biggest to smallest, by some key, etc) only that it has been sorted. Its up to the user to ensure they sort it the right way: # Don't do this and expect it to work! alist.sort(key=random.random) bisect.insort(alist, 1) If you really want to be sure about the state of the list, you have to make a copy and sort it. But that's no different from the situation right now. But for those willing to assume "consenting adults", you might trust the flag and avoid sorting. Downsides: - Every list grows an extra attribute; however, given that lists are already quite big data structures and are often over-allocated, I don't think this will matter much. - insert(), append(), extend(), __setitem__() will be a tiny bit slower due to the need to set the flag. Thoughts? -- Steven

13 26

Exception for developer errors?
by Stefano Borini April 10, 2019

April 10, 2019

I occasionally found situations where I want to raise an exception for errors that can only arise because the developer made a mistake, for example: - an abstract method is supposed to be reimplemented and its execution is supposed to leave some internal constraints of an object unchanged, but these are instead violated. - an impossible "else" condition after an if/elif, where the else cannot simply happen unless someone really screwed up the internal state of the object. In general, when these cases happen, I use RuntimeError, but other people may choose otherwise. I've also seen ValueError, plain Exception or a specifically made subclass used in these cases. Whatever the choice is, it generally lacks clarity of communication to whoever receives it. RuntimeError is a rather generic exception according to the documentation, and you can only rely on the message, which relies on writing something appropriate for the situation: ``` exception RuntimeError Raised when an error is detected that doesn’t fall in any of the other categories. The associated value is a string indicating what precisely went wrong. ``` while it would be useful to communicate clearly to whoever is using or modifying the code "listen, it's your mistake, you misunderstood how I work or you ruined my internals, leading me to an impossible state". Also, customers seeing this kind of exception would understand without a doubt that the application is broken because of an internal error. I tried some search on the mailing list but could not find anything at a glance about this topic. Was this already discussed in the past? Thanks -- Kind regards, Stefano Borini

5 5

Starap function exists but it seems there's not such thing as "doublestarmap"
by Krokosh Nikita April 10, 2019

April 10, 2019

Hello. I have a following question: How come there's no such thing in Python like starmap but which unpacks dicts as kwargs for fuction? For example I have a format string like "{param1}, {param2}" and want to get results passing list of dicts for it's .format(). Of course I can do that with genexpr or some lambda with map. But can you clarify why starmap version of map fuction exists and doublestarmap doesn't? Best Regards.

4 5

Built-in parsing library
by Nam Nguyen April 10, 2019

April 10, 2019

Hello list, What do you think of a universal parsing library in the stdlib mainly for use by other libraries in the stdlib? Through out the years we have had many issues with protocol parsing. Some have even introduced security bugs. The main cause of these issues is the use of simple regular expressions. Having a universal parsing library in the stdlib would help cut down these issues. Such a library should be minimal yet encompassing, and whole parse trees should be entirely expressible in code. I am thinking of combinatoric parsing as the main candidate that fits this bill. What do you say? Thanks! Nam

12 20

cli tool to print value, similar to pydoc
by Thomas Gläßle April 8, 2019

April 8, 2019

Hi, what do you think about adding a tool to print a specified object, similar to pydoc shows its help? I regularly find myself checking the contents of variables to get a quick grasp on what the value looks like during development, and then type something like this: python -c 'import os; print(os.pathsep)' # or even python >>> import os >>> os.pathsep other examples may be MODULE.__version__, sys.path, or sys.platform, but really this happens a lot for a lot of different things. It would be nice to just type, e.g. any of: pyval os.pathsep python -m pprint os.pathsep python -p os.pathsep pydoc --value os.pathsep There is already a tool like this on PyPI [1] (sadly py2 only atm), but if you agree that this is a common pattern, I believe it would be a lot more useful to have it in the stdlib. [1] https://pypi.org/project/pyeval/ Additional considerations: It might be useful to add an option to pass a format specification, which would then do the equivalent of: print(format(value, spec)) Note that [1] is even more powerful, because it can directly evaluate expressions, e.g.: pyeval 'math.sin(math.pi/5)' This can be really useful at times, but maybe the base version would be enough for a first step. What do you think? Best, Thomas

4 5

Re: [Python-ideas] Python list of list matrixes without NumPy
by Juancarlo Añez April 6, 2019

April 6, 2019

On Sat, Apr 6, 2019 at 4:11 AM Steve Barnes <gadgetsteve(a)live.co.uk> wrote: > > ipython > > Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit > (AMD64)] > > Type 'copyright', 'credits' or 'license' for more information > > IPython 7.2.0 -- An enhanced Interactive Python. Type '?' for help. > > > > In [1]: [[False]*2]*5 > > Out[1]: > > [[False, False], > > [False, False], > > [False, False], > > [False, False], > > [False, False]] > > # Looks like exactly what was wanted > [ins] In [1]: x = [[False]*2]*5 [ins] In [2]: x Out[2]: [[False, False], [False, False], [False, False], [False, False], [False, False]] [ins] In [3]: x[1][1] = True [ins] In [4]: x Out[4]: [[False, True], [False, True], [False, True], [False, True], [False, True]] -- Juancarlo *Añez*

1 0

Re: [Python-ideas] Python list of list matrixes without NumPy
by Christopher Barker April 6, 2019

April 6, 2019

By the way, your reply isn't go to the list --bringing it back on. On Sat, Apr 6, 2019 at 5:03 AM Juancarlo Añez <apalala(a)gmail.com> wrote: > Or maybe there is one in PyPi — have you looked? >> > > There's a bunch of "matrix" packages in PyPi, none of them dealing with > the simple use cases I'm talking about. > Be careful about terminology -- "matrix" is a term from mathematics (linear algebra) that has a specific meaning -- I don't think that's what you mean. If it is what you mean, then what you want is numpy (though it fact, numpy is not a matrix system -- it is general purpose n-dimensional arrays designed for numerical computation that includes some features to support linear algebra ). > It could be static methods in `list`? > > x = list.matrix(n, m, default=0.0) > y = list.matrix(n, m, o, p, default=False) > i = list.eye(n, False, True) > If all you want are constructors for common 2-d or n-d arrays that would create lists of lists, then make a handful of utility functions, put them on PyPi, and see if anyone finds them useful -- if so, then offer it up as an addition the standard list object. But for my part, I think simple constructors are of limited utility -- sure, it's an easy source of errors to do it "wrong", e.g.: [[None] * 5] * 6 appears to create a 2 dimensional array, but really has all the rows as references to the same object. But users will discover this pretty quickly, and there is something to be said for folks needing to understand how python works with regard to multiple references to the same mutable object. But what might be more useful is a 2D or ND array class that would provide more than just constructors, but would provide nifty things like 2-dimension indexing: arr = NdArray((3,4), fill=None) arr: [[None, None, None, None], [None, None, None, None], [None, None, None, None]] and nifty things like 2-d indexing: arr[:, 2] to get columns, for instance. Also some controls on resizing -- you really don't want folks able to append arbitrarily to any of this. THAT might gain some traction. (or not, as you get this with numpy, and so much more, anyway. But either way, write it and show its utility first. -CHB -- Christopher Barker, PhD Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython

1 0

Python list of list matrixes without NumPy
by Juancarlo Añez April 5, 2019

April 5, 2019

The semantics of list operators are congruent, but unintuitive sometimes. For example, I've fallen over this many times: >>> truth_matrix = [[False] * n ] * m It doesn't create what you'd want. The only safe way I know to creat an NxM matrix is: >>> truth_matrix = [ [False for _ in range(n) for _ in range(m)] There could be a library to deal with that common case, and those of checking if a list of lists (of lists?) is properly rectangular, square, etc., and several other common cases. It takes experience to be aware that `truth_matrix[i][j]` will change more than one "cell" if the initialization is the former. `numpy` is not part of stdlib. A standard library for list of list would be usefu, specially to newcomersl. I don't remember such being discussed lately. >>> truth_matrix = lol(n, m, init=bool) >>> other_matrix = lol(n, m, o, p, init=float) Maybe it could also be done with syntax, but I don't have any ideas in that regard (I don't think "lol()" is overloaded). Regards, -- Juancarlo *Añez*

1 0

Add output() helper function to subprocess module
by Greg Ewing April 5, 2019

April 5, 2019

The check_output() function of the subprocess module raises an exception if the process returns a non-zero exit status. This is inconvenient for commands such as grep that use the return status to indicate something other than success or failure. The check_call() function has a companion call(), but here is currently no non-checking companion for check_call(). How about adding one with a signature such as output(args) --> (status, output) -- Greg

7 10