Mailman 3 November 2016 - Python-ideas

Method signature syntactic sugar (especially for dunder methods)
by Nathan Dunn 07 Nov '16

07 Nov '16

Python has very intuitive and clear syntax, except when it comes to method definitions, particularly dunder methods. class Vec(object): def __init__(self, x, y): self.x, self.y = x, y def __add__(self, other): return Vec(self.x + other.x, self.y + other.y) def __getitem__(self, key): return self.x if key == 'x' else self.y if key == 'y' else None def __contains__(self, item): return self.x == item or self.y == item def __bool__(self): return self.x or self.y def display(self): print('x:', self.x, 'y:', self.y) Having to declare a self parameter is confusing since you don't pass anything in when you call the method on an instance (I am aware of bound vs. unbound methods, etc. but a beginner would not be). The double underscores are also confusing. I propose syntactic sugar to make these method signatures more intuitive and clean. class Vec(object): def class(x, y): self.x, self.y = x, y def self + other: return Vec(self.x + other.x, self.y + other.y) def self[key]: return self.x if key == 'x' else self.y if key == 'y' else None def item in self: return self.x == item or self.y == item def bool(self): return self.x or self.y def self.display(): print('x:', self.x, 'y:', self.y) There are some immediate problems with this, such as `bool(self)` being indistinguishable from a regular method signature and `class(x, y)` not declaring the `self` identifier. These and other problems can be solved to some extent, but I thought I would see if there is any interest around this before going too in depth.

3 2

Small improvements to the profile/cProfile API
by Ben Hoyt 06 Nov '16

06 Nov '16

Hi folks, Every time I do some Python profiling (with cProfile) the API feels kinda baroque, and I have to write ~10 line helper functions to do what I want. For example, if I want the return value of running some function and the profiling output as a string (e.g., to send as part of a web response), I have to do something like this: import cProfile import io import pstats def profile(func, *args, **kwargs): profiler = cProfile.Profile() result = profiler.runcall(func, *args, **kwargs) stream = io.StringIO() stats = pstats.Stats(profiler, stream=stream) stats.sort_stats('cumulative') stats.print_stats(strip_dirs=True) return (result, stream.getvalue()) Something like that might be a useful function in its own right, but I took a look at the current API, and also an open issue that addresses some of this (https://bugs.python.org/issue9285 last touched almost 4 years ago), and came up with the following: 1) Add a format_stats() function to Profile to give the profiling results as a string (kind of parallel to format_* vs print_* functions in the "traceback" module). Signature would format_stats(self, *restrictions, sort='stdname', strip_dirs=False). 2) Add __enter__ and __exit__ to Profile so you can use it in a "with" statement. 3) Add a top-level runcall() function to the cProfile (and profile) modules. This isn't particularly useful for me, but it'd make the module easier to use from the command line, and it's one of the API improvements over at issue 9285. Other things in issue 9285 that I don't think are a good idea: the separate runblock() context manager (there should be only one way to do it, and I think "with Profile()" is great), and the @profile decorator (I really don't see the use of this -- you don't always want to profile a function when calling it, only it certain contexts). My code implementing the above three things as a separate module (subclassing Profile) is copied below. I think the above additions are non-controversial improvements -- do you think I should make a patch to get this into the standard library? New issue or add it to 9285? Another feature that I wanted and would be useful for a lot of folks, I think, would be some way to fetch the results as proper Python objects, rather than as a file/string. Maybe a Profile.get_results() function that returns a ProfileResult namedtuple which has total time and number of calls etc, as well as a list of ProfileEntry namedtuples that have the data for each function. Thoughts on that (before any bike-shedding on the exact API)? Currently folks who need this data have to use undocumented parts of Profile like .stats and .fcn_list, for example the Pyramid debug toolbar extracts this data so it can render it in an HTML web page: https://github.com/Pylons/pyramid_debugtoolbar/blob/ed406d7f3c8581458c2e7bd… Thanks! Ben # BELOW is my profileutils.py module """Code profiling utilities. >>> def foo(n): ... return sum(sum(range(i)) for i in range(n)) >>> with Profile() as profiler: ... result = foo(5) >>> print(profiler.format_stats(3, sort='nfl')) # doctest: +ELLIPSIS 15 function calls (10 primitive calls) in 0.000 seconds <BLANKLINE> Ordered by: name/file/line List reduced from 5 to 3 due to restriction <3> <BLANKLINE> ncalls tottime percall cumtime percall filename:lineno(function) 6/1 0.000 0.000 0.000 0.000 {built-in method builtins.sum} 6 0.000 0.000 0.000 0.000 <...>:2(<genexpr>) 1 0.000 0.000 0.000 0.000 {method 'disable' ...} <BLANKLINE> <BLANKLINE> <BLANKLINE> >>> result 10 >>> result = runcall(foo, 10) # doctest: +ELLIPSIS 24 function calls (14 primitive calls) in 0.000 seconds <BLANKLINE> Ordered by: standard name <BLANKLINE> ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 0.000 0.000 <...>:1(foo) 11 0.000 0.000 0.000 0.000 <...>:2(<genexpr>) 11/1 0.000 0.000 0.000 0.000 {built-in method builtins.sum} 1 0.000 0.000 0.000 0.000 {method 'disable' ...} <BLANKLINE> <BLANKLINE> >>> result 120 """ import cProfile import io import pstats VALID_SORTS = [ 'calls', 'cumtime', 'cumulative', 'file', 'filename', 'line', 'module', 'name', 'ncalls', 'nfl', 'pcalls', 'stdname', 'time', 'tottime', ] class Profile(cProfile.Profile): def format_stats(self, *restrictions, sort='stdname', strip_dirs=False): """Format stats report but return as string instead of printing to stdout. "restrictions" are as per Stats.print_stats(). Entries are sorted as per Stats.sort_stats(sort), and directories are stripped from filenames if strip_dirs is True. """ sort_keys = (sort,) if isinstance(sort, (int, str)) else sort stream = io.StringIO() stats = pstats.Stats(self, stream=stream) stats.sort_stats(*sort_keys) if strip_dirs: stats.strip_dirs() stats.print_stats(*restrictions) return stream.getvalue() # Define __enter__ and __exit__ to enable "with statement" support def __enter__(self): self.enable() return self def __exit__(self, exc_type, exc_value, exc_traceback): self.disable() def runcall(func, *args, **kwargs): """Profile running func(*args, **kwargs) and print stats to stdout.""" profiler = cProfile.Profile() result = profiler.runcall(func, *args, **kwargs) profiler.print_stats() return result if __name__ == '__main__': import doctest doctest.testmod()

3 4

Null coalescing operator
by Arek Bulski 05 Nov '16

05 Nov '16

Sometimes I find myself in need of this nice operator that I used back in the days when I was programming in .NET, essentially an expression >>> expr ?? instead should return expr when it `is not None` and `instead` otherwise. A piece of code that I just wrote, you can see a use case: def _sizeof(self, context): if self.totalsizeof is not None: return self.totalsizeof else: raise SizeofError("cannot calculate size") With the oprator it would just be def _sizeof(self, context): return self.totalsizeof ?? raise SizeofError("cannot calculate size") pozdrawiam, Arkadiusz Bulski

36 123

Re: [Python-ideas] Query Language extension to Python
by Pavel Velikhov 03 Nov '16

03 Nov '16

> 3 нояб. 2016 г., в 0:28, Denis Akhiyarov <denis.akhiyarov(a)gmail.com> написал(а): > > How does this compare to LINQ in .NET? Denis, PythonQL is very similar to LINQ actually. The syntax is a little different - more Pythonic vs SQL, the clauses can be composed arbitrarily, we have named tuples, that make nested queries easier to write and a few minor bells and whistles. > >> On Tuesday, November 1, 2016 at 2:03:44 PM UTC-5, Pavel Velikhov wrote: >> Hi David! >> >> I haven’t used blaze, but its looks quite similar to pandas, at least conceptually. Thanks for >> the reference! >> >> The big difference with PythonQL is that we actually extend the syntax of Python with a few >> constructs that are typically used in query languages (group by, order by, window, let clause). >> The language extension is quite small and easy to grasp, but its very powerful: you can use >> this language to easily formulate pretty complex queries is a rather simple way. >> >> So traditionally - a query language (PythonQL) is good at expressing complex things easily, but >> then you need a lot of work from the optimizer and the database to turn it into an efficient plan. A >> library like blaze or pandas is more of an “algebra” - its really a plan specification. It will usually >> take much longer to memorize all the operators and ways of doing things in such a library and typically >> you have to go back to the documentation to do things that differ slightly from what you typically do. >> >> Oh yeah, so far our execution engine is pretty simple and not too efficient, but we plan to fix this >> in the future and be at least comparable to pandas performance (need to look at what’ s under the >> hood in blaze). >> >> Of course this is my take (although I heard a few similar things from our early users). It would be >> interesting to see how other folks compare the two approaches. >> >> Btw. we have built a library for working with pandas Dataframes, we could do it for blaze too, I suppose. >> >> >> >>> On 1 Nov 2016, at 21:17, David Mertz <me...(a)gnosis.cx> wrote: >>> >>> How do you see this as different from Blaze (http://blaze.readthedocs.io/en/latest/index.html)? >>> A >>> >>> >>>> On Nov 1, 2016 1:34 AM, "Pavel Velikhov" <pavel.v...(a)gmail.com> wrote: >>>> Hi Folks, >>>> >>>> We have released PythonQL, a query language extension to Python (we have extended Python’s comprehensions with a full-fledged query language, >>>> drawing from the useful features of SQL, XQuery and JSONiq). Take a look at the project here: http://www.pythonql.org and lets us know what you think! >>>> >>>> The way PythonQL currently works is you mark PythonQL files with a special encoding and the system runs a preprocessor for all such files. We have >>>> an interactive interpreter and Jupyter support planned. >>>> >>>> Best regards! >>>> PythonQL team >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python...(a)python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>

1 0

More user-friendly version for string.translate()
by Mikhail V 02 Nov '16

02 Nov '16

Hello all, I would be happy to see a somewhat more general and user friendly version of string.translate function. It could work this way: string.newtranslate(file_with_table, Drop=True, Dec=True) So the parameters: 1. "file_with_table" : a text file with table in following format: #[In] [Out] 97 {65} 98 {66} 99 {67} 100 {} ... 110 {110} Notes: All values are decimal or hex (to switch between parsing format use Dec parameter) As it turned out from my last discussion, majority prefers hex notation, so I am not in mainstream with my decimal notation here, but both should be supported. Empty [Out] value {} means that the character will be deleted. 2. "Drop = True" this will set the default behavior for those values which are NOT in the table. For Drop = True: all values not defined in table set to [out] = {}, and be deleted. For Drop=False: all values not defined in table set [out] = [in], so those remain as is. 3. Dec= True : parsing format Decimal/hex. I use decimal everywhere. Further thoughts: for 8-bit strings this should be simple to implement I think. For 16-bit of course there is issue of memory usage for lookup tables, but the gurus could probably optimise it. E.g. at the parsing stage it is not necessary to build the lookup table for whole 16-bit range of course, but take only values till the largest ordinal present in the table file. About the format of table file: I suppose many users would want also to define characters directly, I am not sure if it is really needed, but if so, additional brackets or escape char could be used, like this for example: a {A} \98 {\66} \99 {\67} but as said I don't like very much the idea and would be OK for me to use numeric values only. So approximately I see it. Feel free to share thoughts or criticise. Mikhail

10 35

Query Language extension to Python
by Pavel Velikhov 01 Nov '16

01 Nov '16

Hi Folks, We have released PythonQL, a query language extension to Python (we have extended Python’s comprehensions with a full-fledged query language, drawing from the useful features of SQL, XQuery and JSONiq). Take a look at the project here: http://www.pythonql.org and lets us know what you think! The way PythonQL currently works is you mark PythonQL files with a special encoding and the system runs a preprocessor for all such files. We have an interactive interpreter and Jupyter support planned. Best regards! PythonQL team

4 6