Mailman 3 March 2015 - Python-ideas

Re: [Python-ideas] PEP 484: Generating test inputs from type hints
by David MacIver 31 Mar '15

31 Mar '15

(Sorry for the manual thread continuation, I wasn't previously on the mailing list) I presume 'support' could and would include a version of given() that > accessed .__annotations__. This is on my list of planned features independently of PEP 484 FWIW. It's on the long list of nice to have but minor things that didn't make it into 1.0. One of the problems is that I am for the moment somewhat committed to supporting Python 2.7 as well as Python 3 (I really wish I wasn't), or I would have probably built Hypothesis on using annotations from the beginning. To a degree though it's fortunate that I didn't given PEP 484. It doesn't look like I would be able to build the full range of features Hypothesis needs for @given using only type annotations, and 484 suggests the goal is to move to those being the only valid ones. The reason for this is that Hypothesis deliberately lets you mix strategies in with the specifiers used for annotation. So for example you can do something like: Palindromes = strategy(str).map(lambda x: x + str(reversed(x))) This is now a SearchStrategy object that still returns strings, but only ones that are palindromes. You can now do something like @given([Palindromes]) def test_this_is_a_list_of_palindromes(xs): # I'm not actually sure what interesting things you can say about a list of palindromes... So the point is that you can use a SearchStrategy deep inside the "type" signatures that Hypothesis uses, despite SearchStrategy not actually being at all type-like. (It actually used to be the case that an earlier version of SearchStrategy allowed you to test whether a value could have come from that strategy, which would have made them a bit more "type-like", but this proved to be a bad idea and the work that let me remove it was one of the best decisions I made in the design of Hypothesis) So basically I would fully intend for Hypothesis to become a consumer of types and annotations, but I don't think they can be used as a primary building block of it. (Note: I don't consider this a problem with PEP 484, which looks like a good proposal. Hypothesis's requirements are just weird and specialized and don't quite match type checking) > Of course, if every 'test_xyz' were to be > decorated with the same decorator, then the decorator could be omitted > and the fuzzing moved into the test runner. I'd be moderately against this being the primary mode of using Hypothesis, even without the above caveats. One of the things I like about Hypothesis right now is that because it just gives you Python functions it can be very unopinionated about what you're using to run your tests (even if in practice basically everyone is using unittest or pytest). However it should be relatively easy to support alongside once I've done some much needed refactoring of the core example exploration code. David

1 0

PEP 484: Generating test inputs from type hints
by Terry Reedy 31 Mar '15

31 Mar '15

PEP 484 proposes "a standard syntax for type annotations", principally for static analysis (and other static uses), with runtime checking or optimization mentioned as a possibility. Another (real) runtime use is generating unit test inputs. Today on python-list, David MacIver announced Hypothesis, a testing library that does just that. (The thread title is "Hypothesis 1.0: A production quality property-based testing library for Python".) Here is an example from http://hypothesis.readthedocs.org/en/latest/quickstart.html from hypothesis import given @given([int]) def test_reversing_twice_gives_same_list(xs): assert xs == list(reversed(reversed(xs))) When a test runner such as pytest calls the wrapper, it calls the real function with a variety of int lists (including [], but details not important here). @given() also works with unittest TestCase methods. When I mentioned PEP 484 and the alternative syntax List(int), he replied "Having something like that as standard would be great for Hypothesis and I intend to support it once it becomes available." I presume 'support' could and would include a version of given() that accessed .__annotations__. Of course, if every 'test_xyz' were to be decorated with the same decorator, then the decorator could be omitted and the fuzzing moved into the test runner. -- Terry Jan Reedy

2 1

Add a context manager to keep stream position unchanged
by Dmitry Kazakov 30 Mar '15

30 Mar '15

I propose adding a context manager to contextlib module, which would restore the stream position of its argument at exit: with contextlib.closing(file_like_object) as file: file.read(1) old = file.tell() with contextlib.keep_stream_position(file) as f: assert file is f assert file_like_object is f # Do something with the file(-like) object e.g. feed it # to `PIL.Image.open` (will consume the whole stream) assert file.tell() == old This CM would allow us to reuse streams efficiently, without requiring to call seek() explicitly. I will open a new issue and submit a patch in case of favorable responses.

7 11

"Loose" descriptors
by James Edwards 30 Mar '15

30 Mar '15

(I apologize in advance if this was covered previously or elsewhere; a fairly thorough search didn't yield anything.) Currently, we can implement a descriptor as follows: class VerboseDescriptor(object): def __init__(self, init): self.val = init def __get__(self, obj, typ=None): print("Get is %s" % (self.val)) return self.val def __set__(self, obj, val): print("Set to %s" % (val)) self.val = val class Holder(object): val = VerboseDescriptor(2) foo = Holder() _ = foo.val foo.val = 4 Which of course works as expected, displaying: Get is 2 Set to 4 But we can't achieve the same indirection without a "Holder"-type class. That is, we can't do the following: class VerboseDescriptor(object): def __init__(self, init): self.val = init def __get__(self, obj, typ=None): print("Get is %s" % (self.val)) return self.val def __set__(self, obj, val): print("Set to %s" % (val)) self.val = val val = VerboseDescriptor(2) _ = val val = 4 Or rather, we can, but what might be expected (the same functionality / output), is not what occurs. My understanding of the resolution of the first example is that python looks for `val` in `foo.__dict__`, then in `Holder.__dict__`, where it finds it, identifies that `Holder.val` implements `__get__`, and handles the indirection accordingly. So `foo.val = 4` ends up being something like `Holder.val.__set__(foo, 4)`. But my question is would it make sense to be able to do this sort of indirection for "loose" objects also, not just attributes of classes or instances? In other words, could descriptors be extended so that they perform the same sort of indirection in the second example as the first? I envision (but with no real knowledge of the implications of this) that just as python identifies that `val` in `Holder.__dict__` implements `__get__`, python could similarly identify that `val` in `globals()` (or `sys.modules[__name__].__dict__`, or wherever is more appropriate) implements `__get__` and handle the indirection. Of course, a syntax issue would arise if this were possible -- notably the class definition of `Holder` would break. So some syntax adjustment would have to be made. But is this reasonable / of interest / attainable? Thanks for your time and consideration.

6 6

"Loose" descriptors?
by jedwards 29 Mar '15

29 Mar '15

(I apologize in advance if this was covered previously or elsewhere; a fairly thorough search didn't yield anything.) Currently, we can implement a descriptor as follows: class VerboseDescriptor(object): def __init__(self, init): self.val = init def __get__(self, obj, typ=None): print("Get is %s" % (self.val)) return self.val def __set__(self, obj, val): print("Set to %s" % (val)) self.val = val class Holder(object): val = VerboseDescriptor(2) foo = Holder() _ = foo.val foo.val = 4 Which of course works as expected, displaying: > Get is 2 > Set to 4 But we can't achieve the same indirection without a "Holder"-type class. That is, we can't do the following: class VerboseDescriptor(object): def __init__(self, init): self.val = init def __get__(self, obj, typ=None): print("Get is %s" % (self.val)) return self.val def __set__(self, obj, val): print("Set to %s" % (val)) self.val = val val = VerboseDescriptor(2) _ = val val = 4 Or rather, we can, but what might be expected (the same functionality / output), is not what occurs. My understanding of the resolution of the first example is that python looks for `val` in `foo.__dict__`, then in `Holder.__dict__`, where it finds it, identifies that `Holder.val` implements `__get__`, and handles the indirection accordingly. So `foo.val = 4` ends up being something like `Holder.val.__set__(foo, 4)`. But my question is would it make sense to be able to do this sort of indirection for "loose" objects also, not just attributes of classes or instances? In other words, could descriptors be extended so that they perform the same sort of indirection in the second example as the first? I envision (but with no real knowledge of the implications of this) that just as python identifies that `val` in `Holder.__dict__` implements `__get__`, python could similarly identify that `val` in `globals()` (or `sys.modules[__name__].__dict__`, or wherever is more appropriate) implements `__get__` and handle the indirection. Of course, a syntax issue would arise if this were possible -- notably the class definition of `Holder` would break. So some syntax adjustment would have to be made. But is this reasonable / of interest / attainable? Thanks for your time and consideration.

2 1

Geo coordinates conversion in stdlib
by anatoly techtonik 29 Mar '15

29 Mar '15

I think it should be in stdlib. 11° 18′ 0″ S, 2° 42′ 0″ E to -11.3, 2.7 and back. This is from Wikipedia. -- anatoly t.

11 19

Format character to center strings
by Rob Cliffe 28 Mar '15

28 Mar '15

As a string can be left-justified (within a given width) using e.g. '%-20s' % s and right-justified using '%+20s' % s # or '%20s' % s why not allow a string to be centered using '%=20s' %s Rob Cliffe

6 6

Proposal for new augmented assignment operator for method calls
by Jamie Willis 28 Mar '15

28 Mar '15

(As posted on python-list) I would like to propose a new piece of syntax for the python language; .= In short, the operator is form of syntactic sugar, for instance consider the following code: hello = "hello world " > hello = hello.strip() This could be written as: hello = "hello world " > hello .= strip() In this slightly contrived example, the programmer saved (a small amount of) time when writing the code. With code with longer variable names, or lots of similar statements all in a row, this helps to keep code more concise, and could cut down on potential bugs due to typos (thanks to Skip Montanaro for pointing this out). The operator would be constricted to one method or field on the right-hand side, which must belong to the object on the left hand side. To provide another example, consider this basic LinkedList class: class LinkedList: > > def __init__(self, data, head=None): > self.data = data > self.next = None > self.head = head if head != None else self > def append(self, data): new_node = LinkedList(data, self.head) > self.next = new_node > return new_node > linked_list = LinkedList(7) > linked_list = linked_list.append(8) > linked_list = linked_list.append(9) > loop_node = linked_list.head while loop_node != None: > print(loop_node.data) > loop_node = loop_node.next Taking a look at "*linked_list = linked_list.append(8)*", we could write this as: linked_list .= append(8) under the new syntax and then "*loop_node = loop_node.next*" could simply become: loop_node .= next Would this idea get much support if submitted as a PEP? What are the potential concerns I could consider with the syntax (Skip has already pointed out that the "dot" is pretty invisible in the operator, so perhaps a different character could be used)? Thanks, Jamie

3 3

A mutable alternative to namedtuple: recordarray -> recordclass:
by Zaur Shibzukhov 27 Mar '15

27 Mar '15

--- *Zaur Shibzukhov* 2015-03-27 16:17 GMT+03:00 Zaur Shibzukhov <szport(a)gmail.com>: > > > --- > *Zaur Shibzukhov* > > > 2015-03-27 7:40 GMT+03:00 Andrew Barnert <abarnert(a)yahoo.com>: > >> On Mar 26, 2015, at 21:26, Zaur Shibzukhov <szport(a)gmail.com> wrote: >> >> 2015-03-27 7:08 GMT+03:00 Joonas Liik <liik.joonas(a)gmail.com>: >> >>> namedlist perhaps? :) >>> >>> if you want:"namedtuple, but mutable. " then namedlist seems like the >>> obvious alternative.. >>> >> >> The name 'namedlist' is already used in >> https://pypi.python.org/pypi/namedlist, 'mutabletuple' -- too in >> https://pypi.python.org/pypi/mutabletuple. >> >> >> If you're trying to provide the same concept, why use a completely >> unrelated name? That's like saying 'I want an ordered set, but there's >> already an "orderedset" on PyPI so I went with "sortedsequence"'. >> >> The name recordarray is means that it 1) array of objects and 2) record >> with access to fields by attributes. >> >> >> But how is being "an array of objects" any different from what a tuple, >> list, array.array, bytearray, bytes, str, etc. already are? What's >> specifically array-like about this type as opposed to all of those? And >> what's specifically record-like about your type compared to namedtuple, >> Struct, or SimpleNamespace? >> >> > I am inclined to the fact that it's better to rename `objectarray` to > 'mutabletuple' in order to be explicit about what is that. > 'recordarray' is a factory function that make exactly the same as > 'namedtuple' factory does (except that it create subclass of 'mutabletuple' > and make '_replace' to update the 'self', not make a copy). So may be it's > better to call it as 'record' or 'recordtype', or even 'recordfactory'? > > After consideration of all arguments recordarray project was renamed tand moved to recordclass <https://pypi.python.org/pypi/recordclass>.

1 0

Python Research Insitute
by anatoly techtonik 27 Mar '15

27 Mar '15

I am well aware of https://www.blender.org/institute/ and I like it. And recently I realized that my claims and accusations towards PSF are just wrong. PSF is a fund - it should process the money to ensure Python Community has at lest some degree of independence from sponsors. It is not its role to register domains, maintain servers, assign copyright to itself from contributors - it is not its role to do anything at all. PSF waits for initiatives to appear and then supports them, but the problem is - there are not many initiatives (except website, which is well, <opinionated opinion stripped>). And that's the problem that PSF is not solving, because it can not do this. It can not just do what it wants - grab a bunch of people together to solve a particular problem, provide them with a place to do stuff, with access to funding from people interested in certain initiative, direct contact with them, being interested people from outside into the team and do other nasty things (like filming movies!). PSF holds a lot of responsibility and therefore is not effective for research works and experiments. This things that PSF can not do, and nobody can, because economy doesn't allow this: 1. Maintain a list of top 10 stdlib modules that need a redesign The one is gathered there: http://sayspy.blogspot.com/2009/07/informal-poll-what-python-stdlib.html Try with Python community a usability study to find out "why people feel these things need a redesign". 2. Do economic study on Python contributions How many people could escape the corporate "slavery" to do the necessary work for high quality patches. Are they allowed to work on Python, do they feel stressed, etc. How do they feel the economic model helps Python development, is there a way for alternative models that help more people to jump in to make good thing better. 3. Get data on Python services, develop participation tools bugs.python.org is not only a GSoC project. There are a lot of things that can be improved, should be improved and the issues are not trivial. 4. Publish a research what will happen if Python 2.8 is released Study the things like https://gist.github.com/techtonik/574401cebf5e17c33f5b with reports on income and outcomes that could actually be referenced (unlike mails, which are good, but hard to filter). 5. Get data on Python usage, a set of areas and most popular use cases This is came from Geo coordinates conversion module in stdlib. >> Most people I know who use Python as a serious data processing language don't care much what's in the stdlib because they're using numpy and/or pandas. > Yes, and that's sad that the Python still doesn't have the basic building blocks of Pandas in stdlib. Study the idea of making a stlib that consists from modules that cover the most common scenarios of using the scripting language, provide a "toy modules" that cover these common scenarios - a mini demos of more powerful tools, like a "swiss-knives toolbelt". 6. Get a tools on measuring Python stdlib usage Build a linked semantic AST from all packages on PyPI (vector data like OpenStreetMap, but from the language level objects). This will allow to grep how many times a particular module is used, and allow people to use that database to extend into their own research (like trying to guess context and usage scenarios for stdlib calls). 7. Record a summaries of Python evolution, build media materials Visual information requires much much much less time to be consumed, but requires a significant amount of time to produce. PyCon talks are exceptionally good for video material, but for the community coordination, a lot of HTML5 and dynamic design works needs to be done. PSF won't be able to make it interesting to HTML5 folks, because it is solely about Python itself, and the object of Python Research Institute include everything that goes side by side and influences the development of the project and community around it. From "cultural aspects of Monty Python in context of computer language programming" to "feasibility of designing an operating system in stackless" and "test of new economic models for sustainable Python development" etc. I just can't stop with ideas - research how appearance of startup scene affected open source contributions, research why open source project die and how to prevent that, what happened with former Python contributors and where their interest is now, how other languages compare to Python, what are the key problems of converting Python code to other languages, how to deal with complexity etc., what this "agile" is about and could it be useful for Python development etc. Please, tell me if that is a worthy utopia and add your ideas.

1 0