Mailman 3 December 2018 - Python-ideas

[asyncio] Suggestion for a major PEP
by Christophe Bailly Dec. 16, 2018

Dec. 16, 2018

Hello, I copy paste the main idea from an article I have written: contextual async <https://pythonc.home.blog/2018/12/14/premier-article-de-blog/> " Imagine you have some code written for monothread. And you want to include your code in a multithread environment. Do you need to adapt all your code which is what you do when you want to migrate to async code ? The answer is no. Functionnally these constraints are not justified neither technically Do we have the tools to do this ? Yes because thanks to boost::context we can switch context between tasks. When a task suspends, it just calls a function (the event loop or reactor) to potentially switch to another task. Just like threads switch contexts… Async/Await logic has introduced a symetric relation wich introduces unnecessary contraints. We should just the same logic as thread logic. " Read the examples in the article I have developped a prototype in C++ and everything works perfectly. My opinion is that sooner or later, it will have to switch to this logic because chaining async/aswait is a huge contraints and does not make sense in my opinion. Maybe I am missing something, Feel free to give me your feedback. Regards, Chris

4 4

It would be great if the json module would allow and clear existing comments.
by João Matos Dec. 13, 2018

Dec. 13, 2018

Hello, Comments in JSON files are a great way to document a configuration file for example. Even JSON's Douglas Crockford agrees that it is a helpful thing and it suggests using JSMin before handing it to the JSON parser in here https://plus.google.com/+DouglasCrockfordEsq/posts/RK8qyGVaGSr So, I would like to suggest adding that feature do the json module. A simple boolean argument, clear_comments, with the default False to keep previous compatibility. With the clear_comments=True and following JSMin "rules", comments in the // form should be replaced with linefeeds and comments in the /* */ form with spaces. Best regards, JM

1 0

__len__() for map()
by Kale Kundert Dec. 13, 2018

Dec. 13, 2018

I just ran into the following behavior, and found it surprising: >>> len(map(float, [1,2,3])) TypeError: object of type 'map' has no len() I understand that map() could be given an infinite sequence and therefore might not always have a length. But in this case, it seems like map() should've known that its length was 3. I also understand that I can just call list() on the whole thing and get a list, but the nice thing about map() is that it doesn't copy data, so it's unfortunate to lose that advantage for no particular reason. My proposal is to delegate map.__len__() to the underlying iterable. Similarly, map.__getitem__() could be implemented if the underlying iterable supports item access: class map: def __init__(self, func, iterable): self.func = func self.iterable = iterable def __iter__(self): yield from (self.func(x) for x in self.iterable) def __len__(self): return len(self.iterable) def __getitem__(self, key): return self.func(self.iterable[key]) Let me know if there any downsides to this that I'm not seeing. From my perspective, it seems like there would be only a number of (small) advantages: - Less surprising - Avoid some unnecessary copies - Backwards compatible -Kale

19 113

Using sha512 instead of md5 on python.org/downloads
by Miro Hrončok Dec. 10, 2018

Dec. 10, 2018

Hi, I see md5 checksums at a release download page such as [1]. My idea is to switch to sha512 for a more reliable outcome. I'm no security expert, but AFAK md5 is generally believed to be unsafe, as it was repeatedly proven it can be vulnerable [2]. [1] https://www.python.org/downloads/release/python-371/ [2] https://en.wikipedia.org/wiki/MD5#Security -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok

16 26

TAPS Implementation
by Franke, Maximilian Julian Shawn Dec. 10, 2018

Dec. 10, 2018

Hi, I am a student worker with the Internet Networks Architecture department at TU Berlin and I am working with APIs for network protocols. We are currently looking into implementing TAPS, a novel way to offer transport layer services to the application layer. The idea is to offer an API on top of multiple different transport protocols, such as TCP and QUIC. Instead of explicitly choosing a transport protocol, the application only provides abstract requirements, e.g., reliability. The TAPS system maps these properties to transport protocols, potentially trying out multiple protocols in parallel. Furthermore, TAPS can select between multiple local interfaces and remote IP addresses. A short talk (~25 minutes) from the All systems go! conference about it is available here: https://media.ccc.de/v/ASG2018-188-the_future_of_networking_apis. TAPS is currently being standardized in the IETF (https://datatracker.ietf.org/wg/taps/about/). Here you can find the proposed architecture: https://datatracker.ietf.org/doc/draft-ietf-taps-arch/, interface: https://datatracker.ietf.org/doc/draft-ietf-taps-interface/ and an informal draft on implementation considerations: https://datatracker.ietf.org/doc/draft-ietf-taps-impl/. One if the implementations currently in the works is done by Apple in form of their Network.framework API (https://developer.apple.com/documentation/network). While this implementation is relatively advanced, it is so far only available for MacOS, iOS and it derivatives. As such, it would be favorable to have a platform agnostic and open-source implementation as well. >From what we can tell, asyncio seems to offer a lot of the ground work necessary to implement it efficiently, so here are some questions we have before beginning with the implementation: - Is something like this in the scope to become part of the standard python library or something that would be done in an external library? If it is in scope, what would the requirements for it to become part of the standard library be? - Are there currently any other active efforts to implement new network functionality into the standard library? - Are there currently any considerations to expand the standard transports offered by asyncio (TCP, UDP and SSL) by additional ones like SCTP, or more importantly QUIC? Any comments or further pointers to sources that could be helpful with this would be greatly appreciated. Best regards Max

3 2

[Brainstorm] Testing with Documented ABCs
by Abe Dillon Dec. 8, 2018

Dec. 8, 2018

I've been pulling a lot of ideas from the recent discussion on design by contract (DBC), the elegance and drawbacks <https://bemusement.org/doctests-arent-code> of doctests <https://docs.python.org/3/library/doctest.html>, and the amazing talk <https://www.youtube.com/watch?v=MYucYon2-lk> given by Hillel Wayne at this year's PyCon entitled "Beyond Unit Tests: Taking your Tests to the Next Level". To recap a lot of previous discussions: - Documentation should tell you: A) What a variable represents B) What kind of thing a variable is C) The acceptable values a variable can take - Typing and Tests can partially take the place of documentation by filling in B and C (respectively) and sometimes A can be inferred from decent naming and context. - Contracts can take the place of many tests (especially when combined with a library like hypothesis) - Contracts/assertions can provide "stable" documentation in the sense that it can't get out of sync with the code. - Attempts to implement contracts using standard Python syntax are verbose and noisy because they rely heavily on decorators that add a lot of repetitive preamble to the methods being decorated. They may also require a metaclass which restricts their use to code that doesn't already use a metaclass. - There was some discussion about the importance of "what a variable represents" which pointed to this article <http://pgbovine.net/python-unreadable.htm> by Philip J. Guo (author of the magnificent pythontutor.com). I believe Guo's usage of "in-the-small" and "in-the-large" are confusing because a well decoupled program shouldn't yield functions that know or care how they're being used in the grand machinations of your project. The examples he gives are of functions that could use a doc string and some type annotations, but don't actually say how they relate to the rest of the project. One thing that caught me about Hillel Wayne's talk was that some of his examples were close to needing practically no code. He starts with: def tail(lst: List[Any]) -> List[Any]: assert len(lst) > 0, "precondition" result = lst[1:] assert [lst[0]] + result == lst, "postcondition" return result He then re-writes the function using a contracts library: @require("lst must not be empty", lambda args: len(args.lst) > 0) @ensure("result is tail of lst", lambda args, result: [args.lst[0]] + result == args.lst) def tail(lst: List[Any]) -> List[Any]: return lst[1:] He then writes a unit test for the function: @given(lists(integers(), 1)) def test_tail(lst): tail(lst) What strikes me as interesting is that the test pretty-much doesn't need to be written. The 'given' statement should be redundant based on the type annotation and the precondition. Anyone who knows hypothesis, just imagine the @require is a hypothesis 'assume' call. Furthermore, hypothesis should be able to build strategies for more complex objects based on class invariants and attribute types: @invariant("no overdrafts", lambda self: self.balance >= 0) class Account: def __init__(self, number: int, balance: float = 0): super().__init__() self.number: int = number self.balance: float = balance A library like hypothesis should be able to generate valid account objects. Hypothesis also has stateful testing <https://hypothesis.readthedocs.io/en/1.4.1/stateful.html> but I think the implementation could use some work. As it is, you have inherit from a class that uses a metaclass AND you have to pollute your class's name-space with helper objects and methods. If we could figure out a cleaner syntax for defining invariants, preconditions, and postconditions we'd be half-way to automated testing UTOPIA! (ok, maybe I'm being a little over-zealous) I think there are two missing pieces to this testing problem: side-effect verification and failure verification. Failure verification should test that the expected exceptions get thrown when known bad data is passed in or when an object is put in a known illegal state. This should be doable by allowing Hypothesis to probe the bounds of unacceptable input data or states, though it might seem a bit silly because if you've already added a precondition, "x >= 0" to a function, then it obviously should raise a PreconditionViolated when passed any x < 0. It may be important, however; if for performance reasons, you need to disable invariant checking but you still want certain bad input to raise exceptions, or your system has two components that interact with slightly mis-matched invariants and you want to make sure the components handle the edge-condition correctly. You can think of Types from a set-theory perspective where the Integer type is conceptually the set of all integers, and invariants would specify a smaller subset than Typing alone, however if the set of all valid outputs of one component is not completely contained within the set of all valid inputs to another component, then there will be edge-cases resulting from the mismatch. In that sense, some of the invariant verification could be static-ish (as much as Python allows). Side-effect verification is usually done by mocking dependencies. You pass in a mock database connection and make sure my object sends and receives data as expected. As crazy as it sounds, this too can be almost completely automated away if all of the above tools are in place AND if Python gained support for Exception annotations. I wrote a Java (yuck) library at work that does this. I wan't to port it to Python and share it, but it basically enumerates a bunch of stuff: the "sources" and "destinations" of the system, how those relate to dependencies, how they relate to each other (if dependency X is unresponsive, I can't get sources A, B, or G and if I can't get source B, I can't write destination Y), the dependency failure modes (Exceptions raised, timeouts, unrecognized key, missing data, etc.), all the public methods of the class under test and what sources and destinations they use. Then I enumerate 'k' from 0 to some limit for the max number of simultaneous faults to test for: Then for each method that can have n >= k simultaneous faults I test all (n choose k) combinations of faults for that method against the desired behavior. I'm sure that explanation is as clear as mud. I will try to get a working Python example at some point to demonstrate. Finally, in the PyCon video; Hillel Wayne shows an example of testing that an "add" function is commutative. It seems that once you write that invariant, it might apply to many different functions. A similar invariant may be "reversibility" like: @given(text()) def test_reversable_codex(s): assert s == decode(encode(s)), "not reversible" That might be a common property that other functions share: @invariant(reversible(decode)) def encode(s: str) -> bytes: ... Having said all that, I wanted to brainstorm some possible solutions for implementing some or all of the above in Python without drowning you code in decorators. NOTE: Please don't get hung up on specific syntax suggestions! Try to see the forest through the trees! An example syntax could be: #Instead of this @require("lst must not be empty", lambda args: len(args.lst) > 0) @ensure("result is tail of lst", lambda args, result: [args.lst[0]] + result == args.lst) def tail(lst: List[Any]) -> List[Any]: return lst[1:] #Maybe this? non_empty = invariant("Must not be empty", lambda x: len(x) > 0) # can be re-used def tail(lst: List[Any] d"Description of what this param represents. {non_empty}") -> List[Any] d"Description of return value {lst == [lst[0]] + __result__}": """ Description of function """ return lst[1:] Python could build the full doc string like so: """ Description of function Args: lst: Description of what this param represents. Must not be empty. Returns: Description of return value. """ d-strings have some description followed by some terminator after which either invariant objects or [optionally strings] followed by an expression on the arguments and __return__? I'm sorry this is so half-baked. I don't really like the d-string concept and I'm pretty sure there are a million problems with it. I'll try to flesh out the side-effect verification concept more later along with all the other poorly explained stuff. I just wanted to get these thoughts out for discussion, but now it's super late and I have to go!

10 24