Mailman 3 July 2018 - Python-ideas

Re: [Python-ideas] Change repr of collections.OrderedDict to be more dict-like
by Jeroen Demeyer 26 Jul '18

26 Jul '18

On 2018-07-27 00:50, David Mertz wrote: > I often use doctests to verify APIs and behaviors when I update code. I > know I'm in a minority and most developers slightly disparage doctests. If it makes you feel better, SageMath development uses only doctests for testing. We have over 300000 doctests and it works great! We do use a customized doctesting framework though.

1 0

Re: [Python-ideas] Python-ideas Digest, Vol 140, Issue 169
by James Lu 26 Jul '18

26 Jul '18

> I several times wished I could have had a reference to the block of code in a `with` clause. Could you abstractly describe a real-world example of this? I’d like to hear your use case. > The example for sorted is also kinda twisted. It is, it’s probably a counterexample on how to use it. I forgot to mention, could it be possible to only treat do as a keyword when it’s used in an as-do expression? So, do remains a legal variable name, and do is only interpreted as a keyword when it’s in an as-do expression. I believe this is possible using the Python grammar.

1 0

Re: [Python-ideas] Python-ideas Digest, Vol 140, Issue 169
by James Lu 26 Jul '18

26 Jul '18

> You mean lambda x: lumberjack(15, x) ? Yes, that was my mistake. > However this syntax you allow to write lumberjack(%, 15) which is only > possible with partial using keyword arguments, like lumberjack (y=15) > (given that the second argument is called "y") and in that case I know at > least one library on pip doing the job with Ellipsis : > > from funcoperators import elipartial > elipartial(lumberjack, ..., 15) > # gives lambda x: lumberjack (x, 15) > > elipartial (lumberjack, ..., 15, ..., 12, ...) > # gives lambda x, y, z: lumberjack(x, 15, y, 12, z) > > And the lib even provide a decorator to have that kind of syntax using "[" : > > partially(lumberjack)[15] > would be the same as partial(lumberjack, 15) > > So, that functionality does really need new syntax. Hm, yes. Perhaps elipartial could be aliased and/or made a global built in. Let’s discuss the merits of the statement itself then.

1 0

Re: [Python-ideas] As-do statements in Python
by James Lu 26 Jul '18

26 Jul '18

> 4 - introducing a new keyword is the hardest thing you can ever ask on > this list. As is already a keyword from with-as and except-as. Perhaps for compatibility “as” is allowed but discouraged as a variable name, and it’s only interpreted as a keyword when followed by an expression and a colon. > 3 - Most of the example (and you show it yourself with flask) are > already possible with a more verbose syntaxe based on decorators Yup, a more terse syntax puts the programmer’s focus on the right things. > 2 - This will be tempting to use for callbacks and chaining things, > leading to the nesting hell we tried very carefully to avoid fom other > languages. The example for sorted is also kinda twisted. I do agree as-do could be easily used for callbacks. Could you please clarify what you mean by nesting hell? Is it better to use as-do where you’d normally pass your own instantiation of a class you create that implements a protocol? It’s always possible to simply pass a bound method of a class you create yourself if you want to keep state.

1 0

Re: [Python-ideas] PEP 505: None-aware operators
by James Lu 25 Jul '18

25 Jul '18

What if we used ? after the statement beginning? name ?= person.name custom_query ?= entity.get_query(context) # Becomes None if entity is None. Raise an exception if entity is not None and get_query is None or undefined. custom_query ??= entity.get_query(context) # If entity, entity.get_query, entity.get_query(context) evaluate to null, the operation short-circuits and custom_query becomes None await? foo with? bar as baz: # this only runs if bar is not None pass ?= only short circuits into None when the first evaluation is None (the weak operator) ??= short circuits into None whenever any evaluation is None or raises an AttributeError. (the strong operator) I’m imagining the strong operator would be useful especially for duck typing. No more hasattr checks, no more isinstance checks. I would like to see some real world use cases for none aware operators that couldn’t be covered by these none aware assignment operators. Previous code: # original code a ?= b.c # another, second, programmer comes along and changes it to a ?= b.c.d The interpreter would raise an exception, if “d” was None or not defined on “c”. If this is intended behavior, this forces the second programmer to explicitly mark that all attribute access is coalescing with the strong operator, instead of the interpreter swallowing the exception now and emitting an exception later. The previous code assumes “b” is None, which in reality may represent the state of a network socket or a file transfer. The program may display that the operation was complete, leading to a bug in the output. I believe this is more readable as well. > On Jul 25, 2018, at 7:32 PM, python-ideas-request(a)python.org wrote: > > PEP 505: None-aware operators

1 0

Multi-core reference count garbage collection
by Jonathan Fine 25 Jul '18

25 Jul '18

Hi Based on other people's work (including in particular talks by Larry Hastings) and my own thinking, I've come up with a scheme for multi-core reference count garbage collection. I think it works, and much or all of it comes from others. It might even be in the literature. Of course, if it's wrong then the debit goes to me. I'll explain it in instalments. Please let me know what you think, as we go along. The basic ideas of reference counting garbage collection are: 1. Each allocated piece of memory is given an ID (which is often its address in memory). 2. For each ID, the system keeps a count of how many references (to that piece of memory). 3. The count starts at ONE. 4. Processes increment and decrement the count, to keep the count correct. 5. When the count reaches zero, the piece of memory is garbage collected. 6. The previous step might result in further reference count decrements. The simplest possible garbage collection scheme is to do nothing (and let the garbage build up). In other words, allocate on a stack, and never free memory. This has very good performance, until memory is exhausted. It is also conceptually useful. I'll call it the do-nothing garbage collection scheme. Suppose for definiteness we have a 6-core CPU, with 5 worker processes and one garbage collection (GC) process. The worker processes will do as little as possible, consistent with not running out of memory. To achieve this: 1. Each worker process with have two GC buffers, one for increment and the other for decrement. 2. For increment, the process will append the ID to the increment buffer (for the process). And similarly for decrement. 3. If the buffer to be appended to is full, the worker process will (a) set a buffer-full flag (b) pause until the buffer-full flag has been cleared (c) and then do what it was previously blocked from doing I call any such scheme BUFFERED multi-core reference count garbage collection. The worker processes know nothing about how garbage collection is managed. Instead, they pass over to the GC process sufficient information to allow it to manage the garbage. This is now a good place to pause. We've specified what it is that the worker processes should do, regarding garbage collection. And we've passed the problem on to the garbage collection process. If that process does nothing, we have refactored the do-nothing garbage collection scheme. Expect another instalment tomorrow. with best regards Jonathan

7 16

Re: [Python-ideas] A better (simpler) approach to PEP 505
by Grégory Lielens 25 Jul '18

25 Jul '18

Interesting, looking into, let's say, 100 of those is [not] None, randomly chosen, should give an very good idea of typical use for your code base. I am curious about why you don't like solutions involving exceptions? Performance? Catching too much, like typing errors? Less easy to distinguish between None and attr missing? Less easy to combine None-aware and classic attr lookup on the same expression? Something else?

2 3

A better (simpler) approach to PEP 505
by David Mertz 24 Jul '18

24 Jul '18

The need addressed by PEP 505 is real; it's also MUCH more niche and uncommon than something that would merit new syntax. Moreover, the actual legitimate purpose served by the PEP 505 syntax is easily served by existing Python simply by using a wrapper class. Here is a way of solving the "deep attribute access to messy data" problem that is: (1) Much more explicit (2) Requires no change in syntax (3) Will not be a bug magnet (4) Inasmuch as there are semantic traps, they are announced by the use of a class whose documentation would be pointed to for readers The API that could be useful might be something like this: In [1]: from none_aware import NoneAware In [2]: from types import SimpleNamespace In [3]: foo = SimpleNamespace() In [4]: foo.bar = SimpleNamespace() In [5]: foo.bar.baz = SimpleNamespace() In [6]: foo.bar.baz.blat = 42 In [7]: NoneAware(foo).bar.blim Out[7]: <none_aware.NoneAware at 0x11156a748> In [8]: NoneAware(foo).bar.blim.unbox() In [9]: NoneAware(foo).bar.baz.blat.unbox() Out[9]: 42 In [10]: NoneAware(foo).bar.baz.blat Out[10]: <none_aware.NoneAware at 0x11157d908> In [11]: NoneAware(foo).bar.baz.flam.unbox() In [12]: NoneAware(foo).bar.baz.flam Out[12]: <none_aware.NoneAware at 0x1115832b0> The particular names I use are nothing special, and better ones might be found. I just called the class NoneAware and the "escape" method `.unbox()` because that seemed intuitive at first brush. I don't disagree that needing to call .unbox() at the end of the chained attribute access is a little bit ugly. But it's a lot less ugly than large family of new operators. And honestly, it's a nice way of being explicit about the fact that we're entering then leaving a special world where attribute accesses don't fail. I haven't implemented the equivalent dictionary lookups in the below. That would be straightforward, and I'm sure my 5 minute throwaway code could be improved in other ways also. But something better than this in the standard library would address ALL the actual needs described in PEP 505. Even the pattern Steve Dower is especially fond of like: favorite = cfg?.user?.profile?.food ?? "Spam" (i.e. a configuration may be incomplete at any level, if levels are missing default favorite food is Spam). We could simply spell that: favorite = NoneAware(cfg, "Spam").user.profile.food.unbox() I think that's 14 characters more in this example, but still compact. We could get that down to 2 characters if we used one-letter names for the class and method. I suppose down to zero characters if .unbox() was a property. So completely toy implementation: class NoneAware(object): def __init__(self, thing, sentinal=None): self.thing = thing self.sentinal = sentinal def __getattr__(self, attr): try: return NoneAware(getattr(self.thing, attr)) except AttributeError: return NoneAware(self.sentinal) def unbox(self): return self.thing -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

14 25

Fwd: A better (simpler) approach to PEP 505
by David Mertz 24 Jul '18

24 Jul '18

I started a new thread where I show a small library I made yesterday that ALSO matches 505 semantics, as well as more general AttributeError swallowing. It has two classes with sightly different behavior. So we don't need syntax for either semantics. On Tue, Jul 24, 2018, 3:29 AM Grégory Lielens <gregory.lielens(a)gmail.com> wrote: > > > On Tuesday, July 24, 2018 at 2:39:19 AM UTC+2, Steven D'Aprano wrote: > >> PEP 505 has a section explaining why catching AttributeError >> is undesirable. I find the reasons it gives are compelling. >> >> Can you explain why you reject the PEP's arguments against catching >> AttributeError? >> >> > > Yes, and people have done it multiple time already. I will try to make it > as explicit as possible, with a terminology that some may find useful... > when u have a object hierarchy linked through attributes, which -seems- > the use pattern of ?. , the hierarchy is either > - 1) fully regular (each attribute contains the same type of objects (same > type=with the same attributes), then normal . access is what you want > - 2) regular-or-None (contain same type of objects, or None). That's what > ?. is for, and ?. is only for that. > - 3) its partially regular (does not contain the same type of objects, > some objects have more attributes than other, e.g. partial objects). > - 4) its irregular (objects may be of completely different type (int, > str,... and do not specailly share attributes). > > ?. is intended for 2). > 2) and 3) can be dealed with swallowing AttributeError > 4) can also be traversed swallowing AttributeError, but it's probably a > very bad idea in almost all cases. > > I've encountered all cases, from time to time, but none is really typical > of most of my code. Which is more common? not sure, I would say 3). > What is sure is that 2) do not stand as hugely more common than the > others, so introducing a special syntax for 2) do not please me, so +0... > Now as I find ?. unpleasant to parse (personal opinion), and having grown > to share the general operaror / line noise allergy mainstream in the python > community, i am a firm -1 on this: too small a niche, do not bring much > compared to current approach, and visual line noise. > firm -1 on ?? too, it's less polemic, less visually displeasing, but even > less usefull I think: it does not gain that much compared to a more general > ternary. > > As the pymaybe/this thread cover 2) and 3), and do it without any change > to python, I find it better. No change > lib change > backward compatible > object change - like dict becoming ordered, or None triggering > NoneAttributeError(AttributeError) > syntax change (expecially a bunch of > linenoisy operators). > > Now I will stop for a while here, arguments have become circular, > proponents and opponents have exposed their view, but mostly proponents > have provided very few additional information for a while, especially about > real use cases, despite a few requests. > We still do not know what motivated the proposal..We had to guess (JSON). > Even the PEP do not say, it provide a set of real world example from the > standard library, which is admirable but different. And personaly, while am > very thankful for providing those concrete examples, I find them > unconvicing: the code before is usually fine and improved little by the new > operators. Sometimes it's worse: only slightly shorter and less clear. > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ >

1 0

A better PEP 505 (test package: coalescing)
by David Mertz 23 Jul '18

23 Jul '18

I proposed a very toy example of a coalescing class that would solve the problem solved by brand new syntax in PEP 505. In my 5 minute version, there were several faults or limitations. For one, the class I wrote was misnamed, since it wasn't really about None-coalescing, but rather about exception-free nested access. I've written a more robust version at https://pypi.org/project/coalescing/. I encourage issues or contributions at https://github.com/DavidMertz/coalesce. Mind you, this is still a few-hour-of-effort version, not rigorously tested. I want to thank Antoine Pitrou for proposing using Graham Dumpleton's warpt.ObjectProxy (which is truly magic). In almost no lines, I think I have something perhaps even better than PyMaybe. In particular, most of the time, there is no need to do an explicit `.unbox()` call... but at the suggestion of several people, I changed my first API to optionally pass a default value other than None within that method, if desired. I even made laziness in getting that value the default (but switchable) behavior. Dumpleton's magic means that boxed values can do pretty much anything the raw values can, so can generally ignore the .unbox() call at the end. Perhaps most magic of all, you can even ASSIGN into proxied values and that gets written to the original value (not shown in the doctests below). This adds something powerful that PEP 505 is completely unable to do. I believe that I have captured ALL the semantics of PEP 505 with no changes to Python syntax... and in a way that reads FAR better than all those new operators do. Access messy nested data structures GreedyAccess will keep marching down trees even if failure occurred earlier: >>> from coalesce import GreedyAccess, make_test >>> cfg = make_test() >>> GreedyAccess(cfg).user.profile.song <GreedyAccess proxy for 'Nightclubbing'> >>> GreedyAccess(cfg).user.profile.song + ' and spam' 'Nightclubbing and spam' >>> GreedyAccess(cfg).user.profile.food <GreedyAccess proxy for None> >>> print(GreedyAccess(cfg).user.profile.food.unbox()) None >>> GreedyAccess(cfg).user.profile.food.unbox('spam') 'spam' >>> GreedyAccess(cfg).user.profile.food.unbox('spam') + ' and spam' 'spam and spam' NoneCoalesce only descends until a None is encountered. Accessing attributes or keys of None will still fail: >>> from coalesce import NoneCoalesce >>> NoneCoalesce(cfg).user.profile.song <NoneCoalesce proxy for 'Nightclubbing'> >>> NoneCoalesce(cfg).user.profile.song.unbox() 'Nightclubbing' >>> NoneCoalesce(cfg).user.profile.food Traceback (most recent call last): ... AttributeError: 'types.SimpleNamespace' object has no attribute 'food' >>> NoneCoalesce(cfg).user.profile <NoneCoalesce proxy for namespace(arms=2, song='Nightclubbing')> >>> val = None >>> print(NoneCoalesce(val).attr) None We provide for returning values other than None if some other default is more useful for your use case (a zero-argument lambda function would often be useful here): >>> def say_spam(): ... return "spam" ... >>> GreedyAccess(cfg).user.profile.food.unbox(say_spam) 'spam' >>> GreedyAccess(cfg).user.profile.food.unbox(say_spam, lazy=False) #doctest: +ELLIPSIS <function say_spam ...> -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th.

1 0