Mailman 3 June 2018 - Python-ideas

Fwd: Allow a group by operation for dict comprehension
by David Mertz June 29, 2018

June 29, 2018

Mike Selik asked for my opinion on a draft PEP along these lines. I proposed a slight modification to his idea that is now reflected in his latest edits. With some details fleshed out, I think this is a promising idea. I like the a collections class better, of course, but a dict classmethod is still a lot smaller change than new syntax change in comprehension. On Thu, Jun 28, 2018, 8:15 PM David Mertz <mertz(a)gnosis.cx> wrote: > I see the utility, but I would prefer a slightly … [View More]different approach than > you suggest; I think my suggestion will have a lower barrier to acceptance > as well. > > Rather than add a new classmethod dict.grouper(), I'd like to have a new > dict subclass collections.Grouper. The name subject to bikeshedding, of > course. I think of this class as a "big sister" of collections.Counter, in > a way. > > There is behavior that I believe would be useful beyond constructing a new > base dictionary. However, I think that construction from an iterable would > be a common use pattern. Oh, I'd also recommend following toolz.groupby() > in keeping a list rather than a set. It's easy enough to convert a list to > a set if wanted, but order and repetitions are preserved in SQL or Pandas > 'groupby' operations, and that seems more general. > > For example (this typed without testing, forgive any typos or thinkos): > > >>> from collections import Grouper # i.e. in Python 3.8+ > >>> grouped = Grouper(range(7), key=mod_2) > >>> grouped > Grouper({0: [0, 2, 4, 6], 1: [1, 3, 5]}) > >>> grouped.update([2, 10, 12, 13], key=mod_2) > >>> grouped > Grouper({0: [0, 2, 4, 6, 2, 10, 12], 1: [1, 3, 5, 13]}) > >>> # Updating with no key function groups by identity > >>> # ... is there a better idea for the default key function? > >>> grouped.update([0, 1, 2]) > >>> grouped > Grouper({0: [0, 2, 4, 6, 2, 10, 12, 0], 1: [1, 3, 5, 13, 1], 2: [2]}) > >>> # Maybe do a different style of update if passed a dict subclass > >>> # - Does a key function make sense here? > >>> grouped.update({0: 88, 1: 77}) > >>> grouped > Grouper({0: [0, 2, 4, 6, 2, 10, 12, 0, 88], > 1: [1, 3, 5, 13, 1, 77], > 2: [2]}) > >>> # Avoiding duplicates might sometimes be useful > >>> grouped.make_unique() # better name? .no_dup()? > >>> grouped > Grouper({0: [0, 2, 4, 6, 10, 12, 88], > 1: [1, 3, 5, 13, 77], > 2: [2]}) > > I think that most of the methods of Counter make sense to include here in > appropriately adjusted versions. Converting to a plain dictionary should > probably just be `dict(grouped)`, but it's possible we'd want > `grouped.as_dict()` or something. > > One thing that *might* be useful is a way to keep using the same key > function across updates. Even with no explicit provision, we *could* > spell it like this: > > >>> grouped.key_func = mod_2 > >>> grouped.update([55, 44, 22, 111], key=grouped.key_func) > > Perhaps some more official API for doing that would be useful though. > > > > > > On Thu, Jun 28, 2018 at 7:35 PM David Mertz <mertz(a)gnosis.cx> wrote: > >> Thanks... Looking now. I'll comment soon. >> >> On Thu, Jun 28, 2018 at 7:05 PM Michael Selik <mike(a)selik.org> wrote: >> >>> Hi David, >>> >>> We talked about this in Seattle about a year ago at a conference. Would >>> you do me a favor and critique this PEP I've drafted? I'd like to get >>> private feedback before sharing with the group. >>> >>> https://github.com/selik/peps/blob/master/pep-9999.rst >>> >>> Thank you, >>> -- Michael >>> >>> >>> On Thu, Jun 28, 2018 at 1:35 PM David Mertz <mertz(a)gnosis.cx> wrote: >>> >>>> I agree with these recommendations. There are excellent 3rd party tools >>>> that do what you want. This is way too much to try to shoehorn into a >>>> comprehension. >>>> >>>> I'd add one more option. You want something that behaves like SQL. >>>> Right in the standard library is sqlite3, and you can create an in-memory >>>> DB to hope the data you expect to group. >>>> >>> [View Less]

3 2

list configuration
by Chris Barker June 29, 2018

June 29, 2018

Hey all, I've been replying to messages lately, and getting a bounce back: """ Hello chris.barker(a)noaa.gov, We're writing to let you know that the group you tried to contact (python-ideas) may not exist, or you may not have permission to post messages to the group. A few more details on why you weren't able to post: """ And it's not quite clar to me if the message actually got through. IIUC, this is a Mailman list -- so it must be getting mirrored through google groups, and at least with … [View More]

5 4

Re: [Python-ideas] Allow mutable builtin types (optionally)
by Greg June 28, 2018

June 28, 2018

As I introduced (a long time ago) this demand, let me add my grain of salt here. The use case is pretty simple, and somewhat common when writing manually C extension class: The reason to write extension class is usually performance, or link into an existing library. When doing this manually (instead of using automatic python-wrapping tools like boost, swig,...) you try to wrap the minimum amount of methods/accessors/...to your underlying c/c++ class, and replicate non-critical methods in … [View More]python. Moreover, extending your class by adding new methods is usually much more easy in Python, especially if it involve complex but not performance-bounded python-data manipulation. Problem is to make those python-implemented methods avaible to instances of your extension class, especially when those instances are returned by the C layer of your extension. The solution we choose was to change the __class__ of each extension type instance to the python derived newclass implementing all those extra-methods.Not too difficult, a simple encapsulation of all methods returning extension-class instances is enough, and can be automated. This solution is quite common I think, it translate something you do for python-class instances, but then you get the __class__ assignment: only for heap types error. The argument about sub-interpreters is a good one, but not really applicable for this use case: we really want to have one extension type (or a hierarchy of it) shared across all interpreter importing the extension, it just happen that instead of being implemented in pure C/C++, the extension is implemented in C/C++ and Python. The fact that the python parts will be seen everywhere is a feature, not a problem: you expect the replacement of C-implemented methods by Python-implemented method to be as transparent as possible. Alternatives would be to use a heap type for our C extensions classes (we need to check what it would imply, but it may be quite painless) or use some form or delegation instead of assigning to __class__. The later is not really painless, AFAIK, in term of coding complexity and possibly performance (extra lookups steps needed). If there are other solutions or if delegation can be made as simple/efficient as the __class__ mechanism, it would be good to know, and it is I think valuable info for many people writing extension classes. Anyway, my personal position on this has not changed in 10y and is in line with Eloi: I think that beeing a heaptype and allowing assigment to the __class__ attribute of instances is indeed quite orthogonal.. . [View Less]

2 2

Re: [Python-ideas] Delivery Status Notification (Failure)
by Jacco van Dorp June 27, 2018

June 27, 2018

Have you tried just instantiating a Counter() instead ? All missing keys are considerd to be 0 in a fresh counter. So for example: >>> c = Counter() >>> c["a"] += 1 >>> c Counter({'a': 1}) >>> c["b"] 0 works exactly this way. Which means there's no difference between what you're suggesting Counter.fromkeys(key-list-or-set, 0) should do and what Counter() actually does. (Resend because somehow gmail tried to reply to the wrong python-ideas) 2018-06-27 22:… [View More]

1 0

"Exposing" `__min__` and `__max__`
by James Edwards June 27, 2018

June 27, 2018

I've only recently looked for these special methods, so that in and of itself may be the reason these methods aren't exposed, but I could think of objects that may wish to implement __min__ and __max__ themselves, for efficiency. For example: # A "self-sorted" list object class AlwaysSortedListObejct: def __min__(self): return self.lst[0] def __max__(self): return self.lst[-1] # An object that maintains indices of extrema (e.g. for complex comparisons) class … [View More]

10 24

collections.Counter should implement fromkeys
by Abe Dillon June 27, 2018

June 27, 2018

Consider the following update function for conway's game of life: from collections import Counter def update(live: Set[Tuple[Integer, Integer]]): counts = Counter.fromkeys(live, 0) + Counter(itertools.chain(neighbors(* cell) for cell in live)) flip = {cell for cell, count in counts.items() if (cell in live and not 1<count<4) or (cell not in live and count==3)} live ^= flip around = frozenset(filter(any, itertools.product(range(-1,2), range(-1,2)))) … [View More]

1 0

random.sample should work better with iterators
by Abe Dillon June 27, 2018

June 27, 2018

The docs on random.sample indicate that it works with iterators: > To choose a sample from a range of integers, use a range() > <https://docs.python.org/3/library/stdtypes.html#range> object as an > argument. This is especially fast and space efficient for sampling from a > large population: sample(range(10000000),k=60). However, when I try to use iterators other than range, like so: random.sample(itertools.product(range(height), range(with)), 0.5*height*width) I get: … [View More]

6 7

Correct way for writing Python code without causing interpreter crashes due to parser stack overflow
by Fiedler Roman June 27, 2018

June 27, 2018

Hello List, Context: we are conducting machine learning experiments that generate some kind of nested decision trees. As the tree includes specific decision elements (which require custom code to evaluate), we decided to store the decision tree (result of the analysis) as generated Python code. Thus the decision tree can be transferred to sensor nodes (detectors) that will then filter data according to the decision tree when executing the given code. Tracking down a crash when executing that … [View More]generated code, we came to following simplified reproducer that will cause the interpreter to crash (on Python 2/3) when loading the code before execution is started: #!/usr/bin/python2 -BEsStt A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A([A(None)])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])])]) The error message is: s_push: parser stack overflow MemoryError Despite the machine having 16GB of RAM, the code cannot be loaded. Splitting it into two lines using an intermediate variable is the current workaround to still get it running after manual adapting. As discussed on Python security list, crashes when loading such decision trees or also mathematical formulas (see bug report [1]) should not be a security problem. Even when not directly covered in the Python security model documentation [2], this case comes too close to "arbitrary code execution", where Python does not attempt to provide any protection. There might be only some border cases of affected software, e.g. Python sandbox systems like Zope/Plone or maybe even Python based smart contract blockchains like Etherereum (do not know if/where the use/derived work from the default Python interpreter for their use). But in both cases they would also be too close violating the security model, thus no changes to Python required from this side. Thus Python security suggested that the discussion should be continued on this list. Even when no security problem involved, the crash is still quite an annoyance. Development of code generators can be a tedious tasks. It is then somehow frustrating, when your generated code is not accepted by the interpreter, even when you do not feel like getting close to some system-relevant limits, e.g. 50 elements in a line like above on a 16GB machine. You may adapt the generator, but as the error does not include any information, which limit you really violated (number of brackets, function calls, list definitions?) you can only do experiments or look on the Python compiler code to figure that out. Even when you fix it, you have no guarantee to hit some other obscure limit the next day or that those limits change from one Python minor version to the next causing regressions. Questions: * Do you deem it possible/sensible to even attempt to write a Python language code generator that will produce non-malicious, syntactically valid decision tree code/mathematical formulas and still having a sufficiently high probability that the Python interpreter will also run that code now and in near future (regressions)? * Assuming yes to the question above, when generating code, what should be the maximal nesting depth a code generator can always expect to be compiled on Python 2.7 and 3.5 on? Are there any other similar restrictions that need to be considered by the code generator? Or is generating code that way not the preferred solution anyway - the code generator should generate e.g. binary python code immediately? Note: in the end the exact same logic code will run as Python process, it seems it is only about how it is loaded into the Python interpreter. * If not possible/recommended/sensible, we might generate Java-bytecode or native x86-code instead, where the likelihood of the (virtual) CPU really executing code that is compliant to the language specification (even with CPU errata like FDIV-bug et al) might be magnitudes higher than with the Python interpreter. Any feedback appreciated! Roman [1] https://bugs.python.org/issue3971) [2] http://python-security.readthedocs.io/security.html#security-model [View Less]

5 7

Have a "j" format option for lists
by Facundo Batista June 27, 2018

June 27, 2018

This way, I could do: >>> authors = ["John", "Mary", "Estela"] >>> "Authors: {:, j}".format(authors) 'Authors: John, Mary, Estela' In this case the join can be made in the format yes, but this proposal would be very useful when the info to format comes inside a structure together with other stuff, like... >>> info = { ... 'title': "A book", ... 'price': Decimal("2.34"), ... 'authors: ["John", "Mary", "Estela"], ... } ... >>> print("{title!r} (${price}… [View More]

14 29

Add an optional type file for Type Annotation
by 李者璈 June 26, 2018

June 26, 2018

I'm inspired by TypeScript. TypeScript allows people to add **.d.ts* *to annotate ECMAScript's type. So this can expand the TypeScript's scenes to be used。 So I think can we add an optional mechanism to allow people add the extra file to annotate the existing code. Type Annotation can be used after Python 3.5. but many lib/framework has to be compatible for the Python 3.0-3.4, they can't use annotation. If we have this mechanism, the lib/framework can continue to focus on the function, … [View More]

2 1