Mailman 3 February 2020 - Python-ideas

Add Binary module.
by ananthakrishnan15.2001＠gmail.com Feb. 20, 2020

Feb. 20, 2020

This module should contain operations that can be performed on binary numbers. In the below examples a and b are binary numbers. binary.add(a,b) binary.sub(a,b) binary.mul(a,b) binary.div(a,b) binary.ones_complement(a) //returns 1's complement of a. binary.twos_complement(a) //returns 2's complement of a. binary.dectobi(a) //converts decimal "a" to binary of "a". binary.bcd(a) //returns bcd of a. binary.gray(a) //returns grey code of a. binary.min() //minimizes. binarybitodec //converts binar "a" to decimal of "a".

12 35

Re: Pandas rookie
by James Lu Feb. 20, 2020

Feb. 20, 2020

On Wed, Feb 19, 2020 at 5:23 PM FilippoM <privacy_please(a)filippo.it> wrote: > > Hi, I've got a Pandas data frame that looks like this > > In [69]: data.head > Out[69]: > <bound method NDFrame.head of OS and Version Status > 0 Android VIDEO_OK > 1 Android 4.2.2 VIDEO_OK > 2 Android 9 VIDEO_OK > 3 iOS 13.3 VIDEO_OK > 4 Windows 10 VIDEO_OK > 5 Android 9 VIDEO_OK > ... ... > 24 Windows 10 VIDEO_OK > 25 Android 9 VIDEO_OK > 26 Android 6.0.1 VIDEO_OK > 27 Windows XP VIDEO_OK > 28 Android 8.0.0 VIDEO_FAILURE > 29 Android 6.0 VIDEO_OK > ... ... > 2994 iOS 9.1 VIDEO_OK > 2995 Android 9 VIDEO_OK > 2996 Windows 10 VIDEO_OK > 2997 Android 9 VIDEO_OK > 2998 Windows 10 VIDEO_OK > 2999 iOS 13.3 VIDEO_OK > > > with 109 possible values of the OS columns and just two possible values > ()VIDEO_OK and VIDEO_FAILURE) in the status column. > > How can I use Pandas' dataframe magic to calculate, for each of the > possible 109 values, how many have VIDEO_OK, and how many have > VIDEO_FAILURE I have respectively? > > I would like to end up with something like > > In[]: num_of_oks{"iOS 13.3"} > Out: 15 > > In[]: num_of_not_oks{"iOS 13.3"} > Out: 3 > > I am trying to do some matplotlib scatter plotting > > Thanks > > > -- > https://mail.python.org/mailman/listinfo/python-list Have you considered using traditional unix tools, like cut and count? Or traditional SQL.

2 1

Add __keys__ or __items__ protocol
by Serhiy Storchaka Feb. 19, 2020

Feb. 19, 2020

It is hard to distinguish a sequence and a mapping in Python 3. Both have __getitem__, __len__ and __iter__ methods, but with different semantic. In Python code you can use isinstance() and issubclass() checks against collections.abc.Mapping, but it works only with types which was explicitly registered in Mapping. Autodectection does not work as for Iterable or Hashable. Also importing collections.abc.Mapping and calling PyObject_IsInstance() or PyObject_IsSubclass() is cumbersome and inefficient in the C code. It is rarely used. It is more common to check for the existence of the keys() method. The dict constructor and update() method do this, as well as other code which emulates the behavior of dict. This is the only non-dunder method used by the core operations. There are problems with this: * Since it is not a dunder method, it is not reserved, and may conflicts with the user attribute. * Since it is not a dunder method, it is checked for the instance, not for the type. Instances of the same type can look as a mapping or as not a mapping for the same code. Checking the attribute of the class does not give you information about the attribute of the instance. * There is no a corresponding slot in the type object. So checking the existence of the keys attribute is slow, non-atomic, can execute an arbitrary code and raise an exception. It can't be used in PyMapping_Check() and slows down the dict constructor. * The keys() method is not called in the dict constructor. Just the existence of the keys attribute is checked, its value is not used. * It is a special case. All other implicitly called methods which determined the behavior of builtin types are dunder names. I propose to add support for special methods `__keys__` or `__items__` and corresponding type slot (`tp_mapping->mp_keys` or `tp_mapping->mp_items`). At first stage, the code which checked for "keys" should check also for the special method and use them if defined. At second stage it should emit a warning if "keys" is defined, but the special method is not defined. At third stage, it should use the result of the special method and ignore "keys". PyMapping_Check() can be made checking the corresponding slot. I am not sure about some details: 1. What special method should be added, `__keys__` or `__items__`? The former returns keys, it needs to call `__getitem__` repeatedly to get values. The latter returns key-value pairs, it does not need to call `__getitem__`, but you should always pay the cost of creating a tuple even if you do not need values. 2. What type should they return? * An iterator. * An iterable which can be iterated only once. Easy to implement in Python as generator methods. * An iterable which can be iterated multiple times. * More complex view object which may support other protocols (for example support `__or__` and `__and__`). What do you think about this?

9 10

Add a way to execute code in the context of different frame
by Serhiy Storchaka Feb. 18, 2020

Feb. 18, 2020

The idea is inspired by the following StackOverflow question: https://stackoverflow.com/questions/40945752/inspect-who-imported-me and the corresponding BPO issue: https://bugs.python.org/issue39643. In the older Python versions the f_back attribute of the frame in which the module code is executed pointed to the frame which executes the import statement or calls __import__() or import_module(). But after rewriting the import machinery in Python, there are other frames between the caller frame and the calling frame. This caused problems with tracebacks which was worked around by removing intermediate frames related to the import machinery when the exception floats to the caller. But there is still unresolved problem with warnings, and all frames of the import machinery are visible from the imported code. I propose to add possibility to execute the code in the context of different frame. Either add a frame argument in exec() and eval(), which will allow to pass an arbitrary frame. Or add a stacklevel argument in exec() and eval() (similar to warnings.warn()), which will limit possible frames to the parent frames of the current frame. I do not know what is more feasible. This will automatically fix problems with warnings. This will allow to get rid the workaround for exceptions, and may make the code simpler. This will allow the imported code to do miscellaneous magic things with the code which performs the import. What do you think about this?

3 5

A new itertools.product variant that supports kwargs and returns dicts
by Daniel Grießhaber Feb. 18, 2020

Feb. 18, 2020

When doing a grid-search over some hyper-parameters of an algorithm I often need to specify the possible value space for each parameter and then evaluate on every combination of these values. In this case, the product function of the itertools module is handy: from itertools import product def train_and_evaluate(lr, num_layers, dataset): print(f'training on {dataset} with {num_layers=} and {lr=}') params = ( (0.01, 0.1, 0.5), #learning rate range(10, 40, 10), #number of hidden neurons ('MNLI', 'SNLI', 'QNLI') # training dataset ) for config in product(*params): train_and_evaluate(*config) However, this code relies on the order of parameters in the function train_and_evaluate, which could be considered not very pythonic, as explicit is better than implicit. In the same way, the 'intention' for the values is only clear with the context of the function signature (or the comments, if someone bothered to write them). Therefore I propose a new variant of the product() function that supports Mappings instead of Iterables to enable to explicitly specify the function of an iterables values in the product. A trivial implementation could be: def named_product(repeat=1, **kwargs): for combination in itertools.product(*kwargs.values(), repeat=repeat): yield dict(zip(kwargs.keys(), combination)) which could then be used like this: params = { 'lr': (0.01, 0.1, 0.5), #learning rate 'num_layers': range(10, 40, 10), #number of hidden neurons 'dataset': ('MNLI', 'SNLI', 'QNLI') # training dataset } for config in named_product(**params): train_and_evaluate(**config) This has the advantage that the order of the parameters in the kwargs does not depend on the method signature of train_and_evaluate. I also would appreciate your input on the following questions: - Support scalar values? I the example use-case it may be nice to not have each kwarg to be an iterable, because you may want to specify all parameters to the function, even if they do not vary in the product items (factors?) params = named_product(a=(1, 2), b=3) instead of params = named_product(a=(1, 2), b=(3, )) However, this may be unexpected to users when they supply a string which would then result in an iteration over its characters. - Would this may be suited for the more-itertools package? However, I would love to have this build-in and not depend on a non-stdlib module - More functionality: In my toolbox-package I implemented a variant with much more functionality: [https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html](https:/…. This version offers additional functionality compared to the proposed function: - accept both, a dict as *arg or **kwargs and merges them - accepts scalar values, not only iterables but treats strings as scalars - if the value of any item is a dict itself (not scalar or Sequence), iterate over the nested dict as a seperate call to named_product would and update the 'outer' dict with those values - copies values before yielding which is useful if the returned values are mutable (can be excluded) Maybe some of these features would also be useful for a general version? However, this version is mostly developed for my special use-case and some behaviour is solely based on my peculiarities ## Example usage: Here is an example from my actual code which may clarify some decisions on functionality of the features of [https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html](https:/… configs = named_product({ 'nb_epoch': 3, 'acquisition_iterations': 0, 'training_function': { partial(train_bert, decoder_function=create_decoder):{ 'decoder_name': 'FFNN' }, partial(train_bert, decoder_function=create_cnn_decoder):{ 'decoder_name': 'CNN' } }, 'training_data': partial(load_glue, initial_training_data_size=10), 'number_queries': 100, 'batch_size': 64, 'dataset_class': [QNLIDataset, MNLIDataset], 'num_frozen_layers': 0, 'acquire_function': { acquire_uncertain: { 'certainty_function': calculate_random, 'predict_function': 'predict', 'dropout_iterations': 1 } } })

1 0

a new itertools.product variant that supports kwargs and returns dicts
by Daniel Grießhaber Feb. 18, 2020

Feb. 18, 2020

When doing a grid-search over some hyper-parameters of an algorithm I often need to specify the possible value space for each parameter and then evaluate on every combination of these values. In this case, the product function of the itertools module is handy: from itertools import product def train_and_evaluate(lr, num_layers, dataset): print(f'training on {dataset} with {num_layers=} and {lr=}') params = ( (0.01, 0.1, 0.5), #learning rate range(10, 40, 10), #number of hidden neurons ('MNLI', 'SNLI', 'QNLI') # training dataset ) for config in product(*params): train_and_evaluate(*config) However, this code relies on the order of parameters in the function train_and_evaluate, which could be considered not very pythonic, as explicit is better than implicit. In the same way, the 'intention' for the values is only clear with the context of the function signature (or the comments, if someone bothered to write them). Therefore I propose a new variant of the product() function that supports Mappings instead of Iterables to enable to explicitly specify the function of an iterables values in the product. A trivial implementation could be: def named_product(repeat=1, **kwargs): for combination in itertools.product(*kwargs.values(), repeat=repeat): yield dict(zip(kwargs.keys(), combination)) which could then be used like this: params = { 'lr': (0.01, 0.1, 0.5), #learning rate 'num_layers': range(10, 40, 10), #number of hidden neurons 'dataset': ('MNLI', 'SNLI', 'QNLI') # training dataset } for config in named_product(**params): train_and_evaluate(**config) This has the advantage that the order of the parameters in the kwargs does not depend on the method signature of train_and_evaluate. Open Questions - Support scalar values? I the example use-case it may be nice to not have each kwarg to be an iterable, because you may want to specify all parameters to the function, even if they do not vary in the product items (factors?) params = named_product(a=(1, 2), b=3) instead of params = named_product(a=(1, 2), b=(3, )) However, this may be unexpected to users when they supply a string which would then result in an iteration over its characters. - Would this may be suited for the more-itertools package? However, I would love to have this build-in and not depend on a non-stdlib module - More functionality: In my toolbox-package I implemented a variant with much more functionality: <https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html> https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html. This version offers additional functionality compared to the proposed function: - accept both, a dict as *arg or **kwargs and merges them - accepts scalar values, not only iterables but treats strings as scalars - if the value of any item is a dict itself (not scalar or Sequence), iterate over the nested dict as a seperate call to named_product would and update the 'outer' dict with those values - copies values before yielding which is useful if the returned values are mutable (can be excluded) Maybe some of these features would also be useful for a general version? However, this version is mostly developed for my special use-case and some behaviour is solely based on my peculiarities Example usage: Here is an example from my actual code which may clarify some decisions on functionality of the features of <https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html> https://py-toolbox.readthedocs.io/en/latest/modules/itertools.html configs = named_product({ 'nb_epoch': 3, 'acquisition_iterations': 0, 'training_function': { partial(train_bert, decoder_function=create_decoder):{ 'decoder_name': 'FFNN' }, partial(train_bert, decoder_function=create_cnn_decoder):{ 'decoder_name': 'CNN' } }, 'training_data': partial(load_glue, initial_training_data_size=10), 'number_queries': 100, 'batch_size': 64, 'dataset_class': [QNLIDataset, MNLIDataset], 'num_frozen_layers': 0, 'acquire_function': { acquire_uncertain: { 'certainty_function': calculate_random, 'predict_function': 'predict', 'dropout_iterations': 1 } } })

1 0

Re: Add Binary module.
by Rupert Spann Feb. 17, 2020

Feb. 17, 2020

The module should also support bit masking (left or right justified) with AND / OR / XOR / NotAND / NotOR / NotXOR operations. ieg following the same structure: binary.AND(a,b) binary.OR(a,b) binary.XOR(a,b) binary.NOTAND(a,b) binary.NOTOR(a,b) binary.NOTXOR(a,b) The number of larger length of bits of a or b is used and the other padded with '0' or '1' Default could be: right justified and '0' padding on the left option1: left justified option2: padding with '1'

1 0

Re: Add Binary module.
by Rupert Spann Feb. 17, 2020

Feb. 17, 2020

The module should also support bit masking (left and/or right justified) with AND / OR / XOR / NotAND / NotOR / NotXOR operations.

1 0

Exception-aware operators
by Soni L. Feb. 17, 2020

Feb. 17, 2020

I looked at the PEP for None-aware operators and I really feel like they miss one important detail of Python's data model: >>> {}[0] Traceback (most recent call last): File "<stdin>", line 1, in <module> KeyError: 0 >>> [][0] Traceback (most recent call last): File "<stdin>", line 1, in <module> IndexError: list index out of range that is: missing items raise, rather than being None. as such I feel like None-aware operators would encourage ppl to put None everywhere, which from what I can tell, goes completely against Python's data model. however, exception-aware operators could be a potential alternative that: 1. works with existing code (no need to go shoving None everywhere) 2. works in lambdas 3. ends up being more useful? I don't know what the potential syntax would look like. perhaps something along the lines of: exception-raising-expression ?: value-if-exception-raised or: exception-raising-expression ? (IndexError, KeyError, Etc) : value-if-exception-raised but, thoughts?

4 4

Re: Traits
by Stephen J. Turnbull Feb. 17, 2020

Feb. 17, 2020

Soni, Answering off-list by moderator request. Before I go on to respond to the details, let me start by saying I'm definitely beginning to understand why you like the Rust syntax for traits, and maybe even some feeling for the semantics, although they're not very well documented anywhere I found in a couple hours of Googling and browsing. On the other hand, I don't think some of Rust's semantics are a good fit for Python (and go into that below). Soni L. writes: > zope.interface appears to be more based on java interfaces than rust > traits? I could be wrong tho. zope.interface precedes Rust traits by at least two decades. I don't know what the relationship is between zope.interface and Java's interface is. Java was released in 1995, Zope in 1998, but Zope's predecessor, the "Bobo object publisher," was released to the public in the early 1990s. Of course interface checking in a very primitive form (function prototypes) go back to C++'s predecessor "C with Classes" in 1979, and I imagine Lisp has had them even longer, and in more elaborated form. C++ got virtual functions in 1985, which is much closer to the modern concept of separating interface from implementation, and abstract classes (classes containing one or more pure virtual functions) in 1989. So this stuff was in the air for several years before either Python or Java started development. > *forget* Schärli et al., this is too hard to explain if you keep > expecting Schärli et al. behaviour Not gonna happen. Thing is, I can hold two behaviors in my mind at the same time, but it would help if you paid some respect to the seminal paper that preceded Rust by more than a decade. At least call them "Rust traits". But that doesn't help much, since the Rust documents are really sparse, even about behavior, let alone about rationale for the behavior. I had to Google bug reports to find even a mention of method name collisions. > when Rust quite explicitly doesn't seem to follow that. I now think you're wrong about Rust following Schärli et al., although it's impossible to be sure since there are differences, and the design decision rationales aren't to be found in the places I looked. > now everyone's been gaslighting me because I got fed up with this > and idk what to do about that. but anyway, "Gaslighting" is lying about something that both parties can see is false. That's not happening here. There's definitely a lot of confusion about what you've been talking about. It's one thing to ask one person (such as me) to learn Rust; it's another to ask all the core developers to do so. It's reasonable for you to say that your time is worth as much as anybody else's, even Guido's (though I don't recommend saying *that* out loud!) It's not reasonable to say that your time is worth as much as everybody else's. > > In the TOSWidget example, as I understand it, in Rust: > > > > 1. In the trivial case where exactly one of TOSWidget, Pack, and > > Place implements 'config', you get that implementation, and > > any code that's not happy with that is incorrect. > > Yes, but also not quite. Rust has a bit of implicitness here for > ergonomics. If there's no conflict, it can easily figure out what you > want/meant. If there's no conflict, strait figures out what you want, too. > Also, it's not that Pack or Place implements 'config' - they > *define* a 'config' interface. TOSWidget implements Pack.config > and/or Place.config. It can, but it doesn't necessarily do so. The traits can supply default implementations. For reasons I don't understand (but presume are valid), Rust emphasizes traits' role in providing interfaces over code-sharing. > > > 2. In the case TOSWidget implements 'config', and Pack or Place or > > both implements it, you get TOSWidget's implementation, but > > anything expecting Pack or Place behavior will not get it, and is > > incorrect. > > No, anything expecting Pack behaviour will get Pack behaviour, anything > expecting Place behaviour gets Place behaviour, and anything dealing > with TOSWidget directly gets TOSWidget behaviour. They're effectively > namespaced. "Expect" here means what the author of the code that invokes a method on a TOSWidget object expects. Of course you can use UFCS to disambiguate, but that's not what most of us mean by "expecting", that's explicitly invoking the behavior. In Python, "expecting" means "duck typing (ie, a method invocation) does the right thing", but UFCS is effectively a cast, and uses normal function call syntax. In English, your use of that word is perfectly correct, but I guarantee using it that way in this context will totally confuse most Pythonistas. > > 3. If both Pack and Place implement 'config', but TOSWidget does not, > > you get a compile-time error. > > No, unless you try to interact with 'config' on a TOSWidget. Of course. That's what most Python programmers expect. "Duck typing." > The error happens in the caller, not the implementer, and then you > need to disambiguate, in the *caller*, which implementation you > meant to call. I really don't like that *for Python*. Rust can do it at low risk because it resolves the call at compile time. Python doesn't, so this means a runtime exception, possibly after I've spent many CPU seconds and hours of wall time on a computation. I suspect a lot of Pythonistas will feel the same way. > This can either be done by expecting Place or Pack explicitly > (i.e. having Place or Pack in the function signature) or by > explicitly specifying that you want to use TOSWidget's > implementation of Place or Pack. In Rust, a TOSWidget can be both a > Place and a Pack, Literally being both is not possible in Python at all if there are method collisions because of the way multiple inheritance works. Python can't invoke a trait in the function signature, because Python functions don't have typed arguments. It would be possible in MyPy if you implemented traits as classes, but that doesn't help you at compile time, you'd have to add checking at runtime. So making this possible would be a major change in Python semantics, creating a new kind of data namespace below the class level. You won't get that in Python without convincing a lot of people that they can't live without true traits in some form, and then that emphasizing the same aspects of traits that Rust does is the right balance. > and things expecting Place behaviour expect it explicitly and get > it explicitly, whereas things expecting Pack behaviour also expect > it explicitly and get it explicitly. If you have to be explicit, I don't see how this has a big advantage over having Place and Pack trait components as instance attributes, and a metaclass that automatically promotes trait methods to instance methods when there is no name collision. > I might've gone too far on this one, sorry. Apology accepted. No offense taken. Steve

2 2