Hey all,
What are thoughts about implementing an object-compare function in the
unittest package? (Compare two objects recursively, attribute by
attribute.) This seems like a common use case in many testing scenarios,
and there are many 3rd party solutions solving the same problem. (Maybe we
can promote standardization by implementing it in the standard library?)
Apologies ahead of time if this idea has already been proposed; I was not
able to find similar posts in the archive.
Best,
-Henry
Hello,
The idea here is to use the same operator already used to type-hint the
return of functions `->` to define the return of a Callable.
* Current syntax:
Callable[[ArgumentList], ReturnType]
eg.
def x(method: Callable[[int, dict], None]) -> None:
pass
* Proposed syntax:
Callable[ArgumentList] -> ReturnType
eg.
def x(method: Callable[int, dict] -> None) -> None:
pass
Do you think this is more intuitive? Is it viable?
Kind regards,
Thiago
Thank you Rob, for your contribution. I find it very helpful. You've raised
important implementation questions.
Regarding the no-iterations case. Suppose we have
for i in items:
loop_body
case:
zero:
print('no items to process')
In this case, the Python compiler knows that the post-iteration block
contains a 'zero:' statement. And in other cases it knows that it does not.
So why not have the compiler produce bytecode that records this fact. (This
could be done by providing an additional bytecode, or if there's room
adding a flag to an existing bytecode.)
Depending on how we choose to implement, if we wish we could keep both the
bytecode and interpreter code exactly the same, for the currently allowed
post-iteration blocks.
I apologize that it wasn't clear to you from my example code, that for
example we allow any identifier as the 'label argument' in a break
statement. As in
break error_1
break error_2
break time_out
The compiler can and I think should ensure that all labels in the
post-iteration block occur in the iteration block. And a code analysis code
aka linter should warn about unreachable code. This unreachable code
should, of course, not generate any bytecode.
Thank you again for your comment. I hope this helps.
--
Jonathan
Hello!
According to PEP8, there should be 2 blank lines before the function definition. But,
I think in case of defining new typing.TypeVar, that will be used only in one function
definition, it would be better to put it right before the function definition, i.e. with no
blank lines. So I suggest to update PEP8, so it'd recommend skipping blank lines in this
case.
Thank you for your attention and sorry for my bad English.
Sincerely yours,
Nikita Serba.
Hello, I'd like to bring to your attention https://bugs.python.org/issue41383. The core idea here is per Elizaveta Shashkova:
I would like to have a lazy repr evaluation for the objects! Sometimes users have many really large objects, and when debugger is trying to show them in Variables View (=show their string representation) it can takes a lot of time. We do some tricks, but they not always work. It would be really-really cool to have parameter in repr, which defines max number of symbols we want to evaluate during repr for this object.
Maybe repr is not the best here, because that should be interpreter meaningful, but instead the __str__ method that's better for this. Maybe we could pass in an optional limit argument to these methods, so that the user can decide what to print depending on how many characters he has left?
Any takes, better ideas how we could help this problem?
Thanks,
Bernat
ELSE-clause in FOR and WHILE has unclear syntax. I suggest new clause instead:
if COND:
...
[elif COND:
...]
[else:
...]
This IF-clause like must be immediately after FOR- or WHILE-cycle (only comment allowed between). It looks like a regular IF, but COND is special.
COND may be "break", "pass" or "finally". "if break:" - if used break-operator to exit cycle. "if pass:" - cycle executed 0 times. "if finally:" - cycle executed 0 or more times ("pass-case" is included in "finally-case"). For compatibility only "else:" means "if finally:".
It's compatible enhancement. No new keyword. There can be no combination "break", "pass" or "finally" after "if"/"elif:" in current version.
-----------------
This idea was suggested as enhancement issue (https://bugs.python.org/issue41272).
Eric V. Smith (eric.smith) advised me to put this on the python-ideas mailing list (https://bugs.python.org/issue41272#msg373486).
Right now in str.format(), we have !s, !r, and !a to allow us to call str(), repr(), and ascii() respectively on the given expression.
I'm proposing that we add a !p conversion to have pprint.pformat() be called to convert the given expression to a 'pretty' string.
Calling
```
print(f"My dict: {d!p}")
```
is a lot more concise than:
```
import pprint
print(f"My dict: {pprint.pformat(d)}")
```
We may even be able to have a static attribute stored to change the various default kwargs of pprint.pformat().
Evening,
I have run into a situations when an enum member becomes obsolete.
However, I'd like to avoid removing the member from the enum to give time for users to catch up to a libraries core types.
I've seen Java has a @Deprecated annotation for their Enums.
Python enum's can also set an _ignore_ attribute which will avoid instantiating a member in the list.
I've implemented a similar approach to solve deprecating enum members.
You can check it out here: https://github.com/foxyblue/cpython/pull/1
Let me first say that the code and discussion, as per title, is also about
possible performance improvements to the dict base type.
TL;DR I implemented a frozendict using CPython 3.9 code. It seems that an
immutable dict *could* be faster than dict in some cases. Furthermore, some
optimization could be applied to dict too.
Long explaining:
Since now I have some time, I decided to experiment a little with the
creation of an immutable dict in Python.
Unluckily, I started this experiment many months ago, so the CPython code I
used is old. Maybe some or all of my considerations are outdated.
Initially, I wrote a quick and dirty implementation:
https://github.com/Marco-Sulla/cpython/commit/fde4e6d236b19636063f8afedea8c…
The code was very simple, and the performance was identical to dict. So in
theory, adding a frozendict to CPython is not hard. But there's more.
After the first implementation, I started to try to improve the performance
of frozendict. The result of the improvements are here:
https://github.com/Marco-Sulla/cpython/blob/master/frozendict/test/bench.txt
For benchmarks, I used simply timeit, with autorange and repeat and, as
suggested in the module documentation, I got the minimum of the results.
Here is the code:
https://github.com/Marco-Sulla/cpython/blob/master/frozendict/test/bench.py
I have not tested with an optimized build, since optimization data is
collected using the unit tests, and I didn't write tests for frozendict in
the official CPython test framework.
The tests and benchmarks were done using CPython 3.9a0. CPU and other pc
resources were not restricted using pyperf or similar tools, to see the
"real" speed. CPython was compiled using gcc and g++.
In benchmarks, I compared methods and operators using dict and frozendict.
The benchmark uses dicts with all integers and dicts with all strings.
Furthermore, I tested dicts with size 8 (the most optimized size in
CPython) and 1000 elements (maybe too much, but I wanted to see how they
perform with a high number of elements).
Every benchmark has a line in the output. The Name describes the
benchmarked method, operator or code snippet. Size is the size of the dict,
8 or 1000. Keys are the keys type, str or int. Type is the dictionary type,
dict or frozendict.
In Name, the "o" represents the object itself (dict or frozendict). "d", in
benchmark with dict, is "o"; in benchmarks with frozendict is an equivalent
instance of type dict.
Some consideration:
1. frozendict is very fast, as expected, at copying. But it's also faster
at creation, using a (not frozen) dict, kwargs or a sequence2. Speedups
range from 20% to 45%.
2. frozendict is also a bit faster when you iterate over it, especially
over values, where is ~15% faster
3. hash seems really fast, but this is because it's cached the first time
hash() is invoked
4. where frozendict is slower is when you unpickle it and with fromkeys and
repr. This is because I wrote a very naif implementation of these methods,
without optimizing them. The other methods have a comparable speed.
Here is the code:
https://github.com/Marco-Sulla/cpython
Here is the diff between the CPython code and my commits:
https://github.com/python/cpython/compare/master...Marco-Sulla:master
About code
I coded the implementation doing a simple copy/paste of the existing dict
functions, modifying their code and renaming them. This way I'm sure dict
continues to work as before, and I can compare the speed gain.
Some of the optimization I adopted can be implemented in `dict` too. For
example, instead of creating an empty dict and resizing it, I create it
with the "maximum" size and I fill it. It seems to work, even if I did not
explore the possibility that a mutable object can change while a frozendict
creation is in progress.
Some problems with optimizing dict and maintaining a frozendict:
1. duplication of code. To gain a significant boost, I had to copy and
paste a lot of code. Functions can be remerged again, but maybe the speedup
will be reduced.
2. split vs combined dicts. As I wrote, split dicts seem to be faster in
reading than combined dicts. For example, iterating over values is faster
with a split dict, as expected.
But writing was not tested; furthermore, some of the optimizations can be
adopted for dicts too, so the convenience of a split table can be lowered.
dict continues to maintain both split and combined tables, so this could be
not a problem. But the code could be less and more fast if only a table
layout is supported
3. the CPython code I used is old, so some of the improvements I adopted
could be already implemented
About frozendict
Apart the considerations done in the [PEP 416](
https://www.python.org/dev/peps/pep-0416/), that was rejected since there
was little gain from its implementation, I think that frozendict can be
useful as a substitute of MappingProxyType, that is really slow.
MappingProxyType is not much used, but it's present in critical parts of
CPython code, for example in _sre. We have to see if a mapping proxy type
*can* be substituted with an immutable map in some critical part of CPython
code.
Furthermore, frozendicts could be used for implementing "immutable" classes
and modules, and can be used as a faster dict if its content does not
change.
Is there ANY chance of setting the default reply-to to the list? I know
everyone else thinks that's a bid idea, but I doubt this was actually
intended just for me.
If it was, I apologize for bringing it back on the list.
On Sun, Jul 12, 2020 at 10:12 PM Inada Naoki <songofacandy(a)gmail.com> wrote:
> On Mon, Jul 13, 2020 at 1:51 PM Christopher Barker <pythonchb(a)gmail.com>
> wrote:
> As I and Eric already said several times, we can do it in C already --
> itertools.islice.
>
> ...
itertools.islice is implemented in C too. No need to call next() from
> Python stack.
>
Ahh -- thanks! I had no idea islice() was in C -- yes, then, there probably
is little to gain from a C implementation.
I'd still be interested in the performance comparison, but not enough to
try to write a direct-access-to-the-internals-of-dict C version myself :-)
I still like the idea of indexing the order-preserving dict, but we do seem
to not have any compelling use cases.
Particularly if someone does get a reservoir sampling implementation in the
stdlib. (see another thread for that)
-CHB
--
Christopher Barker, PhD
Python Language Consulting
- Teaching
- Scientific Software Development
- Desktop GUI and Web Development
- wxPython, numpy, scipy, Cython
