Mailman 3 June 2019 - Python-ideas

Comparison operator support (>= and <=) for type
by eminbugrasaral＠me.com June 20, 2019

June 20, 2019

Let's assume you have this model: ``` class Base: pass class A(Base): pass class B(Base): pass class C(A): pass ``` While we can do `A == B`, or `B == C` or `B == B`, I would expect to be able to compare like this as well: `A >= B (if B is subclass or itself of A)`, or `B <= C (if B is a subclass or itself of C)` Because, since == means equality check. With the same logic, a class wraps another class is actually greater than this class from its type.

23 37

Re: [Python-ideas] Operator as first class citizens -- like in scala -- or yet another new operator?
by Yanghao Hua June 19, 2019

June 19, 2019

On Mon, Jun 3, 2019 at 8:01 PM Andrew Barnert <abarnert(a)yahoo.com> wrote: > > I think it was a mistake to even mention assignment here (much less arbitrary operators). Most of the resistance you’re facing is because of that, and I think you’re missing a fundamental reason behind that resistance. > > Overloading variable assignment makes no sense in Python. In a language with lvalue variable semantics like Scala or C++, a variable is a thing with type, address, and identity, … [View More]and values live in variables. So assignment affects a variable, and variables have types that can overload how that works. In a language with name-binding variable semantics like Python, it’s the value that has type, address, and identity, and it lives somewhere unspecified on the heap; a variable is nothing more than a name that can be bound to a value in a namespace. So assignment doesn’t affect variables, and, even if it did, they don’t have types that could overload how that works. Overloading assignment isn’t impossible just because, it’s impossible because it makes no sense. > > But the fact that in Scala, and Verilog, you send values to signals with an assignment-like operator on an lvalue for the signal doesn’t mean it actually is assignment. What you’re doing is an operation on the signal object itself, not on the signal variable. So neither = nor := would have been a good fit in the first place, and the fact that the variable is just a name rather than an lvalue isn’t a limitation at all, and there’s really no good reason not to have a “send a value to a signal” operator other than the (already very high) bar to new operators. Acknowledged. I do love the dynamic nature of Python, and as many of you could have experienced that you might be developing a type system sooner or later. And newer python with type hint makes it even easier to do that. That being said, I fully agree that we should not mess up with the = or := assignment operators, that was taken as an example. And I fully agree with you I probably should not even use the term "assignment operators", what I needed is an operator that does not collide with all existing number/matrix operators. > If you just explained the operation without reference to assignment, then the questions would be about why << isn’t a good enough fit, how widely the new operator would be used, whether there might be uses in other domains (Erlang-style channels?), etc. At that point, PEP 465 is a much better parallel. Because << is really one of the operator I need for signals ... it is even very straight forward to be translated into actual hardware implementation. I looked at every single operator I could have re-used ... the most promising one being @= operation, but then I realized I really want my signal matrix to use that operator for the sake of readability, it is really far more beautiful to read. > As a side note, if you haven’t already looked at other expression tree libraries like SymPy and SQLAnywhere and their parallels in languages like Scala and C++, it’s worth doing so. For many things (like most desirable uses of SymPy) the lack of variable assignment is no problem; for a few things (like implicit futures, or replacing namespaces with dataflow), it’s an insurmountable problem. I think your problem is more like the former. I do have experience with ORMs (Object-Relational-Mapper) like in Django and others, and just looked at SymPy and it is a great project. And indeed I agree with you that this problem is actually easily fixable ... I can instead use: signal.next = something, or use signal.assign(something), which is doable and many people are doing it, and you have many different ways of doing so. And then the choice of "signal.next/assign" becomes arbitrary ... e.g. you can use signal._next, signal._assign so on and so forth. But all those for me are actually implementation details and it is killing me that there is no way to hide it. Every single other major HDLs in the world uses an operator to do so, (vhdl/verilog <=, chisel :=). And this is not actually forcing a HDL way of doing things on Python, it actually represented a common category of problems in Python: there is no operator support to mean: drop a value on me (where "me" is already defined somewhere else, it has a type and identity). In HDL implementation alone I can see already two significant use cases: * signal assignment: signal <== signal << 2 (vs. signal.next = signal << 2 or signal.assign(signal << 2)), and it allows too something like: signal_a <== signal_b <== signal_c (vs signal_a.next = signal_b, signal_b.next = signal_c). * module integration/connection and pipe line construction: module_a <== module_b <== module_c (vs. module_a.connect(module_b); module_b.connect(module_c)) And yet again let me acknowledge: I understood all these problem can be solved by using a function call instead, but it is not as pretty as an operator. This is the whole point of PEP 465 ... make it looks good, as readability matters. So my question: does this looking good to python community? (You don't care much on hardware design is probably another thing), I think it looks much better, and around <== a well defined concept can be established in a particular domain/project. > Also, I can imagine a reason why << isn’t acceptable: it’s probably not that uncommon to need actual bitshift operators. But if you could build something that uses << anyway, and works perfectly in today’s Python for examples that don’t need shift, and then show that off—and show an example where not having bitshift is a problem—you’d be in an even better position. (See the PEP 465 section on numerical libraries that used * for matmul. They clearly work, and are useful for many things, but not having * for elementwise multiplication significantly limits a wide range of useful programs, which is why @ was needed for numpy.) Yep, that is my plan too. But instead of using <<, I have already modified CPython anyway to support <== and ==> operators and release a fully open sourced, powered-by-python HDL simulator with all major features you'd expect in hardware design (including dumping waveforms to VCD files, automatically building the hierarchy of module instance and signal instance names etc.). Code is already working but I want to make sure a few corner cases are well covered and it will be released alone with a PEP. I understand that after the assignment expression (:=) PEP572 people are already not happy with anything that could be slightly related to assignment overloading, I actually don't understand why people object PEP572 so hard, its practical use is clear and it makes code shorter and easier to read. Sometimes it seems the obsession of people in one idea makes them to even deny the existence of problems ... problems that having been solved well in other programming languages, and is well understood by majority of the programmers. [View Less]

19 60

venv.EnvBuilder environmental variable hooks
by Caleb Donovick June 18, 2019

June 18, 2019

Currently EnvBuilder allows for a number of customizations of virtual environments, changing which python is used, automatically installing libraries, etc... However, it does not allow for any modifications of the activate script itself (unless one wants to rewrite it completely). However, a fairly common requirement for python packages is to edit some environmental variable e.g., DJANGO_SETTINGS_MODULE=<settings>. Which leaves developers with two options set the variables globally or … [View More]

2 1

unittest block
by James Lu June 18, 2019

June 18, 2019

# regular code unittest: # test code goes here This code would be run when the runtime -unittest flag is passed. It would function as test usages and example client code. The code within the block would remain in .pyc, unless the -O switch is passed. Write """""" or ## above a unittest to include it in generated documentation.

2 1

Fwd: unittest block
by Jonathan Fine June 18, 2019

June 18, 2019

Hi James Something similar can already be done, say via from somewhere import unittest @unittest def f(): # test code goes here As it's your idea, I suggest you create and put up an implementation on PyPi, and post again to this list when you have some real-world examples of the use of this feature. -- Jonathan

1 0

Operator as first class citizens -- like in scala -- or yet another new operator?
by Yanghao Hua June 18, 2019

June 18, 2019

I see python3.5 accepted PEP465 adding a new infix operator for matrix (@, @=), which made matrix formula's much less painful to read in Python. There are still more use cases like this in other areas. While looking at Chisel (a hardware construction language build on top of scala), where you can create arbitrary new operators e.g. := used for specific purposes, I realize there is no way to overload the behavior of the assignment operator in python unless new operators are introduced. In the … [View More]

18 95

Re: Comparison operator support (>= and <=) for type
by Stéphane Wirtel June 18, 2019

June 18, 2019

;-)

1 0

Add «iterate non-blocking» wrapper to prevent blocking loop too long
by Nikita Melentev June 17, 2019

June 17, 2019

At work we faced a problem of long running python code. Our case was a short task, but huge count of iterations. Something like: for x in data_list: # do 1ms non-io pure python task So we block loop for more than 100ms, or even 1000ms. The first naive solution was "move this to thread" so python will switch thread context and asyncio loop will not be blocked, but this raised two problems: * If we do asyncio things in our task (create `Future` in our case), then we need to pass loop … [View More]

9 28

Adding context manager interface to contextvars.Token
by Christoph Groth June 10, 2019

June 10, 2019

It seems that the following idea has not been raised here before. If it has been, I'd be grateful for a pointer to the relevant discussion. The idea is: how about adding context manager interface (__enter__ and __exit__ methods) to the contextvars.Token type? This would allow code like this: with var.set(1): # var.get() == 1 that would be equivalent to the following more verbose snippet (taken from PEP 567): token = var.set(1) try: # var.get() == 1 finally: … [View More]

3 11

Improve of threading.Timer
by killerrex＠gmail.com June 9, 2019

June 9, 2019

Hi, I recently needed to run a function every half a second and the first option was threading.Timer, but it just tick once. Based on https://stackoverflow.com/a/48741004 I derived a class from Timer to run endlessly, but there are two things I do not like: - The function, args and kwargs attributes of Timer are just the same as the _target, _args and _kwargs from Thread. - It feels better that Timer can "tick" a given number of times. So I changed Timer to be more inline with the Thread … [View More]class and make it possible to run the function more than once: class Timer(Thread): FOREVER = Ellipsis def __init__(self, interval, function, args=None, kwargs=None, name=None, *, ticks=1, daemon=None): # Keep compatibility with the old Timer, where the default # args and kwargs where mutable. if args is None: args = [] if kwargs is None: kwargs = {} Thread.__init__(self, target=function, args=args, kwargs=kwargs, name=name, daemon=daemon) self.interval = interval self.ticks = ticks self.finished = Event() def cancel(self): """Stop the timer if it hasn't finished yet.""" self.finished.set() def run(self): try: while not self.finished.wait(self.interval): if self._target and not self.finished.is_set(): self._target(*self._args, **self._kwargs) if self.ticks is self.FOREVER: continue self.ticks -= 1 if self.ticks <= 0: break finally: # Remove the reference to the function for the same # reason it is done in Thread.run del self._target # However, for backwards compatibility do not remove # _args and _kwargs as is done in Thread.run # del self._args, self._kwargs self.finished.set() def join(self, timeout=None): # If the timer was set to repeat endlessly # it will never join if it is not cancelled before. if self.ticks is self.FOREVER: self.cancel() super().join(timeout=timeout) @property def function(self): return getattr(self, '_target', None) @function.setter def function(self, value): self._target = value @property def args(self): return self._args @args.setter def args(self, value): self._args = value @property def kwargs(self): return self._kwargs @kwargs.setter def kwargs(self, value): self._kwargs = value The default option for ticks is to run once, just like the current Timer. With this version it is possible to do things like: >>> t = Timer(10.0, f, ticks=3) >>> t.start() # Will run f three times, with 10 seconds between each call. >>> t = Timer(10.0, f, ticks=Timer.FOREVER, daemon=True) >>> t.start() # Will run f every 10 seconds until the end of the program # It also stops the thread if the program is interrupted. >>> t = Timer(10.0, f, ticks=..., daemon=True) >>> t.start() # Same as above, but I like the usage of Ellipsis here :) Other option is to leave Timer as it is and create a new class, so it is not necessary to keep the compatibility with the function, args and kwargs attributes: class Ticker(Thread): FOREVER = Ellipsis def __init__(self, interval, target=None, args=(), kwargs=None, group=None, name=None, *, ticks=1, daemon=None): super().__init__(group=group, target=target, args=args, kwargs=kwargs, name=name, daemon=daemon) self.interval = interval self.ticks = ticks self.finished = Event() def cancel(self): """Stop the timer if it hasn't finished yet.""" self.finished.set() def evaluate(self): if self._target and not self.finished.is_set(): self._target(*self._args, **self._kwargs) def run(self): try: while not self.finished.wait(self.interval): self.evaluate() if self.ticks is self.FOREVER: continue self.ticks -= 1 if self.ticks <= 0: break finally: # Remove the reference to the function for the same # reason it is done in Thread.run del self._target, self._args, self._kwargs self.finished.set() def join(self, timeout=None): # If the timer was set to repeat endlessly # it will never join if it is not cancelled before. if self.ticks is self.FOREVER: self.cancel() super().join(timeout=timeout) Do you think it is useful to try to clean up and add this code to threading or it is just something too specific to consider changing the standard library? Regards, Andrés [View Less]

3 2

python-ideas is now running on Mailman 3
by Brett Cannon June 6, 2019

June 6, 2019

https://mail.python.org/mailman3/lists/python-ideas.python.org/

7 9

Python-ideas] Re: Operator as first class citizens -- like in scala -- or yet another new operator?
by Angus Hollands June 5, 2019

June 5, 2019

Regarding this operator proposal, I also think we can still do this comfortably with the existing Python syntax. The problem that I have is I don't really understand what the *problem* is that you're trying to solve. It would be helpful if you had some explanation on that front. I see there was some discussion, but it's now buried in the email chain. Here's some example "custom" assignments. Note that you could substitute the Ellipsis used here for any unique object, e.g. one that you could … [View More]

1 0

zipfile refactor and AES
by Daniel Hillier June 4, 2019

June 4, 2019

Hi, I've written a package that can read and write zip files encrypted with Winzip's AES encryption scheme (https://github.com/danifus/pyzipper/). It is based on Python's zipfile module which I refactored to facilitate adding the AES code as subclasses of the classes defined in the zipfile module. I would like to explore integrating some of the refactoring effort into Python if they are wanted. While the AES specfic code probably isn't suitable for integration due to its dependence on a … [View More]crypto package, my hope is that the refactor of the zipfile project may be beneficial and, in particular, help development on other features of the zip spec. I'm happy to rework the changes for inclusion in Python. The general goals of the refactor were: - Keep the AES implementation changes in a separate file so changes to zipfile.py could potentially be merged into Python (the cryptically named "minimal_pep8" branch contains minimal cosmetic changes to the code unless the lines around it were changed. The master branch contains all the changes in the minimal_pep8 plus pep8 and a few other changes). - Add hooks for extending the way zipfile works to enable the addition of AES encryption without having to duplicate most of the zipfile module. This included adding hooks to: - Select and call encrypt and decrypt methods. - Read and write new "extra" data records in the central file directory and local header records. - Provide a mechanism to substitute ZipInfo, ZipExtFile and ZipWriteFile classes used in a subclass of ZipFile to ease use of subclassed ZipInfo, ZipExtFile or ZipWriteFile. This avoids having to rewrite large parts of the zipfile module if we only want to change the behaviour of a small part of one of those classes. - Contain all code that reads the header, contents and tail of a file in the archive to within ZipExtFile. Previously reading the header and some other things were done in the ZipFile class before handing the rest of the processing to ZipExtFile. - Contain all code that writes the header, contents and tail of a file in the archive to within ZipWriteFile. Previously reading the header and some other things were done in the ZipFile class before handing the rest of the processing to ZipExtFile. - Move generation of local file header and central directory record content to the ZipInfo class to be alongside the data that it is packing. - Add comments to provide context from the zip spec. Replace explicit numbers to variables with explanatory names or adding comments. Concerns: - The change set is not small. I've attempted to keep each commit in the minimal_pep8 branch focused on a single change to simplify review and kept the unrelated changes, like pep8, to a minimum. I'm happy to put in the work to get these changes into a patch for the Python project, if it is deemed useful. - This could move the internals of zipfile towards a public API (maybe it would be?) which brings additional complexity in managing future changes. - Are the hooks inline with Python's coding style? Is there a different approach to extensibility preferred within the Python project? Let me know any improvements, suggestions or concerns to this refactoring approach you may have. Happy for feedback even if it isn't about integrating the code into Python. Thanks, Dan [View Less]

4 4