Mailman 3 November 2018 - Python-ideas

Adding a thin wrapper class around the functions in stdlib.heapq
by bunslow Feb. 18, 2022

Feb. 18, 2022

Nothing so bombastic this time. The heapq functions are basically all named "heapsomething", and basically all take a "heap" for their first argument, with supplementary args coming after. It's a textbook example of the (hypothetical) Object Oriented Manifesto™ where defining a class increases type safety and programmers' conceptual clarity. There're practically no drawbacks, and the code to be added would be very simple. Updating the tests and docs would probably be harder. In pure Python, … [View More]

17 32

Make fnmatch.filter accept a tuple of patterns
by Andre Delfino March 26, 2021

March 26, 2021

Frequently, while globbing, one needs to work with multiple extensions. I’d like to propose for fnmatch.filter to handle a tuple of patterns (while preserving the single str argument functionality, alas str.endswith), as a first step for glob.i?glob to accept multiple patterns as well. Here is the implementation I came up with: https://github.com/python/cpython/compare/master...andresdelfino:fnmatch-mu… If this is deemed reasonable, I’ll write tests and documentation updates. Any opinion?

9 15

Dataclasses, keyword args, and inheritance
by George Leslie-Waksman March 12, 2021

March 12, 2021

The proposed implementation of dataclasses prevents defining fields with defaults before fields without defaults. This can create limitations on logical grouping of fields and on inheritance. Take, for example, the case: @dataclass class Foo: some_default: dict = field(default_factory=dict) @dataclass class Bar(Foo): other_field: int this results in the error: 5 @dataclass ----> 6 class Bar(Foo): 7 other_field: int 8 ~/.pyenv/versions/3.6.2/envs/… [View More]clover_pipeline/lib/python3.6/site-packages/dataclasses.py in dataclass(_cls, init, repr, eq, order, hash, frozen) 751 752 # We're called as @dataclass, with a class. --> 753 return wrap(_cls) 754 755 ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in wrap(cls) 743 744 def wrap(cls): --> 745 return _process_class(cls, repr, eq, order, hash, init, frozen) 746 747 # See if we're being called as @dataclass or @dataclass(). ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _process_class(cls, repr, eq, order, hash, init, frozen) 675 # in __init__. Use "self" if possible. 676 '__dataclass_self__' if 'self' in fields --> 677 else 'self', 678 )) 679 if repr: ~/.pyenv/versions/3.6.2/envs/clover_pipeline/lib/python3.6/site-packages/dataclasses.py in _init_fn(fields, frozen, has_post_init, self_name) 422 seen_default = True 423 elif seen_default: --> 424 raise TypeError(f'non-default argument {f.name!r} ' 425 'follows default argument') 426 TypeError: non-default argument 'other_field' follows default argument I understand that this is a limitation of positional arguments because the effective __init__ signature is: def __init__(self, some_default: dict = <something>, other_field: int): However, keyword only arguments allow an entirely reasonable solution to this problem: def __init__(self, *, some_default: dict = <something>, other_field: int): And have the added benefit of making the fields in the __init__ call entirely explicit. So, I propose the addition of a keyword_only flag to the @dataclass decorator that renders the __init__ method using keyword only arguments: @dataclass(keyword_only=True) class Bar(Foo): other_field: int --George Leslie-Waksman [View Less]

12 35

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Add the imath module
by Serhiy Storchaka March 22, 2020

March 22, 2020

What are your thoughts about adding a new imath module for integer mathematics? It could contain the following functions: * factorial(n) Is just moved from the math module, but non-integer types are rejected. Currently math.factorial() accepts also integer floats like 3.0. It looks to me, the rationale was that at the time when math.factorial() was added, all function in the math module worked with floats. But now we can revise this decision. * gcd(n, m) Is just moved from the math … [View More]

14 38

Re: [Python-ideas] Allow star unpacking within an slice expression
by Neil Girdhar March 7, 2020

March 7, 2020

I didn't think of this when we were discussing 448. I ran into this today, so I agree with you that it would be nice to have this. Best, Neil On Monday, December 4, 2017 at 1:02:09 AM UTC-5, Eric Wieser wrote: > > Hi, > > I've been thinking about the * unpacking operator while writing some numpy > code. PEP 448 allows the following: > > values = 1, *some_tuple, 2 > object[(1, *some_tuple, 2)] > > It seems reasonable to me that it should be extended to … [View More]

2 2

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Please consider adding of functions file system operations to pathlib
by George Fischhof Nov. 16, 2019

Nov. 16, 2019

Good day all, as a continuation of thread "OS related file operations (copy, move, delete, rename...) should be placed into one module" https://mail.python.org/pipermail/python-ideas/2017-January/044217.html please consider making pathlib to a central file system module with putting file operations (copy, move, delete, rmtree etc) into pathlib. BR, George

3 3

Start argument for itertools.accumulate() [Was: Proposal: A Reduce-Map Comprehension and a "last" builtin]
by Raymond Hettinger Oct. 23, 2019

Oct. 23, 2019

> On Friday, April 6, 2018 at 8:14:30 AM UTC-7, Guido van Rossum wrote: > On Fri, Apr 6, 2018 at 7:47 AM, Peter O'Connor <peter.ed...(a)gmail.com> wrote: >> So some more humble proposals would be: >> >> 1) An initializer to itertools.accumulate >> functools.reduce already has an initializer, I can't see any controversy to adding an initializer to itertools.accumulate > > See if that's accepted in the bug tracker. It did come-up once but was closed … [View More]for a number reasons including lack of use cases. However, Peter's signal processing example does sound interesting, so we could re-open the discussion. For those who want to think through the pluses and minuses, I've put together a Q&A as food for thought (see below). Everybody's design instincts are different -- I'm curious what you all think think about the proposal. Raymond --------------------------------------------- Q. Can it be done? A. Yes, it wouldn't be hard. _sentinel = object() def accumulate(iterable, func=operator.add, start=_sentinel): it = iter(iterable) if start is _sentinel: try: total = next(it) except StopIteration: return else: total = start yield total for element in it: total = func(total, element) yield total Q. Do other languages do it? A. Numpy, no. R, no. APL, no. Mathematica, no. Haskell, yes. * http://docs.scipy.org/doc/numpy/reference/generated/numpy.ufunc.accumulate.… * https://stat.ethz.ch/R-manual/R-devel/library/base/html/cumsum.html * http://microapl.com/apl/apl_concepts_chapter5.html \+ 1 2 3 4 5 1 3 6 10 15 * https://reference.wolfram.com/language/ref/Accumulate.html * https://www.haskell.org/hoogle/?hoogle=mapAccumL Q. How much work for a person to do it currently? A. Almost zero effort to write a simple helper function: myaccum = lambda it, func, start: accumulate(chain([start], it), func) Q. How common is the need? A. Rare. Q. Which would be better, a simple for-loop or a customized itertool? A. The itertool is shorter but more opaque (especially with respect to the argument order for the function call): result = [start] for x in iterable: y = func(result[-1], x) result.append(y) versus: result = list(accumulate(iterable, func, start=start)) Q. How readable is the proposed code? A. Look at the following code and ask yourself what it does: accumulate(range(4, 6), operator.mul, start=6) Now test your understanding: How many values are emitted? What is the first value emitted? Are the two sixes related? What is this code trying to accomplish? Q. Are there potential surprises or oddities? A. Is it readily apparent which of assertions will succeed? a1 = sum(range(10)) a2 = sum(range(10), 0) assert a1 == a2 a3 = functools.reduce(operator.add, range(10)) a4 = functools.reduce(operator.add, range(10), 0) assert a3 == a4 a4 = list(accumulate(range(10), operator.add)) a5 = list(accumulate(range(10), operator.add, start=0)) assert a5 == a6 Q. What did the Python 3.0 Whatsnew document have to say about reduce()? A. "Removed reduce(). Use functools.reduce() if you really need it; however, 99 percent of the time an explicit for loop is more readable." Q. What would this look like in real code? A. We have almost no real-world examples, but here is one from a StackExchange post: def wsieve(): # wheel-sieve, by Will Ness. ideone.com/mqO25A->0hIE89 wh11 = [ 2,4,2,4,6,2,6,4,2,4,6,6, 2,6,4,2,6,4,6,8,4,2,4,2, 4,8,6,4,6,2,4,6,2,6,6,4, 2,4,6,2,6,4,2,4,2,10,2,10] cs = accumulate(cycle(wh11), start=11) yield( next( cs)) # cf. ideone.com/WFv4f ps = wsieve() # codereview.stackexchange.com/q/92365/9064 p = next(ps) # 11 psq = p*p # 121 D = dict( zip( accumulate(wh11, start=0), count(0))) # start from sieve = {} for c in cs: if c in sieve: wheel = sieve.pop(c) for m in wheel: if not m in sieve: break sieve[m] = wheel # sieve[143] = wheel@187 elif c < psq: yield c else: # (c==psq) # map (p*) (roll wh from p) = roll (wh*p) from (p*p) x = [p*d for d in wh11] i = D[ (p-11) % 210] wheel = accumulate(cycle(x[i:] + x[:i]), start=psq) p = next(ps) ; psq = p*p next(wheel) ; m = next(wheel) sieve[m] = wheel [View Less]

17 49

support toml for pyproject support
by Jimmy Girardet July 29, 2019

July 29, 2019

Hi, I don't know if this was already debated but I don't know how to search in the whole archive of the list. For now the adoption of pyproject.toml file is more difficult because toml is not in the standard library. Each tool which wants to use pyproject.toml has to add a toml lib as a conditional or hard dependency. Since toml is now the standard configuration file format, it's strange the python does not support it in the stdlib lije it would have been strange to … [View More]

13 15

Re: [Python-ideas] Trigonometry in degrees
by Hugh Fisher May 8, 2019

May 8, 2019

> Date: Thu, 7 Jun 2018 12:33:29 +0000 > From: Robert Vanden Eynde <robertvandeneynde(a)hotmail.com> > To: python-ideas <python-ideas(a)python.org> > Subject: [Python-ideas] Trigonometry in degrees > Message-ID: > > > I suggest adding degrees version of the trigonometric functions in the math module. > > - Useful in Teaching and replacing calculators by python, importing something is seen by the young students much more easy than to define a … [View More]function. I agree that degrees are useful for teaching. They are also very useful for graphics programming, especially with my favourite OpenGL API. But I think that the use of radians in programming language APIs is more prevalent, so the initial advantage of easy learning will be outweighed by the long term inconvenience of adjusting to what everyone else is doing. Writing degrees(x) and radians(x) is a little inconvenient, but it does make it clear what units are being used. And even if your proposal is adopted, there is still going to be a lot of code around that uses the older math routines. With the current API it is a least safe to assume that angles are radians unless stated otherwise. > - Special values could be treated, aka when the angle is a multiple of 90, young students are often surprise to see that cos(pi/2) != 0 > > Testing for a special value Isn't very costly (x % 90 == 0) but it could be pointed out that there is a small overhead using the "degrees" equivalent of trig function because of the radians to degrees conversion And the special values testing. Not just young students :-) I agree with this, but I would prefer the check to be in the implementation of the existing functions as well. Any sin/cos very close to 0 becomes 0, any close to 1 becomes 1. > - Standard names will be chosen so that everyone will use the same name convention. I suggest adding a "d" like sind, cosd, tand, acosd, asind, atand, atan2d. Not "d". In the OpenGL 3D API, and many 3D languages/APIs since, appending "d" means "double precision". It's even sort of implied by the C math library which has sinf and friends for single precision. > > Creating a new package like 'from math.degrees import cos' however I would not recommend that because "cos" in the source code would mean to lookup the import to know if it's in degrees or radians (and that leads to very filthy bugs). Also "degrees" is already so the name would have to change the name of the package. Agree, not a good idea. -- cheers, Hugh Fisher [View Less]

16 29

f-string "debug" conversion
by Eric V. Smith May 3, 2019

May 3, 2019

This idea was proposed to me at the core sprints last month by Larry Hastings. I've discussed it with a few people, who seem generally positive about it, and we've tweaked it a little bit. I've spent some time implementing it, and I think it's doable. I thought I'd post it here for any additional feedback. Here’s the idea: for f-strings, we add a !d conversion operator, which is superficially similar to !s, !r, and !a. The meaning of !d is: produce the text of the expression (not its … [View More]

12 36

Return for assignment blocks
by Calvin Spealman March 20, 2019

March 20, 2019

I'd like to suggest what I think would be a simple addition to `def` and `class` blocks. I don't know if calling those "Assignment Blocks" is accurate, but I just mean to refer to block syntaxes that assign to a name. Anyway, I propose a combined return-def structure, and optionally also allowing a return-class version. Omitting the name would be allowable, as well. This would only apply to a `def` or `class` statement made as the last part of the function body, of course. def ignore_exc(… [View More]

14 33

Support localization of unicode descriptions
by Pander Feb. 25, 2019

Feb. 25, 2019

Dear all, Who might also be interested in setting up a project that supports localization for Unicode block description and character description. Translations are available from https://github.com/unicode-table/unicode-table-data/tree/master/loc <https://github.com/unicode-table/unicode-table-data/tree/master/loc> If possible, use a gettext approach similar to https://pypi.org/project/pycountry/ <https://pypi.org/project/pycountry/> Implementing this feature will allow users to … [View More]

5 8

Dart like multi line strings identation
by Marius Räsener Feb. 8, 2019

Feb. 8, 2019

Hey List, this is my very first approach to suggest a Python improvement I'd think worth discussing. At some point, maybe with Dart 2.0 or a little earlier, Dart is now supporting multiline strings with "proper" identation (tried, but I can't find the according docs at the moment. probably due to the rather large changes related to dart 2.0 and outdated docs.) What I have in mind is probably best described with an Example: print(""" I am a multiline String. """) the closing … [View More]

23 47

__len__() for map()
by Kale Kundert Dec. 13, 2018

Dec. 13, 2018

I just ran into the following behavior, and found it surprising: >>> len(map(float, [1,2,3])) TypeError: object of type 'map' has no len() I understand that map() could be given an infinite sequence and therefore might not always have a length. But in this case, it seems like map() should've known that its length was 3. I also understand that I can just call list() on the whole thing and get a list, but the nice thing about map() is that it doesn't copy data, so it's … [View More]

19 113

[Brainstorm] Testing with Documented ABCs
by Abe Dillon Dec. 8, 2018

Dec. 8, 2018

I've been pulling a lot of ideas from the recent discussion on design by contract (DBC), the elegance and drawbacks <https://bemusement.org/doctests-arent-code> of doctests <https://docs.python.org/3/library/doctest.html>, and the amazing talk <https://www.youtube.com/watch?v=MYucYon2-lk> given by Hillel Wayne at this year's PyCon entitled "Beyond Unit Tests: Taking your Tests to the Next Level". To recap a lot of previous discussions: - Documentation should tell you: A) … [View More]What a variable represents B) What kind of thing a variable is C) The acceptable values a variable can take - Typing and Tests can partially take the place of documentation by filling in B and C (respectively) and sometimes A can be inferred from decent naming and context. - Contracts can take the place of many tests (especially when combined with a library like hypothesis) - Contracts/assertions can provide "stable" documentation in the sense that it can't get out of sync with the code. - Attempts to implement contracts using standard Python syntax are verbose and noisy because they rely heavily on decorators that add a lot of repetitive preamble to the methods being decorated. They may also require a metaclass which restricts their use to code that doesn't already use a metaclass. - There was some discussion about the importance of "what a variable represents" which pointed to this article <http://pgbovine.net/python-unreadable.htm> by Philip J. Guo (author of the magnificent pythontutor.com). I believe Guo's usage of "in-the-small" and "in-the-large" are confusing because a well decoupled program shouldn't yield functions that know or care how they're being used in the grand machinations of your project. The examples he gives are of functions that could use a doc string and some type annotations, but don't actually say how they relate to the rest of the project. One thing that caught me about Hillel Wayne's talk was that some of his examples were close to needing practically no code. He starts with: def tail(lst: List[Any]) -> List[Any]: assert len(lst) > 0, "precondition" result = lst[1:] assert [lst[0]] + result == lst, "postcondition" return result He then re-writes the function using a contracts library: @require("lst must not be empty", lambda args: len(args.lst) > 0) @ensure("result is tail of lst", lambda args, result: [args.lst[0]] + result == args.lst) def tail(lst: List[Any]) -> List[Any]: return lst[1:] He then writes a unit test for the function: @given(lists(integers(), 1)) def test_tail(lst): tail(lst) What strikes me as interesting is that the test pretty-much doesn't need to be written. The 'given' statement should be redundant based on the type annotation and the precondition. Anyone who knows hypothesis, just imagine the @require is a hypothesis 'assume' call. Furthermore, hypothesis should be able to build strategies for more complex objects based on class invariants and attribute types: @invariant("no overdrafts", lambda self: self.balance >= 0) class Account: def __init__(self, number: int, balance: float = 0): super().__init__() self.number: int = number self.balance: float = balance A library like hypothesis should be able to generate valid account objects. Hypothesis also has stateful testing <https://hypothesis.readthedocs.io/en/1.4.1/stateful.html> but I think the implementation could use some work. As it is, you have inherit from a class that uses a metaclass AND you have to pollute your class's name-space with helper objects and methods. If we could figure out a cleaner syntax for defining invariants, preconditions, and postconditions we'd be half-way to automated testing UTOPIA! (ok, maybe I'm being a little over-zealous) I think there are two missing pieces to this testing problem: side-effect verification and failure verification. Failure verification should test that the expected exceptions get thrown when known bad data is passed in or when an object is put in a known illegal state. This should be doable by allowing Hypothesis to probe the bounds of unacceptable input data or states, though it might seem a bit silly because if you've already added a precondition, "x >= 0" to a function, then it obviously should raise a PreconditionViolated when passed any x < 0. It may be important, however; if for performance reasons, you need to disable invariant checking but you still want certain bad input to raise exceptions, or your system has two components that interact with slightly mis-matched invariants and you want to make sure the components handle the edge-condition correctly. You can think of Types from a set-theory perspective where the Integer type is conceptually the set of all integers, and invariants would specify a smaller subset than Typing alone, however if the set of all valid outputs of one component is not completely contained within the set of all valid inputs to another component, then there will be edge-cases resulting from the mismatch. In that sense, some of the invariant verification could be static-ish (as much as Python allows). Side-effect verification is usually done by mocking dependencies. You pass in a mock database connection and make sure my object sends and receives data as expected. As crazy as it sounds, this too can be almost completely automated away if all of the above tools are in place AND if Python gained support for Exception annotations. I wrote a Java (yuck) library at work that does this. I wan't to port it to Python and share it, but it basically enumerates a bunch of stuff: the "sources" and "destinations" of the system, how those relate to dependencies, how they relate to each other (if dependency X is unresponsive, I can't get sources A, B, or G and if I can't get source B, I can't write destination Y), the dependency failure modes (Exceptions raised, timeouts, unrecognized key, missing data, etc.), all the public methods of the class under test and what sources and destinations they use. Then I enumerate 'k' from 0 to some limit for the max number of simultaneous faults to test for: Then for each method that can have n >= k simultaneous faults I test all (n choose k) combinations of faults for that method against the desired behavior. I'm sure that explanation is as clear as mud. I will try to get a working Python example at some point to demonstrate. Finally, in the PyCon video; Hillel Wayne shows an example of testing that an "add" function is commutative. It seems that once you write that invariant, it might apply to many different functions. A similar invariant may be "reversibility" like: @given(text()) def test_reversable_codex(s): assert s == decode(encode(s)), "not reversible" That might be a common property that other functions share: @invariant(reversible(decode)) def encode(s: str) -> bytes: ... Having said all that, I wanted to brainstorm some possible solutions for implementing some or all of the above in Python without drowning you code in decorators. NOTE: Please don't get hung up on specific syntax suggestions! Try to see the forest through the trees! An example syntax could be: #Instead of this @require("lst must not be empty", lambda args: len(args.lst) > 0) @ensure("result is tail of lst", lambda args, result: [args.lst[0]] + result == args.lst) def tail(lst: List[Any]) -> List[Any]: return lst[1:] #Maybe this? non_empty = invariant("Must not be empty", lambda x: len(x) > 0) # can be re-used def tail(lst: List[Any] d"Description of what this param represents. {non_empty}") -> List[Any] d"Description of return value {lst == [lst[0]] + __result__}": """ Description of function """ return lst[1:] Python could build the full doc string like so: """ Description of function Args: lst: Description of what this param represents. Must not be empty. Returns: Description of return value. """ d-strings have some description followed by some terminator after which either invariant objects or [optionally strings] followed by an expression on the arguments and __return__? I'm sorry this is so half-baked. I don't really like the d-string concept and I'm pretty sure there are a million problems with it. I'll try to flesh out the side-effect verification concept more later along with all the other poorly explained stuff. I just wanted to get these thoughts out for discussion, but now it's super late and I have to go! [View Less]

10 24

[off-topic?] Unwinding generators
by Juancarlo Añez Nov. 30, 2018

Nov. 30, 2018

This is code from the Twisted library: https://github.com/twisted/twisted/blob/trunk/src/twisted/internet/defer.py… It "unwinds" a generator to yield a result before others. I don't have hard evidence, but my experience is that that kind of manipulation leaks resources, specially if exceptions escape from the final callback. Is there a bug in exception handling in the generator logic, or is unwinding just inherently wrong? How could the needs tried to solved with unwinding be handled with … [View More]

1 0

Make None a subclass of int [alternative to iNaN]
by Ken Hilton Nov. 27, 2018

Nov. 27, 2018

Hi all, Reading the iNaN discussion, most of the opposition seems to be that adding iNaN would add a new special value to integers and therefore add new complexity. I propose, instead, that we make None a subclass of int (or even a certain value of int) to represent iNaN. Therefore: >>> None + 1, None - 1, None * 2, None / 2, None // 2 (None, None, None, nan, None) # mathematical operations on NaN return NaN >>> None & 1, None | 1, None ^ 1 # I'm not … [View More]

3 2

hybrid regex engine: backtracking + Thompson NFA
by Celelibi Nov. 25, 2018

Nov. 25, 2018

Hello, I found this topic discussed on the python-dev ML back in 2010 [1]. I'm bringing it up 8 years later with a variation. In short: The article [2] highlight that backtracking-based regex engine (like SRE in python) have pathological cases that run in an exponential time of the input, while they could run in a linear time. Not mentioned by the article is that even quadratic time can be a problem with large inputs. Which happen pretty often when you're looking for delimited stuff like this … [View More]: re.match(r'.*?,(.*),,', "a,"*10000) Of course, there's a catch. Backreferences most-likely cannot be implemented in a guaranteed linear time, and some cases of look-behind assertions might prove difficult as well. But other features like alternatives (foo|bar) and repetitions (.*) are no problem. The general idea of the Thompson's algorithm is that the simulation of the NFA is basically in all the reachable states at the same time while parsing the input string character by character. Of course, for regex engines that use a VM (like Python's SRE) you can also make the execution of the equivalent byte-code be in several states at once. The 2010 discussion seems to be about having two separate engines and selecting the best one for a given regex. What I'm proposing here is to resort to backtracking only for the regex features that need it. Meaning that within a regex like r'(.*),.* .*,\1' the evaluation of the middle part would use the Thompson's algorithm, while the \1 could trigger the backtracking mechanism if the string doesn't match. What do you think about it? Has this already been discussed and rejected? Is it just a matter of showing the code? (_sre.c seems... non-trivial) Has this already been juged not worth the maintainance effort? AFAIK, there's not many hybrid regex engines out there. But one notable implementation is the third version of the Henry Spencer's regex engine [3]. Which he doesn't seem to have documented publicly, but postgres has done a pretty good job at reverse-engineering a high level view of it [4]. I'm still unsure how the backtracking of some parts of the regex interact with the multi-state evaluation of the other parts. But at least it exists and works, so it is feasible. Best regards, Celelibi [1] https://mail.python.org/pipermail/python-dev/2010-March/098354.html [2] https://swtch.com/~rsc/regexp/regexp1.html [3] https://github.com/garyhouston/hsrex [4] https://github.com/postgres/postgres/tree/master/src/backend/regex [View Less]

2 1

Enhancing range object string displays
by Steven D'Aprano Nov. 20, 2018

Nov. 20, 2018

On the bug tracker, there is a proposal to enhance range objects so that printing them will display a snapshot of the values included, including the end points. For example: print(range(10)) currently displays "range(10)". The proposal is for the __str__ method to instead return "<range object [0, 1, ..., 8, 9]>". https://bugs.python.org/issue35200 print(range(2, 200, 3)) would display <range object [2, 5, ..., 194, 197]> Note that the original proposal was for range … [View More]

6 6

Re: [Python-ideas] f-string "debug" conversion
by Rene Nejsum Nov. 17, 2018

Nov. 17, 2018

+1 for this, I would use it all the time for debugging and tracing programs breakpoints and IDE’s can be nice, but my code is filled with lines like: logger.debug(f“transaction_id={transaction_id}, state={state}, amount={amount}, etc={etc}”) So yeah, well +10 actually :-) /rene

1 0

Proposing additions to the standard library
by Jonathan Crall Nov. 15, 2018

Nov. 15, 2018

I'm interested in proposing several additions to the Python standard library, and I would like more information on the procedure for doing so. Are all additions done via a PEP? If not what is the procedure. If so, I've read that the first step was to email this board and get feedback. I have a library called `ubelt` that contains several tools that I think might be worthy of adding to the standard library. Here's my bullet point pitch: - Python is batteries included. Ubelt contains extra … [View More]batteries. function are extra batteries. - Most function in ubelt are fast. All 222 tests takes 7.33 seconds. - Ubelt has 100% test coverage (sans `# nocover` locations). - I'm only championing a subset of the functions in ubelt. There are certainly functions in there that do not belong in the standard library. - I have a Jupyter notebook that give a demo of some select functions (not necessarily the same as the ones proposed here): https://github.com/Erotemic/ubelt/blob/master/docs/notebooks/Ubelt%20Demo.i… - I do have documentation (mostly in docstrings) and in the docs folder, but I've been having trouble auto-updating read-the-docs. Here is the link anyway: https://ubelt.readthedocs.io/en/latest/ Here is a tentative list of interesting functions. Hopefully the names are descriptive (if not, see docstrings: https://github.com/Erotemic/ubelt) ub.cmd ub.compressuser ub.group_items ub.dict_hist ub.find_duplicates ub.AutoDict ub.import_module_from_path ub.import_module_from_name ub.modname_to_modpath, ub.modpath_to_modname ub.ProgIter ub.ensuredir ub.expandpath almost everything in util_list: allsame, argmax, argmin, argsort, argunique, chunks, flatten, iter_window, take, unique These functions might be worth modifying into dictionary methods: ub.dict_subset ub.dict_take ub.map_vals ub.map_keys ub.Timerit ub.Timer Because I built the library, I tend to like all the functions. Its difficult to decide if they are stdlib worthy, so there might be some false positives / negatives. I'm on the fence about: CacheStamp, Cacher, NoParam, argflag, argval, dzip, delete, hash_data, hash_file, memoize, memoize_method, NiceRepr, augpath, userhome, ensure_app_cache_dir, ensure_app_resource_dir, find_exe, find_path, get_app_cache_dir, get_app_resource_dir, platform_cache_dir, platform_resource_dir, CaptureStdout, codeblock, ensure_unicode, hzcat, indent, OrderedSet Its my hope that some of these are actually useful. Let me know any of the following: what you think, if there are any questions, if something else needs to be done, or what the next steps are. -- -Jon [View Less]

6 7

Relative Imports
by danish bluecheese Nov. 14, 2018

Nov. 14, 2018

Hi all, Im tired of not being able to make relative imports freely. Now trying to develop a module which enable any project to use relative imports once it is loaded. Anybody interested in? Best,

6 12

Range and slice syntax
by Nicholas Harrison Nov. 13, 2018

Nov. 13, 2018

I'm aware that syntax for ranges and slices has been discussed a good amount over the years, but I wanted to float an idea out there to see if it hasn't been considered before. It's not really original. Rather, it's a combination of a couple parts of Python, and I find it fascinatingly-consistent with the rest of the language. This will look similar to PEP 204, but there are some important differences and clarifications. (start:stop:step) Meet a range/slice object. Parentheses are required. (… [View More]Its syntax in this regard follows exactly the same rules as a generator expression.) I say both range and slice because it can be used in either role. On the one hand, it is iterable and functions exactly like range(start, stop, step) in those contexts. On the other, it can also be passed into list indexing exactly like slice(start, stop, step). This is a proposal that range and slice are really the same thing, just in different contexts. Why is it useful? I at least find its syntax to be simple, intuitive, and concise -- more so than the range(...) or slice(...) alternatives. It's quite obvious for an experienced Python user and just as simple to pick up as slice notation for a beginner (since it *is* slice notation). It condenses and clears up sometimes-cumbersome range expressions. A couple examples: sum(1:6) # instead of sum(range(1, 6)) list(1:6) for i in (1:6): print(i**2) (i**2 for i in (1:6)) It also makes forming reusable slices clearer and easier: my_slice = (:6:2) # instead of slice(None, 6, 2) my_list[my_slice] It has a couple of siblings that should be obvious (think list or set comprehension): [start:stop:step] # gives a list {start:stop:step} # gives a set This is similar to passing a range/slice object into the respective constructor: [1:6] # list(1:6) or [1, 2, 3, 4, 5] {1:6} # set(1:6) or {1, 2, 3, 4, 5} Note that the parentheses aren't needed when it is the only argument of a function call or is the only element within brackets or braces. It takes on its respective roles for these bracket and brace cases, just like comprehensions. This also gives rise to the normal slice syntax: my_list[1:6:2] # What is inside the brackets is a slice object. my_list[(1:6:2)] # Equivalent. The parentheses are valid but unnecessary. So here's the part that requires a little more thought. Any of the values may be omitted and in the slice context the behavior has no changes from what it already does: start and stop default to the beginning or end of the list depending on direction and the step defaults to 1. In the range context, we simply repeat these semantics, but noting that there is no longer a beginning or end of a list. Step defaults to 1 (just like range or slice). Start defaults to 0 when counting up and -1 when counting down (just like slice). If stop is omitted, the object will act like an itertools.count object, counting indefinitely. I have found infinite iteration to be a natural and oft-desired extension to a range object, but I can understand that some may want it to remain separate and pure within itertools. I also admit that the ability to form an infinite list with only three characters can be a scary thought (though we are all adults here, right? ;). Normally you have to take a couple extra keystrokes: from itertools import count list(count()) # rather than just [:] If that is the case, then raising an error when iter() is called on a range/slice object with no stop value could be another acceptable course of action. The syntax will still be left valid. And that's mainly it. Slice is iterable or range is "indexable" and the syntax can be used anywhere successive values are desired. If you want to know what it does or how to use it in some case, just think, "what would a slice object do?" or "what would a range object do?" or "how would I write a generator expression/list comprehension here?". Here are a few more examples: for i in (:5): # 5 elements 0 to 4, i.e. range(5) print(i**2) for i in (1:): # counts up from one for as long as you want, i.e. count(1) print(i**2) if i == 5: break it = iter(:) # a convenient usage for an infinite counter next(it) ' '.join(map(str, (:5:2))) # gives '0 2 4' [(:5), (5:10)] # list of range/slice objects [[:5], [5:10]] # list of lists [*(:5), *(5:10)] # uses unpacking to get flat list [*[:5], *[5:10]] # same unpacking to get flat list Otherwise you'd have to do: [list(range(5)), list(range(5, 10))] # list of lists [*range(5), *range(5, 10)] # flat list Tuples: tuple(1:6:2) # (1, 3, 5) *(1:6:2), # same I don't actually have experience developing the interpreter and underlying workings of Python, so I don't know how much of a change this requires. I thought it might be possible since the constructs already exist in the language. They just haven't been unified yet. I also realize that there are a few other use-cases that need to be ironed out. The syntax might also be too minimal in some cases to be obvious. One of the trickiest things may be what it will be called, since the current language has the two different terms. In the end it's just another range/slice idea, and the idea has probably already been proposed sometime in the past few decades, but what thoughts are there? - Nicholas [View Less]

8 16