Mailman 3 February 2020 - Python-ideas

str(obj) not calling obj.__str__?
by Jérôme Carretero 25 Feb '20

25 Feb '20

Hello, I just noticed that calling `str(x)` is actually doing (in CPython `PyObject_Str`) `type(x).__str__(x)` rather than `x.__str__()`. Context: I wanted to override __str__ for certain objects in order to “see them better”. I'm wondering why not do `x.__str__()` Best regards, -- Jérôme

3 4

Make ~ (tilde) a binary operator, e.g. __sim__(self, other)
by Aaron Hall 25 Feb '20

25 Feb '20

Currently, Python only has ~ (tilde) in the context of a unary operation (like `-`, with __neg__(self), and `+`, __pos__(self)). `~` currently calls `__invert__(self)` in the unary context. I think it would be awesome to have in the language, as it would allow modelling along the lines of R that we currently only get with text, e.g.: smf.ols(formula='Lottery ~ Literacy + Wealth + Region', data=df) With a binary context for ~, we could write the above string as pure Python, with implications for symbolic evaluation (with SymPy) and statistical modelling (such as with sklearn or statsmodels) - and other use-cases/DSLs. In LaTeX we call this `\sim` (Wikipedia indicates this is for "similar to"). I'm not too particular, but `__sim__(self, other)` would have the benefits of being both short and consistent with LaTeX. This is not a fully baked idea, perhaps there's a good reason we haven't added a binary `~`. It seems like I've seen discussion in the past. But I couldn't find such discussion. And as I'm currently taking some statistics courses, I'm getting R-feature-envy again... What do you think? Aaron Hall

15 40

Re: Make ~ (tilde) a binary operator, e.g. __sim__(self, other)
by Dominik Vilsmeier 24 Feb '20

24 Feb '20

But that behavior is specific to Jupyter and so it could similarly provide auto-complete or syntax highlighting for usage of `smf.ols(formula=...)`. Like PyCharm provides syntax highlighting for strings used as the `pattern` argument for the various functions of the `re` module. So if it's only for the sake of auto-complete or syntax highlighting, this is on the IDE and not on the language, in my opinion. On 24.02.20 17:48, David Mertz wrote: > I get auto-complete on column names in Pandas when I'm in Jupyter. But > yes, I agree with you. > > On Mon, Feb 24, 2020, 11:43 AM Dominik Vilsmeier > <dominik.vilsmeier(a)gmx.de <mailto:dominik.vilsmeier@gmx.de>> wrote: > > I don't see what's wrong with the status quo: > > smf.ols(formula='Lottery ~ Literacy + Wealth + Region', data=df) > > If I understand correctly you want to use instead: > > smf.ols(formula=df.Lottery ~ df.Literacy + df.Wealth + df.Region) > > Or since some people favor indexing over attribute access for > column names: > > smf.ols(formula=df['Lottery'] ~ df['Literacy'] + df['Wealth'] + > df['Region']) > > Both alternatives are much more verbose since you have to repeat the > `df` part or even worse the brackets for indexing. In any case you > need > to type the column names that you would like to include and there's no > auto-complete on column names that would help you typing it. So I > don't > see what's the benefit of the operator version. > > In addition this requires Pandas to implement the modeling but there's > much more to Pandas than just modeling so perhaps that better > remains a > separate project. > > On 24.02.20 01:27, Aaron Hall via Python-ideas wrote: > > I have no behavior for integers in mind. I would expect > high-level libraries to want to implement behavior for it. > > > > - sympy > > - pandas, numpy, sklearn, statsmodels > > - other mathematically minded libraries (monadic bind or compose?) > > > > To do this we need a name. I like `__sim__`. Then we'll need > `__rsim__` and `__isim__` for completeness. We need to make room > for it in the grammar. Is it ok to give it the same priority of > evaluation as `+` or `-`, or slightly higher? > > > > In the past we've made additions to the language when we've been > parsing and evaluating strings. That's what we're currently doing > in statsmodels right now because we lack the binary (in the sense > of two-arguments) `~`. > > > > See: https://www.statsmodels.org/dev/example_formulas.html > > _______________________________________________ > > Python-ideas mailing list -- python-ideas(a)python.org > <mailto:python-ideas@python.org> > > To unsubscribe send an email to python-ideas-leave(a)python.org > <mailto:python-ideas-leave@python.org> > > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/JWC4H… > > Code of Conduct: http://python.org/psf/codeofconduct/ > _______________________________________________ > Python-ideas mailing list -- python-ideas(a)python.org > <mailto:python-ideas@python.org> > To unsubscribe send an email to python-ideas-leave(a)python.org > <mailto:python-ideas-leave@python.org> > https://mail.python.org/mailman3/lists/python-ideas.python.org/ > Message archived at > https://mail.python.org/archives/list/python-ideas@python.org/message/VQOYS… > Code of Conduct: http://python.org/psf/codeofconduct/ >

1 0

Meta: ideas for Python [-ideas]
by André Roberge 23 Feb '20

23 Feb '20

(Apologies if this appears twice: I attempted to create a new thread using the Web interface but do not see any sign of it having been created.) When people suggest changes to Python's syntax on this list, it is often suggested that they might want to try doing so using an import hook. In fact, this was suggested as recently as two days ago when Andrew Barnert wrote [1] > Unfortunately, the boilerplate to write an import hook is more complicated than you’d like (and pretty hard to figure out the first time), and the support for filtering on the token stream (the most obvious way to do this one) rather than the text stream, AST, or bytecode is pretty minimal and clumsy. In an attempt to make it easier to create import hooks to "play" with non-standard syntax, I have created a library that is simply called "ideas" [2]. It includes quite a few toy examples meant to illustrate what can be done. It also includes a slightly less minimal and hopefully less clumsy way of filtering the token stream. A reasonably complete documentation is available [3]. I was planning to work some more on it but the recent comment by Andrew Barnert gave me the impetus to finish a first draft and mention it publicly. While it "works for me", consider it to be a pre-alpha release, which is ready for your suggestions, criticisms, and other comments. My hope is that eventually enough examples will be included so that, instead of simply suggesting "why don't you write an import hook for testing your proposed syntax", one could say "you can use transformations similar to that of examples X and Y of ideas to create an import hook for testing your proposed syntax". André Roberge [1] https://mail.python.org/archives/list/python-ideas@python.org/message/UNL62… [2] Code on github: https://github.com/aroberge/ideas; see the documentation for the choice of name. [3] Documentation: https://aroberge.github.io/ideas/docs/html/

3 4

Ability to specify item sizes for array.array instances in a platform-independent way
by Steve Jorgensen 23 Feb '20

23 Feb '20

Currently, if I know that I want an `array.array` object with `itemsize` of 4 there is no way to do that without first determining what the item sizes are for `'i'`/`'I'` and `'l'`/`'L'` on the current platform. Presumably, things could get even more hairy with future platforms. Below are some ideas for how to support explicit, platform-agnostic item size specification. Allow a non-str sequence with itemsize and signedness members to be given as the `typecode` value. `a = array.array((4, array.SIGNED))` Allow a numeric `itemsize` value to be given as the first positional argument instead of a `typecode` string, and have an optional named argument for signedness, signed by default. `a = array.array(4) # Signed by default.` `a = array.array(4, signedness=array.UNSIGNED)` Allow the "@" and "=" prefixes (same as in `struct` format strings) in `typecode` strings. This is my least preferred because I won't always have the typecode or prefix value choices memorized, and looking them up is an extra step. Also, the appropriate size and signedness might be determined at runtime, so having to write additional code to map from size/signedness to a typecode is an unnecessary annoyance. `a = array.array('=i') # Signed integer of "standard" integer size.`

2 3

Re: Proposal: Complex comprehensions containing statements
by Dominik Vilsmeier 22 Feb '20

22 Feb '20

2 1

Proposal: Complex comprehensions containing statements
by Alex Hall 22 Feb '20

22 Feb '20

This is a proposal for a new syntax where a comprehension is written as the appropriate brackets containing a loop which can contain arbitrary statements. Here are some simple examples. Instead of: [ f(x) for y in z for x in y if g(x) ] one may write: [ for y in z: for x in y: if g(x): f(x) ] Instead of: lst = [] for x in y: if cond(x): break z = f(x) lst.append(z * 2) one may write: lst = [ for x in y: if cond(x): break z = f(x) yield z * 2 ] Instead of: [ {k: v for k, v in foo} for foo in bar ] one may write: [ for foo in bar: {for k, v in foo: k: v} ] ## Specification A list/set/dict comprehension or generator expression is written as the appropriate brackets containing a `for` or `while` loop. In the general case some expressions have `yield` in front and they become the values of the comprehension, like a generator function. If the comprehension contains exactly one expression statement at any level of nesting, i.e. if there is only one place where a `yield` can be placed at the start of a statement, then `yield` is not required and the expression is implicitly yielded. In particular this means that any existing comprehension translated into the new style doesn't require `yield`. If the comprehension doesn't contain exactly one expression statement and doesn't contain a `yield`, it's a SyntaxError. ### Dictionary comprehensions For dictionary comprehensions, a `key: value` pair is allowed as its own pseudo-statement or in a yield. It's not a real expression and cannot appear inside other expressions. This can potentially be confused with variable type annotations with no assigned value, e.g. `x: int`. But we can essentially apply the same rule as other comprehensions: either use `yield`, or only have one place where a `yield` could be added in front of a statement. So if there is only one pair `x: y` we try to implicitly yield that. The only way this could be misinterpreted is if a user declared the type of exactly one expression and completely forgot to give their comprehension elements, and the program would almost certainly fail spectacularly. ### Whitespace If placing the loop on a single line would be valid syntax outside a comprehension (i.e. it just contains a simple statement) then we call this an *inline* comprehension. It can be inserted in the same line(s) as other code and formatted however the writer likes - there are no concerns about whitespace. For a more complex comprehension, the loop must start and end with a newline, i.e. the lines containing the loop cannot contain any tokens from outside, including the enclosing brackets. For example, this is allowed: foo = [ for x in y: if x > 0: f(x) ] but this is not: foo = [for x in y: if x > 0: f(x)] This ensures that code is readable even at a quick glance. The eyes can quickly find where the loop starts and distinguish the embedded statements from the rest of the enclosing expression. Furthermore, it's easy to copy paste entire lines to move them around, whereas refactoring the invalid example above without specific tools would be annoying and error-prone. It also makes it easy to adjust code outside the comprehension (e.g. rename `foo` to something longer) without messing up indentation and alignment. Inside the loop, the rules for indentation and such are the same as anywhere else. The syntax of the loop is valid only if it's also valid as a normal loop outside any expression. The body of the loop must be more indented than the for/while keyword that starts the loop. ### Variable scope Since comprehensions look like normal loops they should maybe behave like them again, including executing in the same scope and 'leaking' the iteration variable(s). Assignments via the walrus operator already affect the outer scope, only the iteration variable currently behaves differently. My understanding is that this is influenced by the fact that there is little reason to use the value of the iteration variable after a list comprehension completes since it will always be the last value in the iterable. But since the new syntax allows `break`, the value may become useful again. I don't know what the right approach is here and I imagine it can generate plenty of debate. Given that this whole proposal is already controversial and likely to be rejected this may not be the best place to start discussion. But maybe it is, I don't know. ## Benefits/comparison to current methods ### Uniform syntax The new comprehensions just look like normal loops in brackets, or generator functions. This should make them easier for beginners to learn than the old comprehensions. A particular concept that's easier to learn is comprehensions that contain multiple loops. Consider this comprehension over a nested list: [ f(cell) for row in matrix for cell in row ] For beginners this can easily be confusing, [and sometimes for experienced coders too](https://mail.python.org/archives/list/python-ideas@python.org/message/… ). Yes there's a rule that one can learn, but putting it in reverse also seems logical, perhaps even more so: [ f(cell) for cell in row for row in matrix ] Now the comprehension is 'consistently backwards', it reads more like English, and the usage of `cell` is right next to its definition. But of course that order is wrong...unless we want a nested list comprehension that produces a new nested list: [ [ f(cell) for cell in row ] for row in matrix ] Again, it's not hard for an experienced coder to understand this, but for a beginner grappling with new concepts this is not great. Now consider how the same two comprehensions would be written in the new syntax: [ for row in matrix: for cell in row: f(cell) ] [ for row in matrix: [ for cell in row: f(cell) ] ] ### Power and flexibility Comprehensions are great and I love using them. I want to be able to use them more often. I know I can solve any problem with a loop, but it's obvious that comprehensions are much nicer or we wouldn't need to have them at all. Compare this code: new_matrix = [] for row in matrix: new_row = [] for cell in row: try: new_row.append(f(cell)) except ValueError: new_row.append(0) new_matrix.append(new_row) with the solution using the new syntax: new_matrix = [ for row in matrix: [ for cell in row: try: yield f(cell) except ValueError: yield 0 ] ] It's immediately visually obvious that it's building a new nested list, there's much less syntax for me to parse, and the variable `new_row` has gone from appearing 4 times to 0! There have been many requests to add some special syntax to comprehensions to make them a bit more powerful: - [Is this PEP-able? "with" statement inside genexps / list comprehensions](https://mail.python.org/archives/list/python-ideas@python.o… - [Allowing breaks in generator expressions by overloading the while keyword](https://mail.python.org/archives/list/python-ideas@python.org/thre… - [while conditional in list comprehension ??](https://mail.python.org/archives/list/python-ideas@python.org/thread/RYBBHV3YBBEIBUZPZ4WNQGKI76VSBWI5/#A36BJCUAGUBZA7FIQ3LN6UMZUYCL2LJG) This would solve all such problems neatly. ### No trying to fit things in a single expression The current syntax can only contain one expression in the body. This restriction makes it difficult to solve certain problems elegantly and creates an uncomfortable grey area where it's hard to decide between squeezing maybe a bit too much into an expression or doing things 'manually'. This can lead to analysis paralysis and disagreements between coders and reviewers. For example, which of the following is the best? clean = [ line.strip() for line in lines if line.strip() ] stripped = [line.strip() for line in lines] clean = [line for line in stripped if line] clean = list(filter(None, map(str.strip, lines))) clean = [] for line in lines: line = line.strip() if line: clean.append(line) def clean_lines(): for line in lines: line = line.strip() if line: yield line clean = list(clean_lines()) You probably have a favourite, but it's very subjective and this kind of problem requires judgement depending on the situation. For example, I'd choose the first version in this case, but a different version if I had to worry about duplicating something more complex or expensive than `.strip()`. And again, there's an awkward sweet spot where it's hard to decide whether I care enough about the duplication. What about assignment expressions? We could do this: clean = [ stripped for line in lines if (stripped := line.strip()) ] Like the nested loops, this is tricky to parse without experience. The execution order can be confusing and the variable is used away from where it's defined. Even if you like it, there are clearly many who don't. I think the fact that assignment expressions were a desired feature despite being so controversial is a symptom of this problem. It's the kind of thing that happens when we're stuck with the limitations of a single expression. The solution with the new syntax is: clean = [ for line in lines: stripped = line.strip() if stripped: stripped ] or if you'd like to use an assignment expression: clean = [ for line in lines: if stripped := line.strip(): stripped ] I think both of these look great and are easily better than any of the other options. And I think it would be the clear winner in any similar situation - no careful judgement needed. This would become the one (and only one) obvious way to do it. The new syntax has the elegance of list comprehensions and the flexibility of multiple statements. It's completely scalable and works equally well from the simplest comprehension to big complicated constructions. ### Easy to change I hate when I've already written a list comprehension but a new requirement forces me to change it to, say, the `.append` version. It's a tedious refactoring involving brackets, colons, indentation, and moving things around. It also leaves me with a very unhelpful `git diff`. With the new syntax I can easily add logic as I please and get a nice simple diff.

9 16

Re: Proposal: Complex comprehensions containing statements
by Dominik Vilsmeier 22 Feb '20

22 Feb '20

2 1

Specify number of items to allocate for array.array() constructor
by Sven Rahmann 22 Feb '20

22 Feb '20

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Re: Proposal: Complex comprehensions containing statements
by Dominik Vilsmeier 21 Feb '20

21 Feb '20

4 4