Mailman 3 June 2018 - Python-ideas

Re: [Python-ideas] [Distutils] Pypi private repo's
by Nick Coghlan Oct. 28, 2018

Oct. 28, 2018

On 5 April 2018 at 07:58, Jannis Gebauer <ja.geb(a)me.com> wrote: > What if there was some kind of “blessed” entity that runs these services and puts the majority of the revenue into a fund that funds development on PyPi (maybe trough the PSF)? Having a wholly owned for-profit subsidiary that provides commercial services as a revenue raising mechanism is certainly one way to approach something like this without alienating sponsors or tax authorities (although it may still alienate the vendors of now competing services). It would require a big time commitment on the PSF side to get everything set up though, as well as interest from key folks in joining what would essentially be a single-language-focused start up in an already crowded cross-language developer tools marketplace. When the PSF as a whole is still operating with only a handful of full or part time employees, it's far from clear that setting something like that up would be the most effective possible use of their time and energy. At a more basic level, that kind of arrangement technically doesn't require anyone's blessing, it could be as straightforward as downstream tooling vendors signing up as PSF sponsors and saying "please allocate our sponsorship contribution to the Packaging WG's budget so that PyPI keeps operating well and the PyPA tooling keeps improving, increasing the level of demand for our commercial Python repository management services". Historically that wouldn't have helped much, since the PSF itself has struggled with effective project management (for a variety of reasons), but one of the things I think the success of the MOSS grant has shown is the significant strides that the PSF has made in budget management in recent years, such that if funding is made available, it can and will be spent effectively. Cheers, Nick. P.S. PyPA contributors are also free agents in their own right, so folks offering Python-centric developer workflow management tools or features may decide that it's worth their while to invest more directly in smoothing out some of the rough edges that currently still exist. It's a mercenary way of looking at things, but in many cases, it is *absolutely* possible to pay for the time and attention of existing contributors, and if you can persuade them that your proposals are reasonable, they'll often have an easier time than most convincing other community contributors that it's a good way to go :) -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

3 5

Asynchronous exception handling around with/try statement borders
by Erik Bray Sept. 24, 2018

Sept. 24, 2018

Hi folks, I normally wouldn't bring something like this up here, except I think that there is possibility of something to be done--a language documentation clarification if nothing else, though possibly an actual code change as well. I've been having an argument with a colleague over the last couple days over the proper way order of statements when setting up a try/finally to perform cleanup of some action. On some level we're both being stubborn I think, and I'm not looking for resolution as to who's right/wrong or I wouldn't bring it to this list in the first place. The original argument was over setting and later restoring os.environ, but we ended up arguing over threading.Lock.acquire/release which I think is a more interesting example of the problem, and he did raise a good point that I do want to bring up. </prologue> My colleague's contention is that given lock = threading.Lock() this is simply *wrong*: lock.acquire() try: do_something() finally: lock.release() whereas this is okay: with lock: do_something() Ignoring other details of how threading.Lock is actually implemented, assuming that Lock.__enter__ calls acquire() and Lock.__exit__ calls release() then as far as I've known ever since Python 2.5 first came out these two examples are semantically *equivalent*, and I can't find any way of reading PEP 343 or the Python language reference that would suggest otherwise. However, there *is* a difference, and has to do with how signals are handled, particularly w.r.t. context managers implemented in C (hence we are talking CPython specifically): If Lock.__enter__ is a pure Python method (even if it maybe calls some C methods), and a SIGINT is handled during execution of that method, then in almost all cases a KeyboardInterrupt exception will be raised from within Lock.__enter__--this means the suite under the with: statement is never evaluated, and Lock.__exit__ is never called. You can be fairly sure the KeyboardInterrupt will be raised from somewhere within a pure Python Lock.__enter__ because there will usually be at least one remaining opcode to be evaluated, such as RETURN_VALUE. Because of how delayed execution of signal handlers is implemented in the pyeval main loop, this means the signal handler for SIGINT will be called *before* RETURN_VALUE, resulting in the KeyboardInterrupt exception being raised. Standard stuff. However, if Lock.__enter__ is a PyCFunction things are quite different. If you look at how the SETUP_WITH opcode is implemented, it first calls the __enter__ method with _PyObjet_CallNoArg. If this returns NULL (i.e. an exception occurred in __enter__) then "goto error" is executed and the exception is raised. However if it returns non-NULL the finally block is set up with PyFrame_BlockSetup and execution proceeds to the next opcode. At this point a potentially waiting SIGINT is handled, resulting in KeyboardInterrupt being raised while inside the with statement's suite, and finally block, and hence Lock.__exit__ are entered. Long story short, because Lock.__enter__ is a C function, assuming that it succeeds normally then with lock: do_something() always guarantees that Lock.__exit__ will be called if a SIGINT was handled inside Lock.__enter__, whereas with lock.acquire() try: ... finally: lock.release() there is at last a small possibility that the SIGINT handler is called after the CALL_FUNCTION op but before the try/finally block is entered (e.g. before executing POP_TOP or SETUP_FINALLY). So the end result is that the lock is held and never released after the KeyboardInterrupt (whether or not it's handled somehow). Whereas, again, if Lock.__enter__ is a pure Python function there's less likely to be any difference (though I don't think the possibility can be ruled out entirely). At the very least I think this quirk of CPython should be mentioned somewhere (since in all other cases the semantic meaning of the "with:" statement is clear). However, I think it might be possible to gain more consistency between these cases if pending signals are checked/handled after any direct call to PyCFunction from within the ceval loop. Sorry for the tl;dr; any thoughts?

7 15

Positional-only parameters
by Victor Stinner Sept. 11, 2018

Sept. 11, 2018

Hi, For technical reasons, many functions of the Python standard libraries implemented in C have positional-only parameters. Example: ------- $ ./python Python 3.7.0a0 (default, Feb 25 2017, 04:30:32) >>> help(str.replace) replace(self, old, new, count=-1, /) # <== notice "/" at the end ... >>> "a".replace("x", "y") # ok 'a' >>> "a".replace(old="x", new="y") # ERR! TypeError: replace() takes at least 2 arguments (0 given) ------- When converting the methods of the builtin str type to the internal "Argument Clinic" tool (tool to generate the function signature, function docstring and the code to parse arguments in C), I asked if we should add support for keyword arguments in str.replace(). The answer was quick: no! It's a deliberate design choice. Quote of Yury Selivanov's message: """ I think Guido explicitly stated that he doesn't like the idea to always allow keyword arguments for all methods. I.e. `str.find('aaa')` just reads better than `str.find(needle='aaa')`. Essentially, the idea is that for most of the builtins that accept one or two arguments, positional-only parameters are better. """ http://bugs.python.org/issue29286#msg285578 I just noticed a module on PyPI to implement this behaviour on Python functions: https://pypi.python.org/pypi/positional My question is: would it make sense to implement this feature in Python directly? If yes, what should be the syntax? Use "/" marker? Use the @positional() decorator? Do you see concrete cases where it's a deliberate choice to deny passing arguments as keywords? Don't you like writing int(x="123") instead of int("123")? :-) (I know that Serhiy Storshake hates the name of the "x" parameter of the int constructor ;-)) By the way, I read that "/" marker is unknown by almost all Python developers, and [...] syntax should be preferred, but inspect.signature() doesn't support this syntax. Maybe we should fix signature() and use [...] format instead? Replace "replace(self, old, new, count=-1, /)" with "replace(self, old, new[, count=-1])" (or maybe even not document the default value?). Python 3.5 help (docstring) uses "S.replace(old, new[, count])". Victor

19 27

Fwd: Replacing Infinite while Loops with an Iterator: async edition
by jab＠math.brown.edu Aug. 24, 2018

Aug. 24, 2018

On Jun 23, 2018, at 21:11, Nathaniel Smith <n...(a)pobox.com> wrote: > He's asking for an async version of the 'iter' builtin, presumably > something like: > async def aiter(async_callable, sentinel): > while True: > value = await async_callable() > if value == sentinel: > break > yield value > -n Yes, exactly (thanks, Nathaniel). Wouldn't that be a useful built-in? (Greg, I too would be surprised if this were the first time this idea has been raised, but I looked before posting and couldn’t immediately find prior discussion.)

3 4

Allow a group by operation for dict comprehension
by Nicolas Rolin July 18, 2018

July 18, 2018

Hi, I use list and dict comprehension a lot, and a problem I often have is to do the equivalent of a group_by operation (to use sql terminology). For example if I have a list of tuples (student, school) and I want to have the list of students by school the only option I'm left with is to write student_by_school = defaultdict(list) for student, school in student_school_list: student_by_school[school].append(student) What I would expect would be a syntax with comprehension allowing me to write something along the lines of: student_by_school = {group_by(school): student for school, student in student_school_list} or any other syntax that allows me to regroup items from an iterable. Small FAQ: Q: Why include something in comprehensions when you can do it in a small number of lines ? A: A really appreciable part of the list and dict comprehension is the fact that it allows the developer to be really explicit about what he wants to do at a given line. If you see a comprehension, you know that the developer wanted to have an iterable and not have any side effect other than depleting the iterator (if he respects reasonable code guidelines). Initializing an object and doing a for loop to construct it is both too long and not explicit enough about what is intended. It should be reserved for intrinsically complex operations, not one of the base operation one can want to do with lists and dicts. Q: Why group by in particular ? A: If we take SQL queries (https://en.wikipedia.org/wiki/SQL_syntax#Queries) as a reasonable way of seeing how people need to manipulate data on a day-to-day basis, we can see that dict comprehensions already covers most of the base operations, the only missing operations being group by and having. Q: Why not use it on list with syntax such as student_by_school = [ school, student for school, student in student_school_list group by school ] ? A: It would create either a discrepancy with iterators or a perhaps misleading semantic (the one from itertools.groupby, which requires the iterable to be sorted in order to be useful). Having the option do do it with a dict remove any ambiguity and should be enough to cover most "group by" applications. Examples: edible_list = [('fruit', 'orange'), ('meat', 'eggs'), ('meat', 'spam'), ('fruit', 'apple'), ('vegetable', 'fennel'), ('fruit', 'pineapple'), ('fruit', 'pineapple'), ('vegetable', 'carrot')] edible_list_by_food_type = {group_by(food_type): edible for food_type, edible in edible_list} print(edible_list_by_food_type) {'fruit': ['orange', 'pineapple'], 'meat': ['eggs', 'spam'], 'vegetable': ['fennel', 'carrot']} bank_transactions = [200.0, -357.0, -9.99, -15.6, 4320.0, -1200.0] splited_bank_transactions = {group_by('credit' if amount > 0 else 'debit'): amount for amount in bank_transactions} print(splited_bank_transactions) {'credit': [200.0, 4320.0], 'debit': [-357.0, -9.99, -15.6, -1200.0]} -- Nicolas Rolin

20 84

Add a __cite__ method for scientific packages
by Andrei Kucharavy July 5, 2018

July 5, 2018

Over the last 10 years, Python has slowly inched towards becoming the most popular scientific computing language, beating or seriously challenging Matlab, R, Mathematica and many specialized languages (S, SAS, ...) in numerous applications. A large part of this growth is driven by amazing community packages, such as numpy, scipy, scikits-learn, scikits-image, seaborn or pandas, just to name a few. Development of such packages represents a significant time investment by people working in academic environments. To be able to justify the investment of time into such package development and support, the developers usually associated them with a scientific article. The number of citations of those articles are considered as measures of the usefulness of articles and are required to justify the time spent on them. Unfortunately, as of now, a significant issue is that such packages are not cited despite being extensively used. Part of this is due to the difficulties with compiling the list of proper citations for each module (and, for libraries associated with multiple update publications, selecting the relevant citation). Part of this is due to users not realizing which of the modules they are using have associated publications and should be cited. To remediate to that situation, I suggest a __citation__ method associated to each package installation and import. Called from the __main__, __citation__() would scan __citation__ of all imported packages and return the list of all relevant top-level citations associated to the packages. As a scientific package developer working in academia, the problem is quite serious, and the solution seems relatively straightforward. What does Python core team think about addition and long-term maintenance of such a feature to the import and setup mechanisms? What do other users and scientific package developers think of such a mechanism for citations retrieval? Best, *Andrei Kucharavy*Post-Doc @ *Joel S. Bader* * Lab*Johns Hopkins University, Baltimore, USA.

18 28

Should nested classes in an Enum be Enum members?
by Ethan Furman July 1, 2018

July 1, 2018

Consider the following Enum definition: class Color(Enum): RED = 1 GREEN = 2 BLUE = 3 @property def lower(self): return self.name.lower() def spam(self): return "I like %s eggs and spam!" % self.lower class SomeClass: pass Which of the above Color attributes are enums, and which aren't? . . . Answer: - RED, GREEN, and BLUE are members - lower and spam() are not - SomeClass /is/ a member (but not its instances) Question: Should `SomeClass` be an enum member? When would it be useful to have an embedded class in an Enum be an enum member? The only example I have seen so far of nested classes in an Enum is when folks want to make an Enum of Enums, and the nested Enum should not itself be an enum member. Since the counter-example already works I haven't seen any requests for it. ;) So I'm asking the community: What real-world examples can you offer for either behavior? Cases where nested classes should be enum members, and cases where nested classes should not be members. Thanks! -- ~Ethan~

10 15

Fwd: collections.Counter should implement fromkeys
by Tim Peters June 30, 2018

June 30, 2018

Reposting because the original got bounced from Google Groups. ---------- Forwarded message --------- From: Tim Peters <tim.peters(a)gmail.com> Date: Fri, Jun 29, 2018 at 5:54 PM Subject: Re: [Python-ideas] collections.Counter should implement fromkeys To: <abedillon(a)gmail.com> Cc: python-ideas <python-ideas(a)googlegroups.com> [Abe Dillon <abedillon(a)gmail.com>] > ... > I'm using the copy-constructor because I know Counter is a subclass of dict. > I'm using fromkeys because I know how that class method works. > So why does the subclass lack functionality that the superclass has? > Because programmers wouldn't be able to wrap their heads around it? > I don't buy it. This feels like nanny-design trumping SOLID design <https://en.wikipedia.org/wiki/SOLID>. More because Counter.fromkeys() could be incoherent. From the implementation (in your Lib/collections/__init__.py): @classmethod def fromkeys(cls, iterable, v=None): # There is no equivalent method for counters because setting v=1 # means that no element can have a count greater than one. raise NotImplementedError( 'Counter.fromkeys() is undefined. Use Counter(iterable) instead.') For a dict, a value appearing multiple times in the iterable doesn't matter. But a fundamental use case for Counters is to tally the _number_ of times duplicate keys appear. So, e.g., someone will be unpleasantly surprised no matter what: Counter.fromkeys("aaaaa", 2) returned. "It should set key 'a' to 2! that's what I said it should do!" "No! It should set key 'a' to 10! that's what a Counter _always_ does - sums the values associated with duplicate keys!" "You're both right - and wrong! It should raise an exception if there's a duplicate key, because there's no compelling answer to what it should do!" I expect Raymond called it NotImplementedError instead so he could release the code instead of waiting 3 years for that debate to end ;-)

4 6

Re: [Python-ideas] collections.Counter should implement fromkeys
by Abe Dillon June 30, 2018

June 30, 2018

[Michael Selik] > You might be pursuing a local optimum of obviousness. If you step back > from the idea of "initialize the counts with all interesting keys" and > looping over them, you might find a better overall solution. > That's a distinct possibility, but this is far from the first time I've wanted for a better way to initialize a Counter. It's just a simple example for which there are many many alternative approaches. If' you're interested, I have a more complete implementation of the game of life here <https://repl.it/@AbeDillon/Life>. [Michael Selik] > That happens more frequently than my OOP professor seemed to think. I end > up with platypi (platypuses?) all too often. Yes, I always try to caution my own students against strict adherence to any ideology, methodology, or cult. "Always do this" and "Never do that" rules are useful crutches for novices trying to navigate the complex world of coding for the first time, but pros are the ones who are experienced enough to know when it's OK to break the rules. When GoTo isn't harmful. When to break the rule of thirds <https://www.youtube.com/watch?v=hyGAgqfoUo8>. Obviously, Python breaks SOLID principals successfully all over the place for pragmatic reasons. I don't think this is one of those cases.

1 0

Re: [Python-ideas] collections.Counter should implement fromkeys
by Abe Dillon June 29, 2018

June 29, 2018

[Michael Selik] > You need an iterable of the keys you're interested in to pass to the > hypothetical ``fromkeys`` method. Why not iterate over that same iterable > paired with zeros instead of passing it into ``fromkeys``? > Because, as in my original example code; the values could be zero or they could be more, I just want to make sure the keys are in the counter when I iterate. I'm not having any trouble finding a work around. I'm having trouble understanding why I need to find a work around when Counter already inherits from dict, dict.fromkeys is perfectly well defined, and there's not really any other *obvious* best way to initialize the value for a set of keys. Counter(dict.fromkeys(keys, value)) works just fine, but it feels wrong. I'm using the copy-constructor because I know Counter is a subclass of dict. I'm using fromkeys because I know how that class method works. So why does the subclass lack functionality that the superclass has? Because programmers wouldn't be able to wrap their heads around it? I don't buy it. This feels like nanny-design trumping SOLID design <https://en.wikipedia.org/wiki/SOLID>.

1 0