Mailman 3 April 2016 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' … [View More]

51 165

https://docs.python.org/fr/ ?
by Julien Palard Feb. 2, 2017

Feb. 2, 2017

o/ The french translation of the Python Documentation [1][2] has translated 20% of the pageviews of docs.python.org. I think it's the right moment to push it do docs.python.org. So there's some questions ! And I'd like feedback. TL;DR (with my personal choices): - URL may be "http://docs.python.org/fr/" - For localized variations of languages we should use dash and lowercase like "docs.python.org/pt-br/" - po files may be hosted on the python's github - existing script to build … [View More]doc may be patched to build translations - each translations may crosslink to others - untranslated strings may be visually marked as so I also opened: http://bugs.python.org/issue26546. # Chronology, dependencies The only blocking decision here is the URL, (also reviewing my patch ...), with those two, translated docs can be pushed to production, and the other steps can be discussed and applied one by one. # The URL ## CCTLD vs path vs subdomain I think we should use a variation of "docs.python.org/fr/" for simplicity and clarity. I think we should avoid using CCTLDs as they're sometime hard or near impossible to obtain (may cost a lot of time), also some are expensive, so it's time and money we clearly don't need to loose. Last possibility I see is to use a subdomain, like fr.docs.python.org or docs.fr.python.org but I don't think it's the role / responsibility of the sub-domain to do it. So I'm for docs.python.org/LANGUAGE_TAG/ (without moving current documentation inside a /en/). ## Language tag in path ### Dropping the default locale of a language I personally think we should not show the region in case it's redundant: so to use "fr" instead of "fr-FR", "de" instead of "de-DE", but keeping the possibility to use a locale code when it's not redundant like for "pt-br" or "de-AT" (German ('de') as used in Austria ('AT')). I think so because I don't think we'll have a lot of locale variations (like de-AT, fr-CH, fr-CA, ...) so it will be most of the time redundant (visually heavy, longer to type, longer to read) but we'll still need some locale (pt-BR typically). ### gettext VS IETF language tag format gettext goes by using an underscore between language and locale [3] and IETF goes by using a dash [4][5]. As sphinx is using gettext, and gettext uses underscore we may choose underscore too. But URLs are not here to leak the underlying implementation, and the IETF looks like to be the standard way to represent language tags. Also I visually prefer the dash over the underscore, so I'm for the dash here. ### Lower case vs upper case local tag RFC 5646 section-2.1 tells us language tags are not case sensitive, yet ISO3166-1 recommends that country codes (part of the language tag) be capitalized. I personally prefer the all-lowercase one as paths in URLs typically are lowercase. I searched for `inurl:"pt-br"` to see if I'm not too far away from the usage here and usage seems to agree with me, although there's some "pt-BR" in urls. # Where to host the translated files Currently we're hosting the *po* files in the afpy's (Francophone association for python) [6] github [1] but it may make sense to use (in the generation scripts) a more controlled / restricted clone in the python github, at least to have a better view of who can push on the documentation. We may want to choose between aggregating all translations under the same git repository but I don't feel it's useful. # How to Currently, a python script [7] is used to generate `docs.python.org`, I proposed a patch in [8] to make this script clone and build the french translation too, it's a simple and effective way, I don't think we need more ? Any idea welcome. In our side, we have a Makefile [12] to build the translated doc which is only a thin layer on top of the Sphinx Makefile. So my proposed patch to build scripts "just" delegate the build to our Makefile which itself delegate the hard work to the Sphinx Makefile. # Next ? ## Document how to translate Python I think I can (should) write a documentation on "how to start a Python doc translation project" and "how to migrate existing [9][10][11] python doc translation projects to docs.python.org" if french does goes docs.python.org because it may hopefully motivate people to do the same, and I think our structure is a nice way to do it (A Makefile to generate the doc, all versions translated, people mainly working on latest version, scripts to propagating translations to older version, etc...). ## Crosslinking between existing translations Once the translations are on `docs.python.org`, crosslinks may be established so people on a version can be aware of other version, and easily switch to them. I'm not a UI/UX man but I think we may have a select box right before the existing select box about version, on the top-left corner. Right before because it'll reflect the path: /fr/3.5/ -> [select box fr][select box 3.5]. ## Marking as "untranslated, you can help" the untranslated paragraphs The translations will always need work to follow upstream modifications: marking untranslated paragraphs as so may transform the "Oh they suck, this paragraph is not even translated :-(" to "Hey, nice I can help translating that !". There's an opened sphinx-doc ticket to do so [13] but I have not worked on it yet. As previously said I'm real bad at designing user interfaces, so I don't even visualize how I'd like it to be. [1] http://www.afpy.org/doc/python/3.5/ [2] https://github.com/afpy/python_doc_fr [3] https://www.gnu.org/software/gettext/manual/html_node/Locale-Names.html [4] http://tools.ietf.org/html/rfc5646 [5] https://en.wikipedia.org/wiki/IETF_language_tag [6] http://www.afpy.org/ [7] https://github.com/python/docsbuild-scripts/ [8] http://bugs.python.org/issue26546 [9] http://docs.python.jp/3/ [10] https://github.com/python-doc-ja/python-doc-ja [11] http://docs.python.org.ar/tutorial/3/index.html [12] https://github.com/AFPy/python_doc_fr/blob/master/Makefile [13] https://github.com/sphinx-doc/sphinx/issues/1246 -- Julien Palard [View Less]

7 8

proposal: "python -m foo" should bind sys.modules['foo']
by Cameron Simpson Jan. 18, 2017

Jan. 18, 2017

Hello all, This is a writeup of a proposal I floated here: https://mail.python.org/pipermail/python-list/2015-August/694905.html last Sunday. If the response is positive I wish to write a PEP. Briefly, it is a natural expectation in users that the command: python -m module_name ... used to invoke modules in "main program" mode on the command line imported the module as "module_name". It does not, it imports it as "__main__". An import within the program of "module_name" makes a new … [View More]instance of the module, which causes cognitive dissonance and has the side effect that now the program has two instances of the module. What I propose is that the above command line _should_ bind sys.modules['module_name'] as well as binding '__main__' as it does currently. I'm proposing that the python -m option have this effect (python pseudocode): % python -m module.name ... runs: # pseudocode, with values hardwired for clarity import sys M = new_empty_module(name='__main__', qualname='module.name') sys.modules['__main__'] = M sys.modules['module.name'] = M # load the module code from wherever (not necessarily a file - CPython # already must do this phase) M.execfile('/path/to/module/name.py') Specificly, this would have the following two changes to current practice: 1) the module is imported _once_, and bound to both its canonical name and also to __main__. 2) imported modules acquire a new attribute __qualname__ (analogous to the recent __qualname__ on functions). This is always the conanoical name of the module as resolved by the importer. For most modules __name__ will be the same as __qualname__, but for the "main" module __name__ will be '__main__'. This change has the following advantages: The current standard boilerplate: if __name__ == '__main__': ... invoke "main program" here ... continues to work unchanged. Importantly, if the program then issues "import module_name", it is already there and the existing instance is found and used. The thread referenced above outlines my most recent encounter with this and the trouble it caused me. Followup messages include some support for this proposed change, and some criticism. The critiquing article included some workarounds for this multiple module situation, but they were (1) somewhat dependent on modules coming from a file pathname and (2) cumbersome and require every end user to adopt these changes if affected by the situation. I'd like to avoid that. Cheers, Cameron Simpson <cs(a)zip.com.au> The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. - George Bernard Shaw [View Less]

5 13

Importing public symbols and simultainiously privatizing them, is too noisy
by Rick Johnson Dec. 29, 2016

Dec. 29, 2016

Has anyone else found this to be too syntactically noisy? from module import Foo as _Foo, bar as _bar That is horrifically noisy IMO. The problem is, how do we remove the noise without sacrificing intuitiveness? My first idea was to do this: from module import_private Foo, bar And while it's self explanatory, it's also too long. So i thought... from module _import Foo, bar I'm leaning more towards the latter, but i'm not loving it either. Any ideas?

9 17

Re: [Python-ideas] Exposing regular expression bytecode
by Jonathan Goble June 5, 2016

June 5, 2016

OK, thanks for the comments, everyone. I'm glad to hear that people generally think this is a useful idea. Some specific replies: On Tue, Feb 16, 2016 at 4:22 AM, Chris Angelico <rosuav(a)gmail.com> wrote: > For what it's worth, I read your post with interest, but didn't have > anything substantive to reply - mainly because I don't use regexes > much. But it would be rather cool to be able to decompile a regex. > Imagine a regex pretty-printer: compile an arbitrary string, … [View More]and if > it's a valid regex, decompile it to a valid source code form, using > re.VERBOSE. That could help _hugely_ with debugging, if the trick can > be pulled off. > > ChrisA That's exactly the type of tools I envision being made available by third parties. Depending on how much I get invested into this project, I may even write such a tool myself (though that's not guaranteed). On Tue, Feb 16, 2016 at 4:55 AM, Paul Moore <p.f.moore(a)gmail.com> wrote: > Sorry. I don't personally have any issue with the proposal, and it > sounds like a reasonable idea. I don't think it's likely to be > *hugely* controversial - although it will likely need a little care in > documenting the feature to ensure that we are clear that there's no > guarantees of backward compatibility that we don't want to commit to > on the newly - exposed data. And we should also ensure that by > exposing this information, we don't preclude changes such as the > incorporation of the regex module (I don't know if the regex module > has a bytecode implementation like the re module does). The regex implementation is indeed something I would need to investigate here, and will do so before I go too far. > The next step is probably simply to raise a tracker issue for this. I > know you said you have little C experience, but the big problem is > that it's unlikely that any of the core devs with C experience will > have the time or motivation to code up your idea. So without a working > patch, and someone willing and able to respond to comments on the > patch, it's not likely to progress. Tracker issue is already filed: http://bugs.python.org/issue26336 I actually filed the issue before I realized that the mailing lists were a better place to discuss it. > But if you are willing to dig into Python's C API yourself (and it > sounds like you are) there are definitely people who will help you. > You might want to join the core mentorship list (see > http://pythonmentors.com/) where you should get plenty of assistance. > This proposal sounds like a great "beginner" task, as well - so even > if you don't want to implement it yourself, still put it on the > tracker, and mark it as an "easy" change, and maybe some other > newcomer who wants a task to help them learn the C API will pick it > up. I'll look into the mentorship list; thanks for the link. As for marking it "easy", I don't seem to have the necessary permissions to change the Keywords field; perhaps you or someone else can set that flag for me? If so, I'd appreciate it. :-) > Hope that helps - thanks for the suggestion and sorry if it seems like > no-one was interested at first. It's an unfortunate fact of life > around here that things *do* take time to get people's interest. You > mention patience in one of your messages - that's definitely something > you'll need to cultivate, I'm afraid... :-) Patience is something I've been working on since I was a little kid. I'm 29 years old now, and it still eludes me from time to time. But yes, it's something I'll have to work on. :-P Also, I received a small patch off-list from Petr Viktorin implementing a getter for the code list (thanks, Petr). I'll need to test it, but from the little I know of the C API it looks like it will get me started in the right direction. Assuming that works, what's left is a public constructor for the regex type (to enable optimizers), a dis-like module, and docs and tests. I don't think this would be major enough to require a PEP, but of course being new here, I'm open to being told I'm wrong. :-) [View Less]

4 7

solving multi-core Python
by Eric Snow June 3, 2016

June 3, 2016

tl;dr Let's exploit multiple cores by fixing up subinterpreters, exposing them in Python, and adding a mechanism to safely share objects between them. This proposal is meant to be a shot over the bow, so to speak. I plan on putting together a more complete PEP some time in the future, with content that is more refined along with references to the appropriate online resources. Feedback appreciated! Offers to help even more so! :) -eric -------- Python's multi-core story is murky at best. … [View More]Not only can we be more clear on the matter, we can improve Python's support. The result of any effort must make multi-core (i.e. parallelism) support in Python obvious, unmistakable, and undeniable (and keep it Pythonic). Currently we have several concurrency models represented via threading, multiprocessing, asyncio, concurrent.futures (plus others in the cheeseshop). However, in CPython the GIL means that we don't have parallelism, except through multiprocessing which requires trade-offs. (See Dave Beazley's talk at PyCon US 2015.) This is a situation I'd like us to solve once and for all for a couple of reasons. Firstly, it is a technical roadblock for some Python developers, though I don't see that as a huge factor. Regardless, secondly, it is especially a turnoff to folks looking into Python and ultimately a PR issue. The solution boils down to natively supporting multiple cores in Python code. This is not a new topic. For a long time many have clamored for death to the GIL. Several attempts have been made over the years and failed to do it without sacrificing single-threaded performance. Furthermore, removing the GIL is perhaps an obvious solution but not the only one. Others include Trent Nelson's PyParallels, STM, and other Python implementations.. Proposal ======= In some personal correspondence Nick Coghlan, he summarized my preferred approach as "the data storage separation of multiprocessing, with the low message passing overhead of threading". For Python 3.6: * expose subinterpreters to Python in a new stdlib module: "subinterpreters" * add a new SubinterpreterExecutor to concurrent.futures * add a queue.Queue-like type that will be used to explicitly share objects between subinterpreters This is less simple than it might sound, but presents what I consider the best option for getting a meaningful improvement into Python 3.6. Also, I'm not convinced that the word "subinterpreter" properly conveys the intent, for which subinterpreters is only part of the picture. So I'm open to a better name. Influences ======== Note that I'm drawing quite a bit of inspiration from elsewhere. The idea of using subinterpreters to get this (more) efficient isolated execution is not my own (I heard it from Nick). I have also spent quite a bit of time and effort researching for this proposal. As part of that, a number of people have provided invaluable insight and encouragement as I've prepared, including Guido, Nick, Brett Cannon, Barry Warsaw, and Larry Hastings. Additionally, Hoare's "Communicating Sequential Processes" (CSP) has been a big influence on this proposal. FYI, CSP is also the inspiration for Go's concurrency model (e.g. goroutines, channels, select). Dr. Sarah Mount, who has expertise in this area, has been kind enough to agree to collaborate and even co-author the PEP that I hope comes out of this proposal. My interest in this improvement has been building for several years. Recent events, including this year's language summit, have driven me to push for something concrete in Python 3.6. The subinterpreter Module ===================== The subinterpreters module would look something like this (a la threading/multiprocessing): settrace() setprofile() stack_size() active_count() enumerate() get_ident() current_subinterpreter() Subinterpreter(...) id is_alive() running() -> Task or None run(...) -> Task # wrapper around PyRun_*, auto-calls Task.start() destroy() Task(...) # analogous to a CSP process id exception() # other stuff? # for compatibility with threading.Thread: name ident is_alive() start() run() join() Channel(...) # shared by passing as an arg to the subinterpreter-running func # this API is a bit uncooked still... pop() push() poison() # maybe select() # maybe Note that Channel objects will necessarily be shared in common between subinterpreters (where bound). This sharing will happen when the one or more of the parameters to the function passed to Task() is a Channel. Thus the channel would be open to the (sub)interpreter calling Task() (or Subinterpreter.run()) and to the new subinterpreter. Also, other channels could be fed into such a shared channel, whereby those channels would then likewise be shared between the interpreters. I don't know yet if this module should include *all* the essential pieces to implement a complete CSP library. Given the inspiration that CSP is providing, it may make sense to support it fully. It would be interesting then if the implementation here allowed the (complete?) formalisms provided by CSP (thus, e.g. rigorous proofs of concurrent system models). I expect there will also be a _subinterpreters module with low-level implementation-specific details. Related Ideas and Details Under Consideration ==================================== Some of these are details that need to be sorted out. Some are secondary ideas that may be appropriate to address in this proposal or may need to be tabled. I have some others but these should be sufficient to demonstrate the range of points to consider. * further coalesce the (concurrency/parallelism) abstractions between threading, multiprocessing, asyncio, and this proposal * only allow one running Task at a time per subinterpreter * disallow threading within subinterpreters (with legacy support in C) + ignore/remove the GIL within subinterpreters (since they would be single-threaded) * use the GIL only in the main interpreter and for interaction between subinterpreters (and a "Local Interpreter Lock" for within a subinterpreter) * disallow forking within subinterpreters * only allow passing plain functions to Task() and Subinterpreter.run() (exclude closures, other callables) * object ownership model + read-only in all but 1 subinterpreter + RW in all subinterpreters + only allow 1 subinterpreter to have any refcounts to an object (except for channels) * only allow immutable objects to be shared between subinterpreters * for better immutability, move object ref counts into a separate table * freeze (new machinery or memcopy or something) objects to make them (at least temporarily) immutable * expose a more complete CSP implementation in the stdlib (or make the subinterpreters module more compliant) * treat the main interpreter differently than subinterpreters (or treat it exactly the same) * add subinterpreter support to asyncio (the interplay between them could be interesting) Key Dependencies ================ There are a few related tasks/projects that will likely need to be resolved before subinterpreters in CPython can be used in the proposed manner. The proposal could implemented either way, but it will help the multi-core effort if these are addressed first. * fixes to subinterpreter support (there are a couple individuals who should be able to provide the necessary insight) * PEP 432 (will simplify several key implementation details) * improvements to isolation between subinterpreters (file descriptors, env vars, others) Beyond those, the scale and technical scope of this project means that I am unlikely to be able to do all the work myself to land this in Python 3.6 (though I'd still give it my best shot). That will require the involvement of various experts. I expect that the project is divisible into multiple mostly independent pieces, so that will help. Python Implementations =================== They can correct me if I'm wrong, but from what I understand both Jython and IronPython already have subinterpreter support. I'll be soliciting feedback from the different Python implementors about subinterpreter support. C Extension Modules ================= Subinterpreters already isolate extension modules (and built-in modules, including sys). PEP 384 provides some help too. However, global state in C can easily leak data between subinterpreters, breaking the desired data isolation. This is something that will need to be addressed as part of the effort. [View Less]

26 130

Type hinting for path-related functions
by Koos Zevenhoven May 19, 2016

May 19, 2016

I actually proposed this already in one of the pathlib threads on python-dev, but I decided to repost here, because this is easily seen as a separate issue. I'll start with some introduction, then moving on to the actual type hinting part. In our seemingly never-ending discussions about pathlib support in the stdlib in various threads, first here on python-ideas, then even more extensively on python-dev, have perhaps almost converged. The required changes involve a protocol method, probably … [View More]named __fspath__, which any path-like type could implement to return a more, let's say, "classical" path object such as a str. However, the protocol is polymorphic and may also return bytes, which has a lot do do with the fact that the stdlib itself is polymophic and currently accepts str as well as bytes paths almost everywhere, including the newly-introduced os.scandir + DirEntry combination. The upcoming improvements will further allow passing pathlib path objects as well as DirEntry objects to any stdlib function that take paths. It came up, for instance here [1], that the function associated with the protocol, potentially named os.fspath, will end up needing type hints. This function takes pathlike objects and turns them into str or bytes. There are various different scenarios [2] that can be considered for code dealing with paths, but let's consider the case of os.path.* and other traditional python path-related functions. Some examples: os.path.join Currently, it takes str or bytes paths and returns a joined path of the same type (mixing different types raises an exception). In the future, it will also accept pathlib objects (underlying type always str) and DirEntry (underlying type str or bytes) or third-party path objects (underlying type str or bytes). The function will then return a pathname of the underlying type. os.path.dirname Currently, it takes a str or bytes and returns the dirname of the same type. In the future, it will also accept Path and DirEntry and return the underlying type. Let's consider the type hint of os.path.dirname at present and in the future: Currently, one could write def dirname(p: Union[str, bytes]) -> Union[str, bytes]: ... While this is valid, it could be more precise: pathstring = typing.TypeVar('pathstring', str, bytes) def dirname(p: pathstring) -> pathstring: ... This now contains the information that the return type is the same as the argument type. The name 'pathstring' may be considered slightly misleading because "byte strings" are not actually strings in Python 3, but at least it does not advertise the use of bytes as paths, which is very rarely desirable. But what about the future. There are two kinds of rich path objects, those with an underlying type of str and those with an underlying type of bytes. These should implement the __fspath__() protocol and return their underlying type. However, we do care about what (underlying) type is provided by the protocol, so we might want to introduce something like typing.FSPath[underlying_type]: FSPath[str] # str-based pathlike, including str FSPath[bytes] # bytes-based pathlike, including bytes And now, using the above defined TypeVar pathstring, the future version of dirname would be type annotated as follows: def dirname(p: FSPath[pathstring]) -> pathstring: ... It's getting late. I hope this made sense :). -Koos [1] https://mail.python.org/pipermail/python-dev/2016-April/144246.html [2] https://mail.python.org/pipermail/python-dev/2016-April/144239.html [View Less]

10 55

Bytecode for calling function with keyword arguments
by Serhiy Storchaka May 15, 2016

May 15, 2016

Currently the call of function with keyword arguments (say `f(a, b, x=c, y=d)`) is compiled to following bytecode: 0 LOAD_NAME 0 (f) 3 LOAD_NAME 1 (a) 6 LOAD_NAME 2 (b) 9 LOAD_CONST 0 ('x') 12 LOAD_NAME 3 (c) 15 LOAD_CONST 1 ('y') 18 LOAD_NAME 4 (d) 21 CALL_FUNCTION 514 (2 positional, 2 keyword pair) For every positional argument … [View More]

4 5

Exposing flat bytecode representation to optimizers
by Andrew Barnert May 15, 2016

May 15, 2016

The biggest pain with dealing with the peephole optimizer is that it happens after all the complicated flattening and fixup[^1] the compiler does, which means you have to hack up the jump targets as you go along. The future bytecode optimizers that PEP 511 enables will have the same headache. But this isn't actually necessary. The optimizer could work on a flat array of instructions[^2] instead of an array of bytes, with relocatable jump targets instead of fixed byte offsets, and then the … [View More]compiler could do the fixup _after_ the optimization.[^3] It would break the optimizer APIs, but `PyCode_Optimize` isn't public, and the API proposed by PEP 511 is public, but that PEP isn't even finalized, much less accepted yet. I don't think we need to expose the intermediate representation any farther along than the `PyCode_Optimize` step.[^4] Just moving the optimize one step earlier in the chain solves more than enough to be worth it. [^1]: If you think numbering the offsets and emitting the jump targets is easy: Every time you fix up a jump, that may require adding an `EXTENDED_ARG`, which means you have to redo any fixups that cross the the current instruction. The compiler does this by just looping until it hasn't added any more `EXTENDED_ARG`s. [^2]: In case anyone's thinking that wordcode would solve this problem, it doesn't. The `EXTENDED_ARG` jump targets are a much bigger hassle than the 1-or-3-byte-ops, and wordcode not only can't eliminate those, but makes `EXTENDED_ARG` more common. [^3]: The compiler doesn't actually have exactly what the optimizers would want, but it's pretty close: it has a linked list of block objects, each of which has an array of instruction objects, with jump targets being pointers to blocks. That's actually even better to work with, but too complicated to expose to optimizers. Flattening it would be trivial. Or, if that's too expensive, we could do something almost as simple and much cheaper: convert it in-line to a deque-like linked list of arrays, with jump targets being indices or pointers into that. Or we could just expose the list of blocks as-is, as an opaque thing with a mutable-deque-of-instructions API around it. [^4]: Later stages--import hooks, optimizing decorators, etc.--have the same pain as the peephole optimizer, but they also have code objects that contain other code objects, and they can be written in Python, and so on, so the same solution isn't feasible there. Of course we could add a function to convert bytecode back to a list of instructions, in some format that can be exposed to Python (in fact, `dis` already does 90% of that), and then another one to convert that back to bytecode and redo the fixup (which is basically the third-party `byteplay` module). But that's almost certainly overkill. (If we wanted that, I'd rather add `byteplay` to the stdlib, port it and `dis` to C, and expose a C API for them. And then we could use that for everything, including the peephole optimizer and PEP 511 optimizers. Which would all be really cool, but I think it's more work than we want to do, and I don't know if we'd actually want something like `byteplay` builtin even if it were easy...) [View Less]

6 16

unittest: assertEqual(a, b, msg): Show diff AND msg
by Thomas Güttler April 27, 2016

April 27, 2016

Up to now assertEqual(a, b, msg) outputs only the msg, not the diff. I know that setting longMessage to True shows the diff and the msg[1] I think the sane default is to show the diff and the message for assertEqual(). What do you think? Regards, Thomas Güttler [1] longMessage: https://docs.python.org/3/library/unittest.html#unittest.TestCase.longMessa… -- Thomas Guettler http://www.thomas-guettler.de/

2 3

Allow with (x as y, z as w):
by Ram Rachum April 26, 2016

April 26, 2016

Hi, I have a problem: I've got code that has a `with` statement with multiple long names, so I have to break the line awkwardly using the \ character: with some_darn_object.my_long_named_method_on_it(argument='yep') as foo, \ a_different_darn_object.and_yet_another_method() as bar: I tried wrapping the context managers in parentheses, but it looks like it's not legal syntax: File "<stdin>", line 1 with (some_darn_object.my_long_named_method_on_it(argument='yep') as foo, … [View More]

4 3

Re: [Python-ideas] Shrink recursion error tracebacks (was: Have REPL print less by default)
by Émanuel Barry April 24, 2016

April 24, 2016

I have been thinking, and especially after Nick’s comments, that it might be better to keep it as simple as possible to reduce the risk of errors happening while we’re printing the tracebacks. Recursive functions (that go deep enough to trigger an exception) are usually within one function, in the REPL, or by using __getattr__ / __getattribute__. Nice to have? Sure. Necessary? Don’t think as much. I’m also a very solid -1 on the idea of prompting the user to do something like full_ex() to … [View More]

4 5

Override dict.__new__ to raise if cls is not dict; do the same for str, list, etc.
by Neil Girdhar April 22, 2016

April 22, 2016

Sometimes users inherit from builtin types only to find that their overridden methods are not called. Instead of this being a trap for unsuspecting users, I suggest overriding the __new__ method of these types so that it will raise with an informative exception explaining that, e.g., instead of inheriting from dict, you should inherit from UserDict. I suggest this modification to any Python implementation that has special versions of classes that cannot easily be extended, such as … [View More]

12 35

Shrink recursion error tracebacks (was: Have REPL print less by default)
by Émanuel Barry April 22, 2016

April 22, 2016

This idea was mentioned a couple of times in the previous thread, and it seems reasonable to me. Getting recursion errors when testing a function in the interactive prompt often screwed me over, as I have a limited scrollback of 1000 lines at best on Windows (I never checked exactly how high/low the limit was), and with Python's recursion limit of 1000, that's a whopping 1000 to 2000 lines in my console, effectively killing off anything useful I might have wanted to see. Such as, for example, … [View More]the function definition that triggered the exception. Of course, the same is true for actual programs, where tracebacks drown off everything else useful. For all we know, a typo caused a NameError in one of the functions which somehow triggered it to call itself over again until Python decided it was enough, but you can't know that because your entire scrollback is full of File "<stdin>", line 1, in func And, for programs that are user-facing, I take all tracebacks and redirect them to a file that the user can submit me so I can debug the issue, and having tens of hundreds of the same line hurts readability. The issue that the output from running a certain script in the REPL and as a script would differ has been raised, and I am of the opinion that in no way, shape or form should the two have different behaviours (the exception being sys.ps1, sys.ps2 and builtins._ in interactive mode, which are 100% fine). I'm suggesting a change to how Python handles specifically recursion limits, to cut after a sane number of times (someone suggested to shrink the output after 3 times), and simply stating how many more identical messages there are. I would also like to extend this to any arbitrary loop of callers (for example foo -> bar -> baz -> foo and so on would be counted as one "call" for the purposes of this proposal). Under this proposal, something similar to this would happen: Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in func File "<stdin>", line 1, in func File "<stdin>", line 1, in func [Previous 1 message(s) repeated 996 more times] RecursionError: maximum recursion depth exceeded With multiple chained calls (I don't know how hard it would be to implement this, probably not trivial): Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz File "<stdin>", line 1, in foo File "<stdin>", line 1, in bar File "<stdin>", line 1, in baz [Previous 3 message(s) repeated 330 more times] RecursionError: maximum recursion depth exceeded We can probably afford to cut out immediately as we notice the recursion, but even just 3 times with multiple chained calls isn't going to be anywhere near as long as it currently is. Thoughts? Suggestions? Something I forgot? -Emanuel [View Less]

9 13

Why was Path.path added?
by Antony Lee April 21, 2016

April 21, 2016

Well, it's already there and it's probably too late to remove it, but introducing the idiom path_str = getattr(arg, "path", arg) was not necessary: you could also write path_str = str(Path(arg)) which works just as well. My 2c., Antony

5 4

Have REPL print less by default
by Franklin? Lee April 21, 2016

April 21, 2016

Two different (probably radical) ideas, with the same justification: reduce often-useless output in the REPL, which can flood out terminal history and overwhelm the user. 1. Limit the output per entered command: If you type into the REPL (AKA interactive shell), list(range(n)) and you forgot that you set n to 10**10, the interpreter should not print more than a page of output. Instead, it will print a few lines ("... and approximately X more lines"), and tell you how to print more. (E.g. … [View More]"Call '_more()' for more. Call '_full()' for full output.") Alternatively, have "less"-like behavior. 2. Only print a few parts of the stack trace. In particular, for a recursive or mutually recursive function, if the error was due to maximum recursion (is this reasonably distinguishable? the error is `RuntimeError('maximum recursion depth exceeded')`), try to print each function on the stack once each. Again, there should be a message telling you how to get the full stacktrace printed. EXACTLY how, preferably in a way that is easy to type, so that a typo won't cause the trace to be lost. It should not use `sys.something()`, because the user's first few encounters with this message will result in, "NameError: name 'sys' is not defined". A few possible rules to reduce stacktrace size: - Always show the last (top) frame(?). - Hide any other stdlib funcs directly below (above) it. - If a function appears more than once in a row, show it once, with the note, "(and X recursive calls)". - If functions otherwise appears more than once (usually by mutual recursion?), and there is a run of them, list them as, "(Mutual recursion: 'x' (5 times), 'y' (144 times), 'z' (13 times).)". These two behaviors and their corresponding functions could go into a special module which is (by default) loaded by the interactive shell. The behaviors can be turned off with some command-line verbosity flag, or tuned with command-line parameters (e.g. how many lines/pages to print). Justification: - Excessive output floods the terminal window. Some terminals have a limit on output history (Windows *defaults* to 300 lines), or don't let you scroll up at all (at least, my noob self couldn't figure it out when I did get flooded). - Students learning Python, and also everyone else using Python, don't usually need 99% of a 10000-line output or stack trace. - People who want the full output are probably advanced users with, like, high-limit or unlimited window size, and advanced users are more likely to look for a verbosity flag, or use a third-party REPL. Default should be newbie friendly, because advanced users can work around it. Thoughts? Even if the specific proposals are unworkable, is limiting REPL output (by default) a reasonable goal? [View Less]

16 39

Re: [Python-ideas] Override dict.__new__ to raise if cls is not dict; do the same for str, list, etc.
by Neil Girdhar April 21, 2016

April 21, 2016

Good point, I withdraw my suggestion. I think it's unfortunate for Python to have traps like this, but I don't see a nice way to protect users while still letting people do whatever they want. On Thu, Apr 21, 2016 at 3:03 AM Neil Girdhar <mistersheik(a)gmail.com> wrote: > I guess that's fair. > > On Thu, Apr 21, 2016 at 2:36 AM Nick Coghlan <ncoghlan(a)gmail.com> wrote: > >> On 21 April 2016 at 12:51, Neil Girdhar <mistersheik(a)gmail.com> wrote: >> &… [View More]

1 0

Anonymous namedtuples
by Joseph Martinot-Lagarde April 20, 2016

April 20, 2016

Hi, list ! namedtuples are really great. I would like to use them even more, for example for functions that return multiple arguments. The problem is that namedtuples have to be "declared" beforehand, so it would be quite tedious to declare a nameedtuple by function, that's why I very rarely do it. Another hting I don't like about namedtuples is the duplication of the name. TYpical declarations look like `Point = namedtuple('Point', ['x', 'y'])`, where `Point` is repeated two times. I'll go … [View More]one step further and say that the name is useless most of the time, so let's just get rid of it. Proposal ======== So I thought about a new (ok, maybe it has been proposed before but I couldn't find it) syntax for anonymous namedtuples (I put the prints as comments, otherwise gmane is complainig about top-posting): my_point = (x=12, y=16) # (x=12, y=16) my_point[0] # 12 my_point.y # 16 type(my_point) # <class 'anonymousnamedtuple'> It's just a tuple, but with names. Parenthesis would be mandatory because `my_point = x = 12, y = 16` wouldn't work. Single elements anonymouns namedtuples would require a trailing comma, similarely to tuples. I'd be happy to make a factory function for my personal use, but the order of kwargs is not respected. To have an elegant syntax, it has to be a construct of the language. The created objects would all be of the same class (with a better name, of course). As it adds no keyword to the language, it would not break compatibility. The created objects could support some namedtuples methods: _asdict, _replace and _fields. The two other methods _make and _source would not apply. Performance-wise, I guess that they would be slower than tuples and namedtuples, But to me the additionnal usability trumps the performance hit. If you have to care about performance, you can still use a namedtuple ot bare tuples. Even further... =============== This initial ideas is useful in itself, but it could be extended even further. I've thought about a possible evolution. The idea is that not all values has to be named: similarely to how args and kwargs work for functions, there could be non-named values, with the same limitations as the arguments (unnamed first, named afterwards). All values could be retrived by indexing, and named values could also be retrieved by their attribute name. This way they could be used in __getitem__ as proposed in PEP 472[0], so that __getitem__ supports additionnal keywords arguments. It would correspond to Strategy "named tuple", with some of the cons removed: "The namedtuple fields, and thus the type, will have to change according to the passed arguments. This can be a performance bottleneck, and makes it impossible to guarantee that two subsequent index accesses get the same Index class;" That would still be true for the performance bottleneck, but since the class would always be the same the second problem disappears. To minimize the performace hit, a standard tuple would be passed to __getitiem__ if there is no keyword argument, and an anonymous namedtuple would be passed if there is a keyword. "the _n "magic" fields are a bit unusual, but ipython already uses them for result history." Those wouldn't be needed if both named and unnamed values are allowed. "Differently from a function, the two notations gridValues[x=3, y=5, z=8] and gridValues[3,5,8] would not gracefully match if the order is modified at call time (e.g. we ask for gridValues[y=5, z=8, x=3]) . In a function, we can pre-define argument names so that keyword arguments are properly matched. Not so in __getitem__ , leaving the task for interpreting and matching to __getitem__ itself." Indexing already doesn't behave like a function call. Keeping the argument order (like this proposal would imply) is a special case of not keeping it, while the reverse is not true, so the more general solution would be implemented. Finally, I admit that I have no idea of how to implement this, I love python but never looked at it's internals. Maybe my proposal is too naive, I really hope not. Thanks for your attention, Joseph [0] https://www.python.org/dev/peps/pep-0472/ [View Less]

12 40

random.choice on non-sequence
by Rob Cliffe April 20, 2016

April 20, 2016

It surprised me a bit the first time I realised that random.choice did not work on a set. (One expects "everything" to "just work" in Python! :-) ) There is a simple workaround (convert the set to a tuple or list before passing it) but ... On 2011-06-23, Sven Marnach posted 'A few suggestions for the random module' on Python-Ideas, one of which was to allow random.choice to work on an arbitrary iterable. Raymond Hettinger commented "It could be generalized to arbitrary iterables (… [View More]Bentley provides an example of how to do this) but it is fragile (i.e. falls apart badly with weak random number generators) and doesn't correspond well with real use cases." Sven replied "Again, there definitely are real world use cases ..." I would like to revive this suggestion. I don't understand Raymond's comment. It seems to me that a reasonable implementation is better and more friendly than none, particularly for newbies. This is the source code for random.choice in Python 3.2 (as far as I know it hasn't changed since): def choice(self, seq): """Choose a random element from a non-empty sequence.""" try: i = self._randbelow(len(seq)) except ValueError: raise IndexError('Cannot choose from an empty sequence') return seq[i] And here is my proposed (tested) version: def choice(self, it): """Choose a random element from a non-empty iterable.""" try: it[0] except IndexError: raise IndexError('Cannot choose from an empty sequence') except TypeError: it = tuple(it) try: i = self._randbelow(len(it)) except ValueError: raise IndexError('Cannot choose from an empty iterable') return it[i] This works on (e.g.) a list/tuple/string, a set, a dictionary view or a generator. Obviously the generator has to be 'fresh' (i.e. any previously consumed values will be excluded from the choice) and 'throw-away' (random.choice will exhaust it). But it means you can write code like random.choice(x for x in xrange(10) if x%3) # this feels quite "Pythonic" to me! (I experimented with versions that, when passed an object that supported len() but not indexing, viz. a set, iterated through the object as far as necessary instead of converting it to a list. But I found that this was slower in practice as well as more complicated. There might be a memory issue with huge iterables, but we are no worse off using existing workarounds for those.) Downsides of proposed version: (1) Backward compatibility. Could mask an error if a "wrong" object, but one that happens to be an iterable, is passed to random.choice. There is also slightly different behaviour if a dictionary (D) is passed (an unusual thing to do): The current version will get a random integer in the range [0, len(D)), try to use that as a key of D, and raise KeyError if that fails. The proposed version behaves similarly except that it will always raise "KeyError: 0" if 0 is not a key of D. One could argue that this is an improvement, if it matters at all (error reproducibility). And code that deliberately exploits the existing behaviour, provided that it uses a dictionary whose keys are consecutive integers from 0 upwards, e.g. D = { 0 : 'zero', 1 : 'one', 2 : 'two', 3 : 'three', 4 : 'four' } d = random.choice(D) # equivalent, in this example, to "d = random.choice(list(D.values()))" if d == 'zero': ... will continue to work (although one might argue that it deserves not to). (2) Speed - the proposed version is almost certainly slightly slower when called with a sequence. For what it's worth, I tried to measure the difference on my several-years-old Windows PC, but is was hard to measure accurately, even over millions of iterations. All I can say is that it appeared to be a small fraction of a millisecond, possibly in the region of 50 nanoseconds, per call. Best wishes Rob Cliffe [View Less]

20 37

Changing the meaning of bool.__invert__
by Antoine Pitrou April 19, 2016

April 19, 2016

Hello, Booleans currently have reasonable overrides for the bitwise binary operators: >>> True | False True >>> True & False False >>> True ^ False True However, the same cannot be said of bitwise unary complement, which returns rather useless integer values: >>> ~False -1 >>> ~True -2 Numpy's boolean type does the more useful (and more expected) thing: >>> ~np.bool_(True) False How about changing the behaviour of bool.__invert__ … [View More]

25 55

Re: [Python-ideas] Have REPL print less by default
by Franklin? Lee April 19, 2016

April 19, 2016

On Apr 19, 2016 5:48 AM, "Paul Moore" <p.f.moore(a)gmail.com> wrote: > > You say "should"? Do you mean that it is likely, or do you mean that it is > > what would happen in an ideal world? My college had CS students SSH into the > > department's Linux server to compile and run their code, and many teachers > > don't believe that students should start with fancy IDE featues like, er, > > syntax highlighting. > > I mean that that is what I hear people on … [View More]lists like this saying as "a > reasonable beginner environment". It means that the people I've > introduced to Python (on Windows) have tended to end up using IDLE > (either from the start menu or via "edit with IDLE" from the right > click menu on a script). But are they perhaps tending to use IDLE because you were the one introducing them to it? Whether it is a recommended environment by the consensus of mailing lists is hardly indicative that IDLE is what people will start on. > My experience (in business, environments) is that people expect an IDE > when introduced to a new programming language, and IDLE, like it or > not, is what is available out of the box with Python. They expect it => they know what an IDE is. Programmers are already half-intermediate, especially programmers who use IDEs. > > You (and most regulars on this list) can adjust your shell to the way you > > like it, or use a more sophisticated shell, like IPython or bpython. On the > > other hand, changing shells and adding display hooks to site.py is not an > > option for those who don't know it's an option. > > As a "scripting expert" and consultant, I typically get asked to knock > up scripts on a variety of environments. I do not normally have what > I'd describe as "my shell", just a series of basic "out of the box" > prompts people expect me to work at. So no, the luxury of configuring > the default experience is *not* something I typically have. I suggested a command-line switch. > >> I view the REPL as more > >> important for intermediate or advanced users who want to quickly test > >> out an idea (at least, that's *my* usage of the REPL). > > > > But that doesn't answer my question: would the proposed change hurt your > > workflow? > > Yes. If I get a stack trace, I want it all. And if I print something > out, I want to see it by default. The REPL for me is an investigative > environment for seeing exactly what's going on. And why is it an insufficient option for you to type, for example, "_exfull()" to get the full trace? > (Also, having the REPL behave differently depending on what version of > Python I have would be a problem - backward compatibility applies here > as much as anywhere else). This is a concern, but not one that is enough, by itself, to justify the status quo. I disagree that it applies *anywhere near as much* here, since I don't see how it could break existing code. It will add a little adjustment and a little more work sometimes, but it doesn't require, say, refactoring a module, or every site page you ever downloaded. [View Less]

2 2

pep 7 line break suggestion differs from pep 8
by Joseph Jevnik April 18, 2016

April 18, 2016

I saw that there was recently a change to pep 8 to suggest adding a line break before a binary operator. Pep 7 suggests the opposite: > When you break a long expression at a binary operator, the operator goes at the end of the previous line, e.g.: > if (type->tp_dictoffset != 0 && base->tp_dictoffset == 0 && > type->tp_dictoffset == b_size && > (size_t)t_size == b_size + sizeof(PyObject *)) > return 0; /* "Forgive" adding a __dict__ … [View More]

3 2

Re: [Python-ideas] Add __main__ for uuid, random and urandom
by Wes Turner April 18, 2016

April 18, 2016

On Apr 18, 2016 12:30 PM, "Wes Turner" <wes.turner(a)gmail.com> wrote: > > > > I think the users of oneline.py could be people that now write lots of > > bash scripts and work on the command line. So whenever someone asks a > > question somewhere about how to do X on the linux command line, we > > might have the answer: """ > > > > Q: On the linux commandline, how do I get only the filename from a > > full path that is in $FILEPATH > … [View More]

1 1

The next major Python version will be Python 8
by Victor Stinner April 18, 2016

April 18, 2016

Hi, Python 3 becomes more and more popular and is close to a dangerous point where it can become popular that Python 2. The PSF decided that it's time to elaborate a new secret plan to ensure that Python users suffer again with a new major release breaking all their legacy code. The PSF is happy to announce that the new Python release will be Python 8! Why the version 8? It's just to be greater than Perl 6 and PHP 7, but it's also a mnemonic for PEP 8. By the way, each minor release will now … [View More]multiply the version by 2. With Python 8 released in 2016 and one release every two years, we will beat Firefox 44 in 2022 (Python 64) and Windows 2003 in 2032 (Python 2048). A major release requires a major change to justify a version bump: the new killer feature is that it's no longer possible to import a module which does not respect the PEP 8. It ensures that all your code is pure. Example: $ python8 -c 'import keyword' Lib/keyword.py:16:1: E122 continuation line missing indentation or outdented Lib/keyword.py:16:1: E265 block comment should start with '# ' Lib/keyword.py:50:1: E122 continuation line missing indentation or outdented (...) ImportError: no pep8, no glory Good news: since *no* module of the current standard library of Python 3 respect the PEP 8, the standard library will be simplified to one unique module, which is new in Python 8: pep8. The standard library will move to the Python Cheeseshop (PyPI), to reply to an old and popular request. DON'T PANIC! You are still able to import your legacy code into Python 8, you just have to rename all your modules to add a "_noqa" suffix to the filename. For example, rename utils.py to utils_noqa.py. A side effect is that you have to update all imports. For example, replace "import django" with "import django_noqa". After a study of the PSF, it's a best option to split again the Python community and make sure that all users are angry. The plan is that in 10 years, at least 50% of the 77,000 packages on the Python cheeseshop will be updated to get the "_noqa" tag. After 2020, the PSF will start to sponsor trolls to harass users of the legacy Python 3 to force them to migrate to Python 8. Python 8 is a work-in-progress (it's still an alpha version), the standard library was not removed yet. Hopefully, trying to import any module of the standard library fails. Don't hesitate to propose more ideas to make Python 8 more incompatible with Python 3! Note: The change is already effective in the default branch of Python: https://hg.python.org/cpython/rev/9aedec2dbc01 Have fun, Victor [View Less]

14 21

__getattr__ bouncer for modules
by Chris Angelico April 17, 2016

April 17, 2016

Every now and then there's been talk of making it easier to subclass modules, and the most common use case that I can remember hearing about is descriptor protocol requiring code on the type. (For instance, you can't change a module-level constant into a property without moving away from the default ModuleType.) How bad would it be for the default ModuleType to have a __getattr__ function which defers to the instance? Something like this: def __getattr__(self, name): if '__getattr__' in … [View More]

4 6

Re: [Python-ideas] Adding a pipe function to functools
by Wes Turner April 16, 2016

April 16, 2016

On Apr 16, 2016 12:07 AM, "Steven D'Aprano" <steve(a)pearwood.info> wrote: > > On Fri, Apr 15, 2016 at 01:51:45PM -0500, Wes Turner wrote: > > > * https://docs.python.org/3.1/howto/functional.html#the-functional-module > > > > from functional import compose, partial > > import functools > > multi_compose = partial(functools.reduce, compose) > > > > * https://docs.python.org/3.5/library/functools.html > > > > * … [View More]

1 0

Adding a pipe function to functools
by Ed Minnix April 16, 2016

April 16, 2016

Hello, I have been looking over the toolz library and one of the functions I like the most is pipe. Since most programmers are familiar with piping (via the Unix `|` symbol), and it can help make tighter code, I think it would be nice to add it to the standard library (such as functools). - Ed Minnix

5 6

Re: [Python-ideas] Welcome to the "Python-ideas" mailing list
by Richard Prosser April 16, 2016

April 16, 2016

Is there any mileage in having a naming convention to indicate the type of a variable? I have never really liked the fact that the Python 'duck typing' policy is so lax, yet the new "Type Hints" package for Python 3 is rather clumsy, IMO. For example: github_response = requests.get('https://api.github.com/user', auth=('user', 'pass')) # Derived from http://docs.python-requests.org/en/master. The above request returns a Response <http://docs.python-requests.org/en/master/api/#requests.… [View More]

5 4

Re: [Python-ideas] Adding a pipe function to functools
by Wes Turner April 15, 2016

April 15, 2016

"[Python-ideas] The pipe protocol, a convention for extensible method chaining" https://mail.python.org/pipermail/python-ideas/2015-May/033673.html https://groups.google.com/forum/m/#!topic/python-ideas/4HgpT5yE06o On Apr 15, 2016 9:51 AM, "Ed Minnix" <egregius313(a)gmail.com> wrote: Hello, I have been looking over the toolz library and one of the functions I like the most is pipe. Since most programmers are familiar with piping (via the Unix `|` symbol), and it can help make tighter … [View More]

1 0

Re: [Python-ideas] Module lifecycle: simple alternative to PEP 3121/PEP 489
by Nikita Nemkin April 15, 2016

April 15, 2016

On Thu, Apr 14, 2016 at 2:57 PM, Petr Viktorin <encukou(a)gmail.com> wrote: > On 04/14/2016 10:23 AM, Nikita Nemkin wrote: >> Reading PEP 3121/PEP 489 I can't stop wondering, why do extension >> modules require such specialized lifecycle APIs? Why not just let >> them subclass ModuleType? (Or any type, really, but ModuleType might >> be a good place to define some standard behavior.) > > Good question. > I'll list some assumptions; if you don't share … [View More]them we can talk more > about these: > - somethings are easy in Python, but not pleasant to do correctly* in C: > - subclassing > - creating simple objects > - setting attributes > - most modules don't need special functionality > > * ("Correctly" includes things like error checking, which many > third-party modules skimp on.) > > Most of the API is in the style of "leave this NULL unless you need > something special", i.e. simple things are easy, but the complex cases > are possible. Creating custom ModuleType subclasses is possible, but I > definitely wouldn't want to require every module author to do that. > > A lot of the API is convenience features, which are important: they do > the right thing (for example, w.r.t. error checking), and they're easier > to use than alternatives. This makes the API grows into several layers; > for example: > - m_methods in the PyModuleDef > - PyModule_AddFunctions > - PyCFunction_NewEx & PyObject_SetAttrString > Usually you just set the first, but if you need more control (e.g. > you're creating modules dynamically), you can use the lower-level tools. > > Your suggestion won't really help with this kind of complexity. I agree that C API is quite difficult and dangerous to use directly (that's why I always use Cython), but I don't agree that separate module init API makes things simpler. PyModuleDef appears to be PyTypeObject surrogate with similar (or identical?) semantics. That's an extra concept to learn about, an extra bit of documentation to consult when writing new code. PyTypeObject, on the other hand, is fundamental and unavoidable, regardless of module init system. Practically speaking, PyModuleDef_Init(&spam_def) and PyType_Ready(&spam_type) differ only in the number of zeroes in their respective static structs. There's also PyType_FromSpec (stable ABI), which might appeal to some people more than PyType_Ready. > Oh, and there is a technical reason against subclassing ModuleType > unless necessary: Custom ModuleType subclasses cannot be made to work > with runpy (i.e. python -m). For ModuleType (the ones without a custom > create_module in the current API), this doesn't *currently* work, but > the PEPs were written so that it's "just" a question of spending some > development effort on runpy. I didn't even know that python -m supported extension modules. >> Module instance naturally encapsulates C level module state. >> GC/finalization happens just like for any other object. PEP 3121 >> becomes redundant. > > It doesn't become fully redundant: m_methods is still useful. ModuleType subclasses have tp_methods. A little check in PyType_Ready can make it behave like PyModuleDef.m_methods. Modules don't need normal methods anyway. > With your suggestion, I fear that someone would quickly come up with a > macro to automate creating simple ModuleType instances, and at that > point the API would be as complex as it is now, but every module > instance would now also have an extra ModuleType subclass – and I don't > think that's either simpler or more effective. One or two standard macros that help with ModuleType subsclassing are in order. Something like PyModule_HEAD_INIT, whose usage is already obvious from the name. > I personally think that your suggestion wouldn't make the API > substantially simpler, assuming you would keep it as robust and easy to > use as the current solution. And if you would want to maintain backwards > compatibility (even with only the pre-PEP 489 state), it would be even > harder. > > Many people thought about the current APIs, and (almost) all of the > unpleasant decisions we had to make do have their reasons. (And if you > ask about a more specific decision, I can give you the specific reasons.) Regarding backward compatibility, PEP 3121 and most of PEP 489 will work just fine as a facade to ModuleType subclass. I understand that existing PEPs are well researched and there's no real incentive to change. My proposal is more of a hypothetical kind. [View Less]

4 8

PEP8 operator must come before line break
by SW April 15, 2016

April 15, 2016

Hi, PEP8 says that: "The preferred place to break around a binary operator is after the operator, not before it." This is ignored in the multiline if examples, and seems to generally be a bad idea as it negatively impacts clarity. For example, the following seems much clearer as the entire line does not need to be scanned to see the intent- only the start of the line is needed to see how the different properties are used for filtering: mylist = [ item for item in mylist if item['… [View More]

16 24

Module lifecycle: simple alternative to PEP 3121/PEP 489
by Nikita Nemkin April 14, 2016

April 14, 2016

Reading PEP 3121/PEP 489 I can't stop wondering, why do extension modules require such specialized lifecycle APIs? Why not just let them subclass ModuleType? (Or any type, really, but ModuleType might be a good place to define some standard behavior.) Module instance naturally encapsulates C level module state. GC/finalization happens just like for any other object. PEP 3121 becomes redundant. Two-step initialization (PEP 489) can be achieved by defining a new kind of PyInit_XXX entry point, … [View More]

2 1

Dictionary views are not entirely 'set like'
by Joshua Morton April 14, 2016

April 14, 2016

Let's start with a quick quiz: what is the result of each of the following (on Python3.5.x)? {}.keys() | set() # 1 set() | [] # 2 {}.keys() | [] # 3 set().union(set()) # 4 set().union([]) # 5 {}.keys().union(set()) # 6 If your answer was set([]), TypeError, set([]), set([]), set([]), AttributeError, then you were correct. That, to me, is incredibly unintuitive. Next up: {}.keys() == {}.keys() # 7 {}.items() == {}.items() # 8 {}.values() == {}.… [View More]values() # 9 d = {}; d.values() == d.values() # 10 True, True, False, False. Numbers 1, 2, 4, 5 are expected behavior. 3 and 6 are not, and 7-10 is up for debate.[1] First thing first, the behavior exhibited by #3 is a bug (or at least it probably should be, and one I'd be happy to fix. However, before doing that I felt it would be good to propose some questions and suggestions for how that might be done. There are, as far as I can tell, two reasons to use a MappingView, memory efficiency or auto-updating (a view will continue to mirror changes in the underlying object). I'll focus on the first because the second can conceivably be solved in other ways. Currently, if I want to union a dictionaries keys with a non-set iterable, I can do `{}.keys() | []`. This is a bug[2], the set or operator should only work on another set. That said, fixing this bug removes the ability to efficiently or keys with another iterable, `set({}.keys()).update([])` for efficiency, or `set({}.keys()).union([])` for clarity. Fixing this is simply a matter of adding a `.union` method to the KeysView (and possibly to the Set abc as well). Although that may not be something that is wanted. The issue, as far as I can tell, is whether we want people converting from MappingViews to "primitives" as soon as possible, or if we want to encourage people people to use the views for as long as possible. There are arguments on both sides: encouraging people to use the views causes these objects to become more complex, introducing more surface area for bugs, more code to be maintained, etc. Further, there's there is currently one obvious way to do things, convert to a primitive, if you're doing any complex action. On the other hand, making MappingViews more like the primitives they represent has positives for performance, simplifies user code, and would probably make testing these objects easier, since many tests could be stolen from test_set.py. My opinion is that the operators on MappingViews should be no more permissive than their primitive counterparts. A KeysView is in various ways more restrictive than a set, so having it be also occasionally less restrictive is surprising and in my opinion bad. This causes the loss of an efficient way to union a dict's keys with a list (among other methods). I'd then add .union, .intersection, etc. to remedy this. This solution would bring the existing objects more in line with their primitive counterparts, while still allowing efficient actions on large dictionaries. In short: - Is #3 intended behavior? - Should it (and the others be)? - As a related aside, should .union and other frozen ops be added to the Set interface? - If so, should the fix solely be a bugfix, should I do what I proposed, or something else entirely? - More generally, should there be a guiding principle when it comes to MappingViews and similar special case objects? [1]: There's some good conversation in this prior thread on this issue https://mail.python.org/pipermail/python-ideas/2015-December/037472.html. The consensus seemed to be that making ValuesViews comparable by value is technically infeasible (O(n^2) worst case), while making it comparable based on the underlying dictionary is a possibility. This would be for OrderedDict, although many of the same arguments apply for a normal dictionary. [2]: Well, it probably should be a bug, its explicitly tested for ( https://github.com/python/cpython/blob/master/Lib/test/test_dictviews.py#L1…), whereas sets are explicitly tested for the opposite functionality ( https://github.com/python/cpython/blob/master/Lib/test/test_set.py#L92) Thanks, I'm looking forward to the feedback, Josh [View Less]

7 11

Dunder method to make object str-like
by Chris Angelico April 13, 2016

April 13, 2016

This is a spin-off from the __fspath__ discussion on python-dev, in which a few people said that a more general approach would be better. Proposal: Objects should be allowed to declare that they are "string-like" by creating a dunder method (analogously to __index__ for integers) which implies a loss-less conversion to str. This could supersede the __fspath__ "give me the string for this path" protocol, or could stand in parallel with it. Obviously str will have this dunder method, returning … [View More]

22 91

Modifying yield from's return value
by Bar Harel April 12, 2016

April 12, 2016

I asked a question in stackoverflow <http://stackoverflow.com/q/36582679/1658617> regarding a way of modifying yield from's return value. There doesn't seem to be any way of modifying the data yielded by the yield from expression without breaking the yield from "pipe". By "pipe" I mean the fact that .send() and .throw() pass to the inner generator. Useful cases are parsers as I've demonstrated in the question, transcoders, encoding and decoding the yielded values without interrupting .… [View More]

2 3

(no subject)
by Keith Curtis April 12, 2016

April 12, 2016

Hello again, Many FOSS communities struggle with sustainability, but Python is very rich so you've clearly figured it out. It is easy to say that bug metrics are going in the wrong direction. It is also easy to do nothing, or to criticize those who are concerned about a systemic issue. Data can be turned into action by leadership and plans. It can be helpful to have people who think 5,000 is a big number. (Automatically closing old issues doesn't take grit and isn't a valid solution.) It … [View More]

8 7

A tuple of various Python suggestions
by Keith Curtis April 11, 2016

April 11, 2016

Hi all, I just discovered this alias and I thought I'd post a few ideas. I wouldn't call myself a Python master yet, but it's an amazing language and my biggest wish is that it was more widely used in the industry. Here are a few suggestions: 1. Decrease the bug count. I recently noticed that there are about 5,400 active bugs in http://bugs.python.org/. That surprised me because I almost never see anyone complain about bugs in Python (compared to the number who complain about bugs in … [View More]LibreOffice, graphics drivers, Gnome / KDE, etc.) There are a lot of people on this list, and if the brainpower can be applied to practical, known, existing problems, it is a great way to improve Python while also considering more exotic ideas. I can also suggest making a pretty graph of the bug count and putting it on the front page of python.org for greater visibility. 2. Python is somewhat popular on servers, and there is a lot of potential for more. WordPress is easy to use and powerful, but lots of people don't want to program in PHP. Or Javascript, Java, Ruby, etc. Codebases like Whoosh full-text search (https://bitbucket.org/mchaput/whoosh/wiki/Home) are important, but have minimal dev resources as most people are using Lucene / ElasticSearch. The common choice is between 1.3M lines of Java: https://www.openhub.net/p/elasticsearch, containing 1100 todos and 1000 references to "deprecated", or 41K lines of Python written mostly by one person. Hadoop is another big Java project (1.9M lines), and there is even an ecosystem around it. Python interoperates with Hadoop, but it should be possible to build a radically simpler framework that provides the same functionality using Python-native functionality and without all the baggage. Hadoop has several interesting sister projects: a distributed database, scalable machine learning, a high-level data flow language, a coordination service, etc. I'm sure you'd build something smaller, cleaner, faster in many cases, more reliable, etc. 3. It was a sad mistake that Google picked Java over Python for Android. However, there is now a great program called Kivy which allows people to write apps for IOS or Android with one codebase, but it could also use more resources, as for example it doesn't fully support Python 3.x yet. There are 10s of thousands of bugs in the popular Python libraries and I would fix those before proposing more language changes. 4. I enjoy reading about the Python performance improvements, but it is mostly a perception problem with all the existing workarounds. Gnome wrote version 3 of their shell in Javascript because they didn't think Python would be fast enough. Lots of people write Node because it's compiled and "fast". I suggest taking some of the effort working on performance, and spend it on evangelizing to other programmers that Python / Cython / PyPy, etc. are already good enough! There are a lot of programmers out there who would be happier if they could work in Python. 5. It would be great to get Python in the web browsers as an alternative to Javascript. There are a number of projects which convert Python to Javascript, but this would be more direct. LibreOffice ships with a Python interpreter, why can't Firefox and Webkit? ;-) Obviously there are interoperability issues, but it would be great to just side-step all the complexity of Javascript (Here is a server-oriented article, but it gives a flavor: http://notes.ericjiang.com/posts/751) This might sound like a crazy idea, but the engineering problems aren't that hard. 6. In a few cases, there are two many codebases providing the same functionality, and none of them are really doing the job. For example, the de-facto MySQL Python interop library (https://pypi.python.org/pypi/MySQL-python) only supports Python 2.x and appears to be abandoned. There are several other libraries out there with different features, performance, compatibility, etc. and it's kind of a minefield for what should be a basic scenario. It takes leadership to jump in and figure out how to fix it and make one codebase good enough that the others can switch over. 7. Focus more on evolving the libraries rather than the language. I've recently discovered Toolz, which has a more complete set of functional language methods. I think some of them should be included in the official versions. A lot of people don't think Python is good enough for functional programming and this would help. These new routines add complexity, but a newbie doesn't need to write in a functional way, so it obeys the "only pay for what you use" rule. There are a number of under-staffed libraries and frameworks. I see people complain about the YAML parsing library being unmaintained, the default HTTP functionality being difficult and limiting, poor SOAP support, etc. There are a million ways to improve the Python ecosystem without making any language changes. You don't have a big rich company who can pay for thousands of full-time developers working on libraries, but the bug reports are a great way to prioritize. 8. I've yet to find a nice simple free IDE with debugging for Python. I use Atom, but it has primitive debugging. I tried PyCharm but it's very complicated (and not free, and Java). I use Jupyter sometimes also but I'd prefer a rich client app with watch windows, etc. 9. It would be interesting to re-imagine the spreadsheet with a more native Python interface. Pandas and matplotlib are great, but it would be cool to have it in LibreOffice Calc that supports drag and drop, copy and paste, can read and write ODS, etc. (Also, LibreOffice Base is basically unmaintained. I think if 10 Python programmers passionate about databases and GUIs showed up, it could re-invigorate this dead codebase.) 10. Being simple to learn and powerful is very hard. Fortunately, you can break compatibility every 10 years. My only suggestion is to get rid of the __self__ somehow ;-) Regards, -Keith http://keithcu.com/ [View Less]

9 15

simpler weakref.finalize
by Matthew Stoltenberg April 11, 2016

April 11, 2016

Currently, if I want to have weakref.finalize call an object's cleanup method, I have to do something like: class foo: _finalizer = None def __init__(self): self._finalizer = weakref.finalize(self, foo._cleanup, weakref.WeakMethod(self.cleanup)) def __del__(self): self.cleanup() @classmethod def _cleanup(cls, func): func()() def cleanup(self): if self._finalizer is not None: … [View More]

1 0

Re: [Python-ideas] Consistent programming error handling idiom
by Rian Hunter April 11, 2016

April 11, 2016

>> On Fri, Apr 08, 2016 at 11:24:15AM -0700, rian(a)thelig.ht wrote: >> I want to live in a world where I can do this: >> >> while cbs: >> cb = cbs.pop() >> try: >> cb() >> except Exception as e: >> logging.exception("In main loop") >> if is_a_bug(e): >> raise SystemExit() from e > > And I want a pony :-) > > So long as Python allows people to write … [View More]

5 5

Boundaries for unpacking
by Michel Desmoulin April 10, 2016

April 10, 2016

Python is a lot about iteration, and I often have to get values from an iterable. For this, unpacking is fantastic: a, b, c = iterable One that problem arises is that you don't know when iterable will contain 3 items. In that case, this beautiful code becomes: iterator = iter(iterable) a = next(iterator, "default value") b = next(iterator, "default value") c = next(iterator, "default value") More often than not, I wish there were a way to specify a default value for unpacking. This would … [View More]

6 8

Hijacking threads [was: Changing the meaning of bool.__invert__]
by Ethan Furman April 9, 2016

April 9, 2016

In the recent thread about changing the meaning of __invert__ on bools I was accused of an attempted hijack. Can anybody please enlighten me as to what, exactly, I did wrong? Original post follows: > I think the str() of a value, while possibly being the most > interesting piece of information (IntEnum, anyone?), is hardly the > most intrinsic. > > If we do make this change, besides needing a couple major versions to > make it happen, will anything else be different? … [View More]

6 7

New scope for exception handlers
by Joseph Jevnik April 9, 2016

April 9, 2016

I would like to propose a change to exception handlers to make it harder to accidently leak names defined only in the exception handler blocks. This change follows from the decision to delete the name of an exception at the end of a handler. The goal of this change is to prevent people from relying on names that are defined only in a handler. As an example, let's looks at a function with a try except: def f(): try: ... except: a = 1 return a This function will … [View More]only work if the body raises some exception, otherwise we will get an UnBoundLocalError. I propose that we should make `a` fall out of scope when we leave the except handler regardless to prevent people from depending on this behavior. This will make it easier to catch bugs like this in testing. There is one case where I think the name should not fall out of scope, and that is when the name is already defined outside of the handler. For example: def g(): a = 1 try: ... except: a = 2 return a I think this code is well behaved and should continue to work as it already does. There are a couple of ways to implment this new behavior but I think the simplest way to do this would be to treat the handler as a closure where all the free variables defined as nonlocal. This would need to be a small change to the compiler but may have some performance implications for code that does not hit the except handler. If the handler is longer than the bytecode needed to create the inner closure then it may be faster to run the function when the except handler is not hit. This changes our definition of f from: 2 0 SETUP_EXCEPT 8 (to 11) 3 3 LOAD_CONST 1 (Ellipsis) 6 POP_TOP 7 POP_BLOCK 8 JUMP_FORWARD 14 (to 25) 4 >> 11 POP_TOP 12 POP_TOP 13 POP_TOP 5 14 LOAD_CONST 2 (1) 17 STORE_FAST 0 (a) 20 POP_EXCEPT 21 JUMP_FORWARD 1 (to 25) 24 END_FINALLY 6 >> 25 LOAD_FAST 0 (a) 28 RETURN_VALUE to something more like: f - 3 0 SETUP_EXCEPT 8 (to 11) 4 3 LOAD_CONST 0 (Ellipsis) 6 POP_TOP 7 POP_BLOCK 8 JUMP_FORWARD 20 (to 31) 5 >> 11 POP_TOP 12 POP_TOP 13 POP_TOP 14 LOAD_CONST 1 (<code object <excepthandler> at 0x7febcd6e2300, file "<code>", line 1>) 17 LOAD_CONST 2 ('<excepthandler>') 20 MAKE_FUNCTION 0 23 CALL_FUNCTION 0 (0 positional, 0 keyword pair) 26 POP_EXCEPT 27 JUMP_FORWARD 1 (to 31) 30 END_FINALLY 7 >> 31 LOAD_FAST 0 (a) 34 RETURN_VALUE f.<excepthandler> ----------------- 1 0 LOAD_CONST 0 (1) 3 STORE_FAST 0 (a) 6 LOAD_CONST 1 (None) 9 RETURN_VALUE This new code properly raises the unbound locals exception when executed. For g we could use a MAKE_CLOSURE instead of MAKE_FUNCTION. [View Less]

5 5

Re: [Python-ideas] Consistent programming error handling idiom
by rian＠thelig.ht April 8, 2016

April 8, 2016

>> Oops, good call. That was a bad example. Maybe this one is a bit >> better: >> >> try: >> assert no_bugs() >> except AssertionError: >> # bugs are okay >> pass > > Not sure I understand this example. It's obviously a toy, because > you'd never put an 'assert' just to catch and discard the exception - > the net result is that you call no_bugs() if you're in debug mode and > don't if you're not, which … [View More]is more cleanly spelled "if __debug__: > no_bugs()". The example is removed from its original context. The point was to show that there is no reason to locally catch AssertionError (contrast with AttributeError, ValueError, TypeError, etc.). AssertionError is *always* indicative of a bug (unlike AttributeError which only may be indicative of a bug). >>> This is what I'd call a boundary location. You have "outer" code and >>> "inner" code. Any uncaught exception in the inner code should get >>> logged rather than aborting the outer code. >> >> I'm very familiar with this pattern in Python and I've used it myself >> countless times. Unfortunately I've seen instances where it can lead >> to disastrous behavior, I think we all have. It seems to have limited >> usefulness in production and more usefulness in development. The point >> of my original email was precisely to put the legitimacy and >> usefulness of code like that into question. >> > > When does this lead to disaster? Is it because someone creates a > boundary that shouldn't exist, or because the code maintains global > state in bad ways? The latter is a problem even without exceptions; > imagine a programming error that doesn't cause an exception, but just > omits some crucial "un-modify global state" call. You have no way of > detecting that in the outer code, and your state is messed up. This leads to disaster in the case of buggy code. This leads to a greater disaster when the boundary allows the code to continue running. It's not a black and white thing, there is no 100% way to detect for buggy code but my argument is that you shouldn't ignore an assert when you're lucky enough to get one. >>> As to resetting stuff: I wouldn't bother; your functions should >>> already not mess with global state. The only potential mess you >>> should >>> consider dealing with is a database rollback; and actually, my >>> personal recommendation is to do that with a context manager inside >>> the inner code, rather than a reset in the exception handler in the >>> outer code. >> >> So I agree this pattern works if you assume all code is exception-safe >> (context managers will clean up intermediate state) and there are no >> programming errors. There are lots of things that good code should do >> but as we all well know good code doesn't always do those things. My >> point is that except-log loops are dangerous and careless in the face >> of programming errors. >> > > I don't think the except-log loop is the problem here. The problem is > the code that can go in part way, come out again, and leave itself in > a mess. I'm not saying except-log is to blame for buggy code. I'm saying that when an exception occurs, there's no way to tell whether it was caused by an internal bug or an external error. Except-log should normally be limited to exceptions caused by external errors in production code. >> Programming errors are unavoidable. In a large system they are a daily >> fact of life. When there is a bug in production it's very dangerous to >> re-enter a code block that has demonstrated itself to be buggy, else >> you risk corrupting data. For example: >> >> def random_code(state): >> assert is_valid(state.data) >> >> def user_code_a(state): >> state.data = "bad data" >> # the following throws >> random_code(state) >> >> def user_code_b(state): >> state.db.write(state.data) >> >> def main_loop(): >> state = State() >> loop = [user_code_a, >> user_code_b] >> for fn in loop: >> try: >> fn() >> except Exception: >> log_exception() >> >> This code allows user_code_b() to execute and corrupt data even though >> random_code() was lucky enough to be called and detect bad state early >> on. > > Can you give a non-toy example that has this kind of mutable state at > top level? I suspect it's bad design. If it's truly necessary, use a > context manager to guarantee the reset: > > def user_code_a(state): > with state.set_data("bad data"): > random_code(state) > Yes it may be bad design or the code may have bugs but that's precisely the point. By the time the exception hits your catch-all, there's no universal way of determining whether or not the exception was due to an internal bug (or bad design or whatever) or an external error. >> You may say the fix is to assert correct data before writing to the >> database and, yes, that would fix the problem for future executions in >> this instance. That's not the point, the point is that incorrect buggy >> code is running in production today and it's imperative to have >> multiple safeguards to limit its damage. For example, you wouldn't >> have an except-log loop in an airplane control system. >> > > Actually, yes I would. The alternative that you're suggesting is to > have any error immediately shut down the whole system. Is that really > better? To have the entire control system disabled? The alternative I'm suggesting is to reset that flight computer and switch to the backup one (potentially written by a different team). >> Sometimes an error is just an error but sometimes an error signifies >> the running system itself is in a bad state. It would be nice to >> distinguish between the two in a consistent way across all Python >> event loops. Halting on any escaped exception is inconvenient, but >> continuing after any escape exception is dangerous. >> > > There's no way for Python to be able to fix this for you. The tools > exist - most notably context managers - so the solution is to use > them. Context managers don't allow me to determine whether an exception is caused by an internal bug or external error, so it's not a solution to this problem. I want to live in a world where I can do this: while cbs: cb = cbs.pop() try: cb() except Exception as e: logging.exception("In main loop") if is_a_bug(e): raise SystemExit() from e Python may be able to do something here. One possible thing is a new exception hierarchy, there may be other solutions. This may be sufficient but I doubt it: def is_a_bug(e): return isinstance(e, AssertionError) The reason I am polling python-ideas is that I can't be the only one who has ever encountered this deficiency in Python. Rian [View Less]

7 11

Re: [Python-ideas] Consistent programming error handling idiom
by Rian Hunter April 8, 2016

April 8, 2016

> From: Chris Angelico <rosuav(a)gmail.com> >> On Fri, Apr 8, 2016 at 12:17 PM, Rian Hunter <rian(a)thelig.ht> wrote: >> Contrast with the following code that never makes sense (and is why I >> said that NameError definitely signifies a programming error): >> >> try: >> foo = bar >> except NameError: >> foo = 0 > > This is exactly the idiom used to cope with builtins that may or may > not exist. If you … [View More]want to support Python 2 as well as 3, you might use > something like this: > > try: > input = raw_input > except NameError: > raw_input = input Oops, good call. That was a bad example. Maybe this one is a bit better: try: assert no_bugs() except AssertionError: # bugs are okay pass > This is what I'd call a boundary location. You have "outer" code and > "inner" code. Any uncaught exception in the inner code should get > logged rather than aborting the outer code. I'd spell it like this: > > try: > response = ... > except BaseException as e: > logging.exception(...) > response = create_500_response() > > [snip] > > At this kind of boundary, you basically catch-and-log *all* > exceptions, handling them the same way. Doesn't matter whether it's > ValueError, SyntaxError, NameError, RuntimeError, GeneratorStop, or > SystemExit - they all get logged, the client gets a 500, and you go > back and look for another request. I'm very familiar with this pattern in Python and I've used it myself countless times. Unfortunately I've seen instances where it can lead to disastrous behavior, I think we all have. It seems to have limited usefulness in production and more usefulness in development. The point of my original email was precisely to put the legitimacy and usefulness of code like that into question. > As to resetting stuff: I wouldn't bother; your functions should > already not mess with global state. The only potential mess you should > consider dealing with is a database rollback; and actually, my > personal recommendation is to do that with a context manager inside > the inner code, rather than a reset in the exception handler in the > outer code. So I agree this pattern works if you assume all code is exception-safe (context managers will clean up intermediate state) and there are no programming errors. There are lots of things that good code should do but as we all well know good code doesn't always do those things. My point is that except-log loops are dangerous and careless in the face of programming errors. Programming errors are unavoidable. In a large system they are a daily fact of life. When there is a bug in production it's very dangerous to re-enter a code block that has demonstrated itself to be buggy, else you risk corrupting data. For example: def random_code(state): assert is_valid(state.data) def user_code_a(state): state.data = "bad data" # the following throws random_code(state) def user_code_b(state): state.db.write(state.data) def main_loop(): state = State() loop = [user_code_a, user_code_b] for fn in loop: try: fn() except Exception: log_exception() This code allows user_code_b() to execute and corrupt data even though random_code() was lucky enough to be called and detect bad state early on. You may say the fix is to assert correct data before writing to the database and, yes, that would fix the problem for future executions in this instance. That's not the point, the point is that incorrect buggy code is running in production today and it's imperative to have multiple safeguards to limit its damage. For example, you wouldn't have an except-log loop in an airplane control system. Sometimes an error is just an error but sometimes an error signifies the running system itself is in a bad state. It would be nice to distinguish between the two in a consistent way across all Python event loops. Halting on any escaped exception is inconvenient, but continuing after any escape exception is dangerous. Rian [View Less]

2 1

Consistent programming error handling idiom
by Rian Hunter April 8, 2016

April 8, 2016

I like that in Python all errors are exceptions. This enables elegant code and provides a default error idiom for all Python code. An important distinction between exceptions arises when handling an exception in a top-level exception handler that doesn't have the context to properly handle it. In these situations some exceptions can be ignored (and hopefully logged) and some exceptions should the terminate/reset the program (not necessarily literally). I'd call exceptions that should … [View More]terminate the program "programming errors” or “bugs.” It's practically impossible to gracefully recover from a programming error when encountered (you can potentially hot reload a bug fix but I digress). Examples of current exceptions that definitely represent programming errors include AssertionError, NameError and SyntaxError. In the case of AssertionError, it's very important the program terminates or the state is reset lest you run the risk of corrupting persistent data. I'd argue that other exceptions like AttributeError, ValueError, and TypeError can also represent programming errors depending on where they are caught. If one of these exceptions is caught near the top of the call stack where nothing useful can be done it's very likely a programming error. If it's caught locally where it can be predictably handled, no hard reset is necessary, e.g.: try: foo = val.bar except AttributeError: foo = 0 Contrast with the following code that never makes sense (and is why I said that NameError definitely signifies a programming error): try: foo = bar except NameError: foo = 0 Toy examples aside, this problem arises in real programs, like an extensible HTTP server: try: response = client_request_handler(global_state, connection_state, request) except Exception: response = create_500_response() # TODO: should the server be reset? # is global_state invalid? is connection_state invalid? In this case I think it’s polite to always send a “500” error (the HTTP status code for “internal server error”) but the question remains as to whether or not the server should reset its global state or close the connection. Something smells here and I don’t think Python currently has a solution to this problem. I don’t think it should be ambiguous by the time an exception is caught whether or not the program has a bug in it or whether it has simply run into an external error. I don’t know what the solution to this should be. I’ve seen proposals like a new exception hierarchy for “bad code” but I don’t think a good solution necessarily requires changes to the language. I think the solution could be as simple as a universally accepted PEP (like PEP 8) that discusses some of these issues, and related ones, and presents correct / pythonic ways of dealing with them. Maybe the answer is recommending that top-level exception handlers should only be used with extreme care and, unless you know what you’re doing, it’s best to let your program die (or affected state reset) and bias towards more fine-grained exception handling. Thoughts? Thanks for reading. Rian [View Less]

2 1

Re: [Python-ideas] Dunder method to make object str-like
by Nathaniel Smith April 7, 2016

April 7, 2016

Why strings and not, say, floats or bools or lists or bytes? Is it because strings have a uniquely compelling use case, or...? (For context from the numpy side of things, since IIUC numpy's fixed-width integer types were the impetus for __index__: numpy actually has a shadow type system that has versions of all of those types above except list. It also has a general conversion/casting system with a concept of different levels of safety, so for us __index__ is like a "safe" cast to int, and … [View More]__int__ is like a "unsafe", and we have versions of this for all pairs of internal types, which if you care about this in general then having a single parametrized concept of safe casting might make more sense then adding new methods one by one. I'm not aware of any particular pain points triggered by numpy's strings not being real strings, though, at least in numpy's current design.) -n On Apr 7, 2016 6:05 AM, "Chris Angelico" <rosuav(a)gmail.com> wrote: This is a spin-off from the __fspath__ discussion on python-dev, in which a few people said that a more general approach would be better. Proposal: Objects should be allowed to declare that they are "string-like" by creating a dunder method (analogously to __index__ for integers) which implies a loss-less conversion to str. This could supersede the __fspath__ "give me the string for this path" protocol, or could stand in parallel with it. Obviously str will have this dunder method, returning self. Most other core types (notably 'object') will not define it. Absence of this method implies that the object cannot be treated as a string. String methods will be defined as accepting string-like objects. For instance, "hello"+foo will succeed if foo is string-like. Downside: Two string-like objects may behave unexpectedly - foo+bar will concatenate strings if either is an actual string, but if both are other string-like objects, depends on the implementation of those objects. Bikeshedding: 1) What should the dunder method be named? __str_coerce__? __stringlike__? 2) Which standard types are sufficiently string-like to be used thus? 3) Should there be a bytes equivalent? 4) Should there be a format string "coerce to str"? "{}".format(x) is equivalent to str(x), but it might be nice to be able to assert that something's stringish already. Open season! _______________________________________________ Python-ideas mailing list Python-ideas(a)python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ [View Less]

2 1

unpacking in index expression
by Alexander Heger April 7, 2016

April 7, 2016

I was trying to unpack an array or tuple in index expression (python 3.5.1) but this failed. I think for consistency of language use it would be desirable to allow that as well. Below an example of my attempt indexing a numpy array, x: >>> import numpy as np >>> x = np.arange(12).reshape(3,-1) >>> x[*[0,1]] x[*[0,1]] ^ SyntaxError: invalid syntax >>> x[*(0,1)] x[*(0,1)] ^ SyntaxError: invalid syntax for the case of numpy, the latter … [View More]

2 1

Making pathlib paths inherit from str
by Koos Zevenhoven April 6, 2016

April 6, 2016

Hi all, Recently, I have spent quite a lot of time thinking about what should be done to improve the situation of pathlib. Last week, after all kinds of wild ideas and trying hard every night to prove to myself that duck-typing-like compatibility with other path-representing objects is sufficient, I noticed that I had failed to prove that and that I had gone around a full circle back to where I started: "why can't paths subclass ``str``?" In the "Working with Path objects: p-strings?" thread, … [View More]I said I was working on a proposal. Since it's been several days already, I think i should post it here and get some feedback before going any further. Maybe I should have done that even earlier. Anyway, there are some rough edges, and I will need to add links to references etc. So, do not hesitate to give feedback or criticism, which is especially appreciated it you take the time to read through the whole thing first :). You can read a github-rendered version here: https://github.com/k7hoven/strpathlib/blob/master/proposal.rst And the raw rst is here below: Proposal: Making path objects inherit from ``str``. =================================================== Abstract -------- This proposal addresses issues that limit the usability of the object-oriented filesystem path objects, most notably those of ``pathlib``, introduced in the standard library in Python 3.4 via PEP 0428. One issue being that the path objects are not directly compatible with nearly any libraries, including the standard library. A further goal of this proposal is to provide a smooth transition into a Python with better Path handling, while keeping backwards compatiblity concerns to a minimum. The approach involves making the path classes in ``pathlib`` (and optionally also DirEntry) subclasses of ``str``, but takes further measures to avoid problems and unnecessary additions. Introduction ------------ Filesystem paths are strings that give instructions for traversing a directory tree. In Python, they have traditionally been represented as byte strings, and more recently, unicode string. However, Python now has ``pathlib`` in the standard library, which is an object-oriented library for dealing with objects specialized in representing a path and working with it. In this proposal, such objects are generally referred to as *path objects*, or sometimes, in the specific context of instances of the ``pathlib`` path classes, they are explicitly referred to as ``pathlib`` objects. In ``pathlib`` there is a hierarchy of path classes with a common base class ``PurePath``. It has a subclass ``Path`` which essentially assumes the path is intended to represent a path on the current system. However, both of these classes, when called, instantiate a subclass of the ``Windows`` or ``Posix`` flavor, which have slightly different behavior. In total, there are thus five public classes: ``PurePath``, ``PurePosixPath``, ``PureWindowsPath``, ``Path``, ``PosixPath`` and ``WindowsPath``. Since Python 3.5 and the introduction of ``os.scandir``, the family of path classes has a new member, ``DirEntry``, which is a performance-oriented path object with significant duck-typing compatibility with ``pathlib`` objects. The adoption of the different types of path objects is still quite low, which is perhaps unsurprising, because they were only introduced very recently. However, it can also be inconvenient to work with these objects, because, they usually need to be explicitly converted into strings before passing them to functions, and path strings returned by functions need to be explicitly converted into path objects. Especially the latter issue is difficult in terms of backwards compatibility of APIs. While many things were recently discussed on Python ideas regarding the future of path-like objects, this proposal has a much more limited scope, to provide first steps in the right direction. However, the last part of this proposal considers possible future directions that this may optionally lead to. Rationale --------- Filesystem paths (or comparable things like URIs) are strings of characters that represent information needed to access a file or directory (or other resource). In other words, they form a subset of strings, involving specialized functionality such as joining absolute and relative paths together, accessing different parts of the path or file name, and even accessing the resources the path points to. In Python terms, for a path ``path``, one would have ``isinstance(path, str)``. It is also clear that not all strings are paths. On the one hand, this would make an ideal case for making all path-representing objects inherit from ``str``; while Python tries not to over-emphasize object-oriented programming and inheritance, it should not try to avoid class hierarchies when they are appropriate in terms of both purity and practicality. Regarding practicality, making specialized *path objects* also instances of ``str`` would make almost any stdlib or third-party function accept path objects as path arguments, assuming that they accept any instance of ``str``. Furthermore, functions now returning instances of ``str`` to represent paths could in future versions return path objects, with only minor backwards-incompatibility worries. On the other hand, strings are a very general concept, and the Python ``str`` class provides a large variety of methods to manipulate and work with them, including ``.split()``, ``.find()``, ``.isnumeric()`` and ``.join()``. These operations may be defined just as well for a string that represents a path than for any other string. In fact, this is the status quo in Python, as the adoption of ``pathlib`` is still quite limited and paths are in most cases represented as strings (sometimes byte strings). But while the string operations are *defined* on path-representing strings, the results of these operations may not be of any use in most cases, even if in some cases, they may be. While it is not the responsibility of the programming language to prevent doing things that are not useful, it may be practical in some cases. For instance, the string method ``.find()`` could be mistaken to mean finding files on the file system, while it in fact searches for a substring. String concatenation, in turn, can be a perfectly reasonable thing to do: ``show_msg("Data saved to file: " + file_path)``. The result of the concatenation of a string and a path is not a path, but a general string. Directly concatenating two path objects together as strings, however, likely has no sensible use cases. There is prior art in subclassing the Python ``str`` type to build a path object type. Packages on PyPI (TODO: list more?) that do this include ``path.py`` and ``antipathy``. The latter also supports ``bytes``-based paths by instantiating a different class, a subclass of ``bytes``. Since these libraries have existed for several years, experience from them is available for evaluating the potential benefits and weaknesses of this proposal (as well as other aspects regarding ``pathlib``). However, this proposal goes a step further to avoid potential problems and to provide a smooth transition plan that, if desired, can be followed to painlessly move towards a Python with a clear distinction between strings and paths. An optional long-term goal that this proposal facilitates may be to gradually move away from using strings (or even their subclasses) as paths. Specification of standard library changes ----------------------------------------- Making ``pathlib`` classes subclasses of ``str`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Assuming the present class hierarchy in ``pathlib``, inheritance from ``str`` will be introduced by making the base class ``pathlib.PurePath`` a subclass of ``str``. Methods will further be overridden in ``PurePath`` as described in the following. Overriding all ``str``-specific methods ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Since most of the ``str`` methods are not of any use on paths and can be confusing, leading to undesired behavior, *most* ``str`` methods (including magic methods, but excluding methods listed below) are overridden in ``PurePath`` with methods that by default raise ``TypeError("str method '<name>' is not available for paths."``. This will help programmers to immediately notice when they are using the wrong method. The perhaps unusual practice of disabling most base-class methods can be regarded as being conservative in adding ``str`` functionality to path objects. All methods, including double-underscore methods are overridden, except for the following, which are *not* overridden: - Methods of the ``str`` or ``object`` types that are already overridden by ``PurePath`` - Methods of the ``object`` type that are not overridden by ``str`` - ``__getattribute__`` - ``__len__`` (this could be debated, but not having it might be weird for a str instance) - ``encode`` - ``startswith`` and ``endswith`` (TODO: override these with case-insensitive behavior on the windows flavor) - ``__add__`` will be overriden separately, as described in later subsections. This will allow ``open(...)`` as well as most ``os`` and ``os.path`` functionality to work immediately, although there are cases that need special handling. Later, if shown to be desirable, some additional string methods may be enabled on paths. Overriding ``.__add__`` to disable adding two path objects together ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Overloading of the ``+`` operator in ``str`` will be overridden with a version which disables concatenation of two path objects together while allowing other type combinations (TODO: consider also fully disabling +): .. code:: python def __add__(self, other): if isinstance(other, PurePath): raise TypeError("Operator + for two paths is not defined; use / for joining paths.") return str.__add__(self, other) Optional enabling of string methods ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Since many APIs currently have functions or methods that return paths as strings, existing code may expect to have all string functionality available on the returned objects. While most users are unlikely to use much of the ``str`` functionality, a library function may want to explicitly allow these operations on a path object that it returns. Therefore, the overridden ``str`` methods can be enabled by setting a ``._enable_str_functionality`` method on a path object as follows: - ``pathobj._enable_str_functionality = True #`` -- Enable ``str`` methods - ``pathobj._enable_str_functionality = 'warn' #`` -- Enable ``str`` methods, but emit a ``FutureWarning`` with the message ``"str method '<name>' may be disabled on paths in future versions."`` The warning will help the API users notice that the return value is no longer a plain path. .. code:: python def <name>(self, *args, **kwargs): """Method of str, not for use with pathlib path objects.""" try: enable = self._enable_str_functionality except AttributeError: raise TypeError("str method '{}' is not available for paths." .format('<name>')) from None if enable == 'warn': warnings.warn("str method '{}' may be disabled on paths in future versions." .format('<name>'), FutureWarning, stacklevel = 2) elif enable is True: pass else: raise ValueError("_enable_str_functionality can be True or 'warn'") return getattr(str, name)(self, *args, **kwargs) New APIs, however, do not need to enable ``str`` functionality and may return default path objects. Helping interactive python tools and IDEs ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Interactive Python tools such as Jupyter are growing in popularity. When they use ``dir(...)`` to give suggestions for code completion, it is harmful to have all the disabled ``str`` methods show up in the list, even if they typically would raise exceptions. Therefore, the ``__dir__`` method should be overridden on ``PurePath`` to only show the methods that are meaningful for paths. Some tools used for code completion, such as ``rope`` and ``jedi`` may need some changes for optimal code completion. This in fact includes also the standard Python REPL, which currently does not respect ``__dir__`` in tab completion. Changes needed to other stdlib modules ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ In stdlib modules other than ``pathlib``, mainly ``os``, ``ntpath`` and ``posixpath``, The stdlib functions in modules that use the methods/functionality listed below on path or file names, will be modified to explicitly convert the name ``name`` to a plain string first, e.g., using ``getattr(name, 'path', name)``, which also works for ``DirEntry`` but may return ``bytes``: - ``split`` - ``find`` - ``rfind`` - ``partition`` - ``__iter__`` - ``__getitem__`` (However, if ``DirEntry`` is not made to subclass ``str``, the idiom ``getattr(name, 'path', name)`` which is already supported in the development version, should be implemented in stdlib functions to accept not only ``str`` and path objects, but also DirEntry.) Guidelines for third-party package maintainers ---------------------------------------------- Libraries that take paths as arguments or return them ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Since all of the standard library will accept path objects as path arguments, most third-party libraries will automatically do so. However, those that directly manipulate or examine the path name using ``str`` methods may not work. Those libraries will not immediately be ``pathlib``-compatible. To achieve full ``pathlib``-compatiblity, the libraries are advised to: 1. Make sure they do not explicitly check the ``type(...)`` of arguments, but use ``isinstance(...)`` instead, if needed. 2. See if their functions use disabled ``str``/``bytes`` methods on paths that they take as arguments. If so, they should either: \* change their code to, achieve the same using ``os.path`` functions (*this is the preferred option*), or \* convert the argument first using ``name = getattr(name, 'path', name)``, which does not require importing pathlib 3. Consider, when returning a path or file name, to convert it to a path object first if a ``str``-subclassing ``pathlib`` is available. During a transition period, the attribute ``._enable_str_functionality = 'warn'`` should be set before returning the object. For an even softer, transition period it is also possible to set ``._enable_str_functionality = True``, which enables ``str`` methods with no warnings. Pathlib-compatible or near-compatible libraries ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ To have the best level of compatibility, all path-like objects should preferably behave similarly to pathlib objects regarding subclassing ``str``. However, for the best level of *compatibility*, the safest options is to subclass ``str`` and *not* disable ``str`` functionality (which is already done by some known libraries). However, they may want to further disable methods of ``str`` to achieve the additional clarity that ``pathlib`` has regarding \* Having a ``.path`` attribute/property which gives a ``str`` (or ``bytes``) instance representing the path Older Python versions ~~~~~~~~~~~~~~~~~~~~~ The ``pathlib2`` module, which provides ``pathlib`` to pre-3.4 python versions, can also subclass ``str``, but it should by default have ``._enable_str_functionality = 'warn'`` or ``.enable_str_functionality = True``, because the stdlib in the older Python-versions is not compatible with paths that have ``str`` functionality disabled. Transition plans and future possibilities for long-term consideration --------------------------------------------------------------------- ``DirEntry`` ~~~~~~~~~~~~ ``DirEntry`` should also undergo a similar transition, which was, at first, part of this proposal, but it was removed to limit the scope (It could be added back, of course, if desired). Since ``DirEntry`` focuses on performance, it is important not to cause any significant performance drops. It would, however, simplify things if ``DirEntry`` did the same as ``pathlib`` regarding subclassing and disabling methods. A slight complication, however, arises from the fact that ``DirEntry`` may represent a path using ``bytes``, making the ``.path`` attribute also an instance of ``bytes`` instead of ``str``. This issue could be solved by at least two different approaches: 1. Make ``bytes``-kind DirEntry instances, interpreted as ``str`` instances, equivalent to ``os.fsdecode(direntry.path)``. 2. Instantiate a different ``DirEntry`` class for ``bytes`` paths, perhaps in a way similar to how the ``antipathy`` path library instantiates ``bPath`` when the ``bytes`` type is used. The future of plain string paths and ``os.path``? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It is possible to imagine having both ``os.path`` and ``pathlib`` coexist, as long as they co-operate well. Potentially, things like ``open("filename.txt")`` with a plain string argument will always be accepted. However, if regardless of what people use Python for, they slowly adopt path objects as the way to represent a path, the support for plain string paths may be deprecated and eventually dropped. On the one hand, to support the former situation, ``os.path`` functions can choose their return type to match the type of the arguments; with multiple different types in the arguments, ``pathlib`` might 'win' because it is already imported. On the other hand, to support the latter, all path-returning functions in the stdlib can begin to return pathlib objects, at first with ``str`` methods enabled with or without warning, and eventually, with ``str`` methods disabled. Literal syntax for paths: p-strings? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Should Python choose the *path* towards not allowing plain strings as paths, a convenient way to instantiate a path is desperately needed. As discussed in the recent python-ideas thread "Working with path objects: p-strings?", one possibility would be a new syntax like ``p"/path/to/file.ext"``, which would instantiate a path object. Another way of turning a string into a path could be to have a ``.path`` property on ``str`` objects that instantiates and returns a path object. It can be debated whether this 'Pythonic' or not. See also the next section. The ``.path`` attribute on path-like objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ``DirEntry`` already had the ``.path`` attribute when it was introduced to the standard library in Python 3.5. It represents the absolute or relative path as a whole as a ``str`` or ``bytes`` instance. However, several people have raised the concern that the word ``path`` not referring to an actual path object may be misleading. However, if path objects are instances of str, the ``.path`` may in the future shift to mean the path object. In the case of ``pathlib`` paths, it would could be implemented as a property that returns ``self``, or during a transition phase, a path object with ``str`` functions enabled: .. code:: python @property def path(self): path = type(self)(self) path._enable_str_functionality = 'warn' return self ``DirEntry`` objects, on the other hand, could be converted to pathlib objects using the ``.path`` method. Similarly, ``str`` objects could have a similar property for conversion into a pathlib object (see previous section). Possibilities for making ``pathlib`` more lightweight ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If path objects were to become the norm in handling paths and file names, there may be a need for optimizations in terms of the speed and memory usage of path objects as well as the import time and memory footprint. Dependencies that are not always used by pathlib objects could also be imported lazily. Another base class for path-like objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Python already has multiple types that can represent paths-like objects. There could be one common base class for all of them, which would (at least at first) inherit from ``str``. ``DirEntry`` and ``PurePath`` would both be subclasses of this class. One would, however, need to answer the questions of what this class would be called, what it would look like, and what module would it be in (if not builtin). For now, let us call it ``PyRL`` for Python (Pyniversal?-) Resource Locator. This could also be a base class for URLs/URIs. Generalized Resource Locator addresses: a-strings? l-strings? ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ... not to mention g-strings! A generalized concept may be valuable in the future, because the distinction between local and remote is getting more and more vague. As discussed in the python-ideas thread "URLs/URIs + pathlib.Path + literal syntax = ?", it is possible to quite reliably distinguish common types of URLs from filesystem paths. If this became the norm, many Python-written programs could 'magically' accept URLs as input file paths by simply calling the ``PyRL(...)``, which could be equivalent to some literal syntax for use in a scripting, testing or interactive setting, or when loading config files from fixed locations. [View Less]

5 4

Control Flow - Never Executed Loop Body
by Sven R. Kunze April 6, 2016

April 6, 2016

Hi Python-ideas, this is a follow up from an thread of Python-List: https://mail.python.org/pipermail/python-list/2016-March/705205.html What do you think about the following proposal? *Abstract* Control structures such as "for", "while" and "try" feature some various additional clauses being executed under certain conditions. This proposal adds another extra clause to "for" and "while" in order to provide an easy way to discover whether the body of a loop as never been executed at … [View More]all. *Motivation** *Coming from the Web application development, we periodically can see a benefit of a hypothetical "empty" clause of Python's for loop like this: for item in my_iterator: # do per item empty: # do something else The same goes for "while" although there the choice of the name "empty" might be questionable. "empty" refers to the involved iterable which is not really existing in case of "while". However, there is not need of introducing inconsistencies among the loop constructs as both feature the "else" clause as well. Better suggestions are welcome; also see the keyword section. *Keyword* Keywords under consideration have been in (an attempt of dispassionate) order of preference: 1) empty -> most obvious, most people responded with a solution to solve the intended problem 2) else -> already taken but to some would be the preferred one 3) or -> as alternative if a new keyword is too much 4) except -> as alternative if a new keyword is too much 5) instead -> less obvious and new keyword *Use-Cases* 1) As humans mostly have a hard time to discover the difference between *nothing**as error* and *nothing**as really empty*, GUI/Web interfaces display an explicit notice in case of empty lists to signify the correct execution of a process/action/search or the other case. 2) Single way and no boilerplate of handling empty loops. As the discussion on Python-List showed are there quite some number of different ways of achieving this kind of functionality. 3) Less errors in development and maintenance. Some of the proposed solutions would not work in some circumstances (either during development or depending on the type of the items) which can hide errors. *Implementation* To be done. *Maintainer* Me. *Issues** *People I talked to suggested "else" as an alternative to "empty" because "empty is not quite clear". Unfortunately, "else" is already taken and is itself not quite clear. "then" has been proposed as an alternative for "else" in an attempt for proper understanding of "else". If that would be an accepted proposal, it would make room for "else" to be used for the usage of the "empty keyword proposed here. *Related Work** *This proposal is partly inspired by template engines like Django or jinja2 which both already provide such facility. Django: {% for item in my_iterator %} ... {% empty %} ... {% endfor %} jinja2: {% for item in my_iterator %} ... {% else %} ... {% endfor %} Best, Sven [View Less]

29 144

`to_file()` method for strings
by Nick Eubank April 4, 2016

April 4, 2016

As a social scientists trying to help other social scientists move from language like R, Stata, and Matlab into Python, one of the behaviors I've found unnecessarily difficult to explain is the "file.open()/file.close()" idiom (or, alternatively, context managers). In normal operating systems, and many high level languages, saving is a one-step operation. I understand there are situations where an open file handle is useful, but it seems a simple `to_file` method on strings (essentially … [View More]

24 73

Re: [Python-ideas] Decorators for variables
by Ian Kelly April 3, 2016

April 3, 2016

(Resending to correct list. Sorry about that.) On Fri, Apr 1, 2016 at 11:25 AM, Ian Kelly <ian.g.kelly(a)gmail.com> wrote: > On Fri, Apr 1, 2016 at 11:14 AM, Matthias welp <boekewurm(a)gmail.com> wrote: >>> An example of the transformation would help here >> >> An example, that detects cycles in a graph, and doesn't do an update if >> the graph has cycles. > > Thanks. > >> class A(object): >> def __init__(self, parent): >… [View More]

9 19

Fwd: Make parenthesis optional in parameterless functions definitions
by Mahan Marwat April 3, 2016

April 3, 2016

---------- Forwarded message ---------- From: Mahan Marwat <mahanmarwat(a)gmail.com> Date: Thu, Mar 31, 2016 at 9:09 PM Subject: Make parenthesis optional in parameterless functions definitions To: python-ideas(a)python.org Hi, I have an idea of making parenthesis optional for functions having no parameters. i.e def greet: # note the missing parenthesis print('hello') The less awkward characters we have, the more readable our code will be (Beautiful is better then ugly). Some … [View More]

16 34

Decorators for variables
by Matthias welp April 3, 2016

April 3, 2016

tldr: Using three method declarations or chaining method calls is ugly, why not allow variables and attributes to be decorated too? Currently the way to create variables with custom get/set/deleters is to use the @property decorator or use property(get, set, del, doc?), and this must be repeated per variable. If I were able to decorate multiple properties with decorators like @not_none or something similar, it would take away a lot of currently used confusing code. Feedback is appreciated. --… [View More]

6 8

Working with Path objects: p-strings?
by Koos Zevenhoven April 2, 2016

April 2, 2016

So, let's start a new thread about how to deal with pathlib.Path objects, involving how/whether to convert between str and Path. On Fri, Mar 25, 2016 at 7:14 PM, Sven R. Kunze <srkunze(a)mail.de> wrote: >> On 24.03.2016 22:06, Koos Zevenhoven wrote: >> >> Or even better, that you could do p"filename.txt", which would give you a >> Path string object. Has this been discussed? >> > > Interesting. I had this thought half a year ago. Quite recently I … [View More]mentioned > this to Andrew in a private conversation. > Yeah, I first thought about this at some point last summer when I had written the little np package for creating numpy arrays as np[3,4,5] (which I am planning to polish and maybe advertise), probably also inspired by the f-string discussions on this list. And now that you asked me about dealing with Paths... Anyway, as you say, if different people have thought about this independently, maybe its worth discussing. > p'/etc/hosts' would make a perfect path which subclassed from str. > Yeah, and regarding the "`to_file()` method for strings" thread, one could simply do p"targetfile.txt".write_text(string_to_write) using the write_text method introduced to Path objects in 3.5. [And of course there's also read_text(), write_bytes(...) and read_bytes()] One downside to p-strings like this is that they don't help you when you have the path in a plain str variable (i.e. not a literal), which has led me to think Path perhaps should be in builtins. But one thing that just occured to me now is that, for instance in interactive sessions, you could do p"" / path_as_str instead of `import pathlib; pathlib.Path(path_as_str)`. Not sure if this is the most beautiful thing ever seen, but it would work for quick scripting at least. > The p-string idea has not been discussed in the PEP I think. The subclassing > thing however was and I think its resolution was a mistake. The explanation, > at least from my point of view, is a bit weird. > Yeah, having Path subclass str would immediately solve a lot of compatibility issues. One question is how large a part of those issues would be solved by just making the builtin open(...) and a few places here and there in the stdlib accept Path objects. Maybe there are implementation issues too, although having them lead to bad decisions would be sad. Anyway, I think the question of whether Path should subclass str is partly separate from the p-string concept. After all, b'byte strings' are not subclassed from str either. - Koos [View Less]

26 141

Re: [Python-ideas] Decorators for variables
by Matthias welp April 1, 2016

April 1, 2016

> An example of the transformation would help here An example, that detects cycles in a graph, and doesn't do an update if the graph has cycles. class prevent_cycles(property): """ This uses nodes that point to only one other node: if there is a cycle in the current subgraph, it will detect that within O(n) """ def __init__(self, value): super().__init__() self._value = None self.__set__(value) def getter(self): return self.… [View More]

1 0

The future of Python: fixing broken error handling in Python 8
by Ryan Gonzalez April 1, 2016

April 1, 2016

Python's exception handling system is currently badly brokeTypeError: unsupported operand type(s) for +: 'NoneType' and 'NoneType'n. Therefore, with the recent news of the joyous release of Python 8 ( https://mail.python.org/pipermail/python-dev/2016-March/143603.html), I have decided to propose a revolutionary idea: safe mock objects. A "safe" mock object (qualified name `_frozensafemockobjectimplementation.SafeMockObjectThatIsIncludedWithPython8`; Java-style naming was adopted for … [View More]readability purposes; comments are now no longer necessary) is a magic object that supports everything and returns itself. Since examples speak more words than are in the Python source code, here are some (examples, not words in the Python source code): a = 1 b = None c = a + b # Returns a _frozensafemockobjectimplementation.SafeMockObjectThatIsIncludedWithPython8 print(c) # Prints the empty string. d = c+1 # All operations on `_frozensafemockobjectimplementation.SafeMockObjectThatIsIncludedWithPython8`'s return a new one. e = d.xyz(1, 2, 3) # `e` is now a `_frozensafemockobjectimplementation.SafeMockObjectThatIsIncludedWithPython8`. def f(): assert 0 # Causes the function to return a `_frozensafemockobjectimplementation.SafeMockObjectThatIsIncludedWithPython8`. raise 123 # Does the same thing. print(L) # L is undefined, so it becomes a `_frozensafemockobjectimplementation.SafeMockObjectThatIsIncludedWithPython8`. Safe mock objects are obviously the Next Error Handling Revolution ™. Unicode errors now simply disappear and return more `_frozensafemockobjectimplementation.SafeMockObjectThatIsIncludedWithPython8`s. As for `try` and `catch` (protest the naming of `except`!!) statements, they will be completely ignored. The `try`, `except`, and `finally` bodies will all be executed in sequence, except that printing and returning values with an `except` statement does nothing: try: xyz = None.a # `xyz` becomes a `_frozensafemockobjectimplementation.SafeMockObjectThatIsIncludedWithPython8`. except: print(123) # Does nothing. return None # Does nothing. finally: return xyz # Returns a `_frozensafemockobjectimplementation.SafeMockObjectThatIsIncludedWithPython8`. Aggressive error handling (as shown in PanicSort [https://xkcd.com/1185/]) that does destructive actions (such as `rm -rf /`) will always execute the destructive code, encouraging more honest development. In addition, due to errors simply being ignored, nothing can ever quite go wrong. All discussions about a safe navigation operator can now be immediately halted, since any undefined attributes will simply return a `_frozensafemockobjectimplementation.SafeMockObjectThatIsIncludedWithPython8`. Although I have not yet destroy--I mean, improved CPython to allow for this amazing idea, I have created a primitive implementation of the `_frozensafemockobjectimplementation` module: https://github.com/kirbyfan64/_frozensafemockobjectimplementation I hope you will all realize that this new idea is a drastic improvement over current technologies and therefore support it, because we can Make Python Great Again™. -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something’s wrong. http://kirbyfan64.github.io/ [View Less]

2 1