Mailman 3 December 2016 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum March 14, 2018

March 14, 2018

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' … [View More]

51 165

https://docs.python.org/fr/ ?
by Julien Palard Feb. 2, 2017

Feb. 2, 2017

o/ The french translation of the Python Documentation [1][2] has translated 20% of the pageviews of docs.python.org. I think it's the right moment to push it do docs.python.org. So there's some questions ! And I'd like feedback. TL;DR (with my personal choices): - URL may be "http://docs.python.org/fr/" - For localized variations of languages we should use dash and lowercase like "docs.python.org/pt-br/" - po files may be hosted on the python's github - existing script to build … [View More]doc may be patched to build translations - each translations may crosslink to others - untranslated strings may be visually marked as so I also opened: http://bugs.python.org/issue26546. # Chronology, dependencies The only blocking decision here is the URL, (also reviewing my patch ...), with those two, translated docs can be pushed to production, and the other steps can be discussed and applied one by one. # The URL ## CCTLD vs path vs subdomain I think we should use a variation of "docs.python.org/fr/" for simplicity and clarity. I think we should avoid using CCTLDs as they're sometime hard or near impossible to obtain (may cost a lot of time), also some are expensive, so it's time and money we clearly don't need to loose. Last possibility I see is to use a subdomain, like fr.docs.python.org or docs.fr.python.org but I don't think it's the role / responsibility of the sub-domain to do it. So I'm for docs.python.org/LANGUAGE_TAG/ (without moving current documentation inside a /en/). ## Language tag in path ### Dropping the default locale of a language I personally think we should not show the region in case it's redundant: so to use "fr" instead of "fr-FR", "de" instead of "de-DE", but keeping the possibility to use a locale code when it's not redundant like for "pt-br" or "de-AT" (German ('de') as used in Austria ('AT')). I think so because I don't think we'll have a lot of locale variations (like de-AT, fr-CH, fr-CA, ...) so it will be most of the time redundant (visually heavy, longer to type, longer to read) but we'll still need some locale (pt-BR typically). ### gettext VS IETF language tag format gettext goes by using an underscore between language and locale [3] and IETF goes by using a dash [4][5]. As sphinx is using gettext, and gettext uses underscore we may choose underscore too. But URLs are not here to leak the underlying implementation, and the IETF looks like to be the standard way to represent language tags. Also I visually prefer the dash over the underscore, so I'm for the dash here. ### Lower case vs upper case local tag RFC 5646 section-2.1 tells us language tags are not case sensitive, yet ISO3166-1 recommends that country codes (part of the language tag) be capitalized. I personally prefer the all-lowercase one as paths in URLs typically are lowercase. I searched for `inurl:"pt-br"` to see if I'm not too far away from the usage here and usage seems to agree with me, although there's some "pt-BR" in urls. # Where to host the translated files Currently we're hosting the *po* files in the afpy's (Francophone association for python) [6] github [1] but it may make sense to use (in the generation scripts) a more controlled / restricted clone in the python github, at least to have a better view of who can push on the documentation. We may want to choose between aggregating all translations under the same git repository but I don't feel it's useful. # How to Currently, a python script [7] is used to generate `docs.python.org`, I proposed a patch in [8] to make this script clone and build the french translation too, it's a simple and effective way, I don't think we need more ? Any idea welcome. In our side, we have a Makefile [12] to build the translated doc which is only a thin layer on top of the Sphinx Makefile. So my proposed patch to build scripts "just" delegate the build to our Makefile which itself delegate the hard work to the Sphinx Makefile. # Next ? ## Document how to translate Python I think I can (should) write a documentation on "how to start a Python doc translation project" and "how to migrate existing [9][10][11] python doc translation projects to docs.python.org" if french does goes docs.python.org because it may hopefully motivate people to do the same, and I think our structure is a nice way to do it (A Makefile to generate the doc, all versions translated, people mainly working on latest version, scripts to propagating translations to older version, etc...). ## Crosslinking between existing translations Once the translations are on `docs.python.org`, crosslinks may be established so people on a version can be aware of other version, and easily switch to them. I'm not a UI/UX man but I think we may have a select box right before the existing select box about version, on the top-left corner. Right before because it'll reflect the path: /fr/3.5/ -> [select box fr][select box 3.5]. ## Marking as "untranslated, you can help" the untranslated paragraphs The translations will always need work to follow upstream modifications: marking untranslated paragraphs as so may transform the "Oh they suck, this paragraph is not even translated :-(" to "Hey, nice I can help translating that !". There's an opened sphinx-doc ticket to do so [13] but I have not worked on it yet. As previously said I'm real bad at designing user interfaces, so I don't even visualize how I'd like it to be. [1] http://www.afpy.org/doc/python/3.5/ [2] https://github.com/afpy/python_doc_fr [3] https://www.gnu.org/software/gettext/manual/html_node/Locale-Names.html [4] http://tools.ietf.org/html/rfc5646 [5] https://en.wikipedia.org/wiki/IETF_language_tag [6] http://www.afpy.org/ [7] https://github.com/python/docsbuild-scripts/ [8] http://bugs.python.org/issue26546 [9] http://docs.python.jp/3/ [10] https://github.com/python-doc-ja/python-doc-ja [11] http://docs.python.org.ar/tutorial/3/index.html [12] https://github.com/AFPy/python_doc_fr/blob/master/Makefile [13] https://github.com/sphinx-doc/sphinx/issues/1246 -- Julien Palard [View Less]

7 8

proposal: "python -m foo" should bind sys.modules['foo']
by Cameron Simpson Jan. 18, 2017

Jan. 18, 2017

Hello all, This is a writeup of a proposal I floated here: https://mail.python.org/pipermail/python-list/2015-August/694905.html last Sunday. If the response is positive I wish to write a PEP. Briefly, it is a natural expectation in users that the command: python -m module_name ... used to invoke modules in "main program" mode on the command line imported the module as "module_name". It does not, it imports it as "__main__". An import within the program of "module_name" makes a new … [View More]instance of the module, which causes cognitive dissonance and has the side effect that now the program has two instances of the module. What I propose is that the above command line _should_ bind sys.modules['module_name'] as well as binding '__main__' as it does currently. I'm proposing that the python -m option have this effect (python pseudocode): % python -m module.name ... runs: # pseudocode, with values hardwired for clarity import sys M = new_empty_module(name='__main__', qualname='module.name') sys.modules['__main__'] = M sys.modules['module.name'] = M # load the module code from wherever (not necessarily a file - CPython # already must do this phase) M.execfile('/path/to/module/name.py') Specificly, this would have the following two changes to current practice: 1) the module is imported _once_, and bound to both its canonical name and also to __main__. 2) imported modules acquire a new attribute __qualname__ (analogous to the recent __qualname__ on functions). This is always the conanoical name of the module as resolved by the importer. For most modules __name__ will be the same as __qualname__, but for the "main" module __name__ will be '__main__'. This change has the following advantages: The current standard boilerplate: if __name__ == '__main__': ... invoke "main program" here ... continues to work unchanged. Importantly, if the program then issues "import module_name", it is already there and the existing instance is found and used. The thread referenced above outlines my most recent encounter with this and the trouble it caused me. Followup messages include some support for this proposed change, and some criticism. The critiquing article included some workarounds for this multiple module situation, but they were (1) somewhat dependent on modules coming from a file pathname and (2) cumbersome and require every end user to adopt these changes if affected by the situation. I'd like to avoid that. Cheers, Cameron Simpson <cs(a)zip.com.au> The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. - George Bernard Shaw [View Less]

5 13

api suggestions for the cProfile module
by Thane Brimhall Jan. 11, 2017

Jan. 11, 2017

I use cProfile a lot, and would like to suggest three backwards-compatible improvements to the API. 1: When using cProfile on a specific piece of code I often use the enable() and disable() methods. It occurred to me that this would be an obvious place to use a context manager. 2: Enhance the `print_stats` method on Profile to accept more options currently available only through the pstats.Stats class. For example, strip_dirs could be a boolean argument, and limit could accept an int. This … [View More]

5 6

Re: [Python-ideas] New PyThread_tss_ C-API for CPython
by Masayuki YAMAMOTO Jan. 7, 2017

Jan. 7, 2017

I have read the discussion and I'm sure that use structure as Py_tss_t instead of platform-specific data type. Just as Steve said that Py_tss_t should be genuinely treated as an opaque type, the key state checking should provide macros or inline functions with name like PyThread_tss_is_created. Well, I'd resolve the specification a bit more :) If PyThread_tss_create is called with the created key, it is no-op but which the function should succeed or fail? In my opinion, It is better to return … [View More]a failure because it is a high possibility that the code is incorrect for multiple callings of PyThread_tss_create for One key. In this opinion PyThread_tss_is_created should return a value as follows: (A) False while from after defining with Py_tss_NEED_INIT to before calling PyThread_tss_create (B) True after calling PyThread_tss_create succeeded (C) Unchanging before and after calling PyThread_tss_create failed (D) False after calling PyThread_tss_delete regardless of timing (E) For other functions, the return value of PyThread_tss_is_created does not change before and after calling I think that it is better to write a test about the state of the Py_tss_t. Kind regards, Masayuki 2016-12-31 2:38 GMT+09:00 Erik Bray <erik.m.bray(a)gmail.com>: > On Fri, Dec 30, 2016 at 5:05 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > On 29 December 2016 at 22:12, Erik Bray <erik.m.bray(a)gmail.com> wrote: > >> > >> 1) CPython's TLS: Defines -1 as an uninitialized key (by fact of the > >> implementation--that the keys are integers starting from zero) > >> 2) pthreads: Does not definite an uninitialized default value for > >> keys, for reasons described at [1] under "Non-Idempotent Data Key > >> Creation". I understand their reasoning, though I can't claim to know > >> specifically what they mean when they say that some implementations > >> would require the mutual-exclusion to be performed on > >> pthread_getspecific() as well. I don't know that it applies here. > > > > > > That section is a little weird, as they describe two requests (one for a > > known-NULL default value, the other for implicit synchronisation of key > > creation to prevent race conditions), and only provide the justification > for > > rejecting one of them (the second one). > > Right, that is confusing to me as well. I'm guessing the reason for > rejecting the first is in part a way to force us to recognize the > second issue. > > > If I've understood correctly, the situation they're worried about there > is > > that pthread_key_create() has to be called at least once-per-process, but > > must be called before *any* call to pthread_getspecific or > > pthread_setspecific for a given key. If you do "implicit init" rather > than > > requiring the use of an explicit mechanism like pthread_once (or our own > > Py_Initialize and module import locks), then you may take a small > > performance hit as either *every* thread then has to call > > pthread_key_create() to ensure the key exists before using it, or else > > pthread_getspecific() and pthread_setspecific() have to become > potentially > > blocking calls. Neither of those is desirable, so it makes sense to leave > > that part of the problem to the API client. > > > > In our case, we don't want the implicit synchronisation, we just want the > > known-NULL default value so the "Is it already set?" check can be moved > > inside the library function. > > Okay, we're on the same page here then. I just wanted to make sure > there wasn't anything else I was missing in Python's case. > > >> 3) windows: The return value of TlsAlloc() is a DWORD (unsigned int) > >> and [2] states that its value should be opaque. > >> > >> So in principle we can cover all cases with an opaque struct that > >> contains, as its first member, an is_initialized flag. The tricky > >> part is how to initialize the rest of the struct (containing the > >> underlying implementation-specific key). For 1) and 3) it doesn't > >> matter--it can just be zero. For 2) it's trickier because there's no > >> defined constant value to initialize a pthread_key_t to. > >> > >> Per Nick's suggestion this can be worked around by relying on C99's > >> initialization semantics. Per [3] section 6.7.8, clause 21: > >> > >> """ > >> If there are fewer initializers in a brace-enclosed list than there > >> are elements or members of an aggregate, or fewer characters in a > >> string literal used to initialize an array of known size than there > >> are elements in the array, the remainder of the aggregate shall be > >> initialized implicitly the same as objects that have static storage > >> duration. > >> """ > >> > >> How objects with static storage are initialized is described in the > >> previous page under clause 10, but in practice it boils down to what > >> you would expect: Everything is initialized to zero, including nested > >> structs and arrays. > >> > >> So as long as we can use this feature of C99 then I think that's the > >> best approach. > > > > > > > > I checked PEP 7 to see exactly which features we've added to the > approved C > > dialect, and designated initialisers are already on the list: > > https://gcc.gnu.org/onlinedocs/gcc/Designated-Inits.html > > > > So I believe that would allow the initializer to be declared as something > > like: > > > > #define Py_tss_NEEDS_INIT {.is_initialized = false} > > Great! One could argue about whether or not the designated > initializer syntax also incorporates omitted fields, but it would seem > strange to insist that it doesn't. > > Have a happy new year, > > Erik > [View Less]

2 2

incremental hashing in __hash__
by jab＠math.brown.edu Jan. 6, 2017

Jan. 6, 2017

Suppose you have implemented an immutable Position type to represent the state of a game played on an MxN board, where the board size can grow quite large. Or suppose you have implemented an immutable, ordered collection type. For example, the collections-extended package provides a frozensetlist[1]. One of my own packages provides a frozen, ordered bidirectional mapping type.[2] These types should be hashable so that they can be inserted into sets and mappings. The order-sensitivity of the … [View More]contents prevents them from using the built-in collections.Set._hash() helper in their __hash__ implementations, to keep from unnecessarily causing hash collisions for objects that compare unequal due only to having a different ordering of the same set of contained items. According to https://docs.python.org/3/reference/datamodel.html#object.__hash__ : """ it is advised to mix together the hash values of the components of the object that also play a part in comparison of objects by packing them into a tuple and hashing the tuple. Example: def __hash__(self): return hash((self.name, self.nick, self.color)) """ Applying this advice to the use cases above would require creating an arbitrarily large tuple in memory before passing it to hash(), which is then just thrown away. It would be preferable if there were a way to pass multiple values to hash() in a streaming fashion, such that the overall hash were computed incrementally, without building up a large object in memory first. Should there be better support for this use case? Perhaps hash() could support an alternative signature, allowing it to accept a stream of values whose combined hash would be computed incrementally in *constant* space and linear time, e.g. "hash(items=iter(self))". In the meantime, what is the best way to incrementally compute a good hash value for such objects using built-in Python routines? (As a library author, it would be preferable to use a routine with explicit support for computing a hash incrementally, rather than having to worry about how to correctly combine results from multiple calls to hash(contained_item) in library code. (Simply XORing such results together would not be order-sensitive, and so wouldn't work.) Using a routine with explicit support for incremental hashing would allow libraries to focus on doing one thing well.[3,4,5]) I know that hashlib provides algorithms that support incremental hashing, but those use at least 128 bits. Since hash() throws out anything beyond sys.hash_info.hash_bits (e.g. 64) bits, anything in hashlib seems like overkill. Am I right in thinking that's the wrong tool for the job? On the other hand, would binascii.crc32 be suitable, at least for 32-bit systems? (And is there some 64-bit incremental hash algorithm available for 64-bit systems? It seems Python has no support for crc64 built in.) For example: import binascii, struct class FrozenOrderedCollection: def __hash__(self): if hasattr(self, '__hashval'): # Computed lazily. return self.__hashval hv = crc32(b'FrozenOrderedCollection') for i in self: hv = binascii.crc32(struct.pack('@l', hash(i)), hv) hv &= 0xffffffff self.__hashval = hv return hv Note that this example illustrates two other common requirements of these use cases: (i) lazily computing the hash value on first use, and then caching it for future use (ii) priming the overall hash value with some class-specific initial value, so that if an instance of a different type of collection, which comprised the same items but which compared unequal, were to compute its hash value out of the same constituent items, we make sure our hash value differs. (On that note, should the documentation in https://docs.python.org/3/reference/datamodel.html#object.__hash__ quoted above be updated to add this advice? The current advice to "return hash((self.name, self.nick, self.color))" would cause a hash collision with a tuple of the same values, even though the tuple should presumably compare unequal with this object.) To summarize these questions: 1. Should hash() add support for incremental hashing? 2. In the meantime, what is the best way to compute a hash of a combination of many values incrementally (in constant space and linear time), using only what's available in the standard library? Ideally there is some routine available that uses exactly hash_info.hash_bits number of bits, and that does the combining of incremental results for you. 3. Should the https://docs.python.org/3/reference/datamodel.html#object.__hash__ documentation be updated to include suitable advice for these use cases, in particular, that the overall hash value should be computed lazily, incrementally, and should be primed with a class-unique value? Thanks in advance for a helpful discussion, and best wishes. Josh References: [1] http://collections-extended.lenzm.net/api.html#collections_extended.frozens… [2] https://bidict.readthedocs.io/en/dev/api.html#bidict.frozenorderedbidict [3] http://stackoverflow.com/questions/2909106/python-whats-a-correct-and-good-… [4] http://stackoverflow.com/a/2909572/161642 [5] http://stackoverflow.com/a/27952689/161642 [View Less]

18 55

AtributeError inside __get__
by Zahari Dim Dec. 31, 2016

Dec. 31, 2016

Hi, The other day I came across a particularly ugly bug. A simplified case goes like: class X: @property def y(self): return self.nonexisting hasattr(X(),'y') This returns False because hasattr calls the property which in turn raises an AttributeError which is used to determine that the property doesn't exist, even if it does. This is arguably unexpected and surprising and can be very difficult to understand if it happens within a large codebase. Given the precedent with … [View More]

7 12

New PyThread_tss_ C-API for CPython
by Erik Bray Dec. 30, 2016

Dec. 30, 2016

Greetings all, I wanted to bring attention to an issue that's been languishing on the bug tracker since last year, which I think would best be addressed by changes to CPython's C-API. The original issue is at http://bugs.python.org/issue25658, but I have made an effort below in a sort of proto-PEP to summarize the problem and the proposed solution. I haven't written this up in the proper PEP format because I want to see if the idea has some broader support first, and it's also not clear to … [View More]me whether C-API changes (especially to undocumented APIs) even require their own PEP. Abstract ======== The proposal is to add a new Thread Local Storage (TLS) API to CPython which would supersede use of the existing TLS API within the CPython interpreter, while deprecating the existing API. Because the existing TLS API is only used internally (it is not mentioned in the documentation, and the header that defines it, pythread.h, is not included in Python.h either directly or indirectly), this proposal probably only affects CPython, but might also affect other interpreter implementations (PyPy?) that implement parts of the CPython API. Specification ============= The current API for TLS used inside the CPython interpreter consists of 5 functions: PyAPI_FUNC(int) PyThread_create_key(void) PyAPI_FUNC(void) PyThread_delete_key(int key) PyAPI_FUNC(int) PyThread_set_key_value(int key, void *value) PyAPI_FUNC(void *) PyThread_get_key_value(int key) PyAPI_FUNC(void) PyThread_delete_key_value(int key) These would be superseded with a new set of analogous functions: PyAPI_FUNC(int) PyThread_tss_create(Py_tss_t *key) PyAPI_FUNC(void) PyThread_tss_delete(Py_tss_t key) PyAPI_FUNC(int) PyThread_tss_set(Py_tss_t key, void *value) PyAPI_FUNC(void *) PyThread_tss_get(Py_tss_t key) PyAPI_FUNC(void) PyThread_tss_delete_value(Py_tss_t key) and includes the definition of a new type Py_tss_t--any opaque type the specification of which is not given here, and may depend on the underlying TLS implementation. The new PyThread_tss_ functions are almost exactly analogous to their original counterparts with a minor difference: Whereas PyThread_create_key takes no arguments and returns a TLS key as an int, PyThread_tss_create takes a Py_tss_t* as an argument, and returns a Py_tss_t by pointer--the int return value is a status, returning zero on success and non-zero on failure. Further, the old PyThread_*_key* functions will be marked as deprecated. Additionally, the pthread implementations of the old PyThread_*_key* functions will either fail or be no-ops on platforms where sizeof(pythead_t) != sizeof(int). Motivation ========== The primary problem at issue here is the type of the keys (int) used for TLS values, as defined by the original PyThread TLS API. The original TLS API was added to Python by GvR back in 1997, and at the time the key used to represent a TLS value was an int, and so it has been to this day. This used CPython's own TLS implementation, the current generation of which can still be found, largely unchanged, in Python/thread.c. Support for implementation of the API on top of native thread implementations (NT and pthreads) was added much later, and the built-in implementation may still be used on other platforms. The problem with the choice of int to represent a TLS key, is that while it was fine for CPython's internal TLS implementation, and happens to be fine for NT (which uses DWORD), it is not compatible the POSIX standard for the pthreads API, which defines pthread_key_t as an opaque type not further designed by the standard (as with Py_tss_t described above). This leaves it up to the underlying implementation how a pthread_key_t value is used to look thread-specific data. This has not generally been a problem for Python's API, as it just happens that on Linux pthread_key_t is just defined as an unsigned int, and so is fully compatible with Python's TLS API--pthread_key_t's created by pthread_create_key can be freely cast to ints and back (well, not really, even this has issues as pointed out by issue #22206). However, as issue #25658 points out there are at least some platforms (namely Cygwin, CloudABI, but likely others as well) which have otherwise modern and POSIX-compliant pthreads implementations, but are not compatible with Python's API because their pthread_key_t is defined in a way that cannot be safely cast to int. In fact, the possibility of running into this problem was raised by MvL at the time pthreads TLS was added [1]. It could be argued that PEP-11 makes specific requirements for supporting a new, not otherwise officially-support platform (such as CloudABI), and that the status of Cygwin support is currently dubious. However, this places a very barrier to supporting platforms that are otherwise Linux- and/or POSIX-compatible and where CPython might otherwise "just work" except for this one hurdle which Python itself imposes by way of an API that is not compatible with POSIX (and in fact makes invalid assumptions about pthreads). Rationale for Proposed Solution =============================== The use of an opaque type (Py_tss_t) to key TLS values allows the API to be compatible, at least in this regard, with CPython's internal TLS implementation, as well as all present (NT and posix) and future (C11?) native TLS implementations supported by CPython, as it allows the definition of Py_tss_t to depend on the underlying implementation. A new API must be introduced, rather than changing the function signatures of the current API, in order to maintain backwards compatibility. The new API also more clearly groups together these related functions under a single name prefix, "PyThread_tss_". The "tss" in the name stands for "thread-specific storage", and was influenced by the naming and design of the "tss" API that is part of the C11 threads API. However, this is in no way meant to imply compatibility with or support for the C11 threads API, or signal any future intention of supporting C11--it's just the influence for the naming and design. Changing PyThread_create_key to immediately return a failure status on systems using pthreads where sizeof(int) != sizeof(pthread_key_t) is intended as a sanity check: Currently, PyThread_create_key will report initial success on such systems, but attempts to use the returned key are likely to fail. Although in practice this failure occurs quickly during interpreter startup, it's better to fail immediately at the source of failure (PyThread_create_key) rather than sometime later when use of an invalid key is attempted. Rejected Ideas ============== * Do nothing: The status quo is fine because it works on Linux, and platforms wishing to be supported by CPython should follow the requirements of PEP-11. As explained above, while this would be a fair argument if CPython were being to asked to make changes to support particular quirks of a specific platform, in this case the platforms in question are only asking to fix a quirk of CPython that prevents it from being used to its full potential on those platforms. The fact that the current implementation happens to work on Linux is a happy accident, and there's no guarantee that will never change. * Affected platforms should just configure Python --without-threads: This is a possible temporary workaround to the issue, but only that. Python should not be hobbled on affected platforms despite them being otherwise perfectly capable of running multi-threaded Python. * Affected platforms should not define Py_HAVE_NATIVE_TLS: This is a more acceptable alternative to the previous idea, and in fact there is a patch to do just that [2]. However, CPython's internal TLS implementation being "slower and clunkier" in general than native implementations still needlessly hobbles performance on affected platforms. At least one other module (tracemalloc) is also broken if Python is built without Py_HAVE_NATIVE_TLS. * Keep the existing API, but work around the issue by providing a mapping from pthread_key_t values to ints. A couple attempts were made at this [3] [4], but this only injects needless complexity and overhead into performance-critical code on platforms that are not currently affected by this issue (such as Linux). Even if use of this workaround were made conditional on platform compatibility, it introduces platform-specific code to maintain, and still has the problem of the previous rejected ideas of needlessly hobbling performance on affected platforms. Implementation ============== An initial version of a patch [5] is available on the bug tracker for this issue. The patch is proposed and written by Masayuki Yamamoto, who should be considered a co-author of this proto-PEP, though I have not consulted directly with him in writing this. If he's reading, he should chime in in case I've misrepresented anything. If you've made it this far, thanks for reading and thank you for your consideration, Erik [1] https://bugs.python.org/msg116292 [2] http://bugs.python.org/file45548/configure-pthread_key_t.patch [3] http://bugs.python.org/file44269/issue25658-1.patch [4] http://bugs.python.org/file44303/key-constant-time.diff [5] http://bugs.python.org/file45763/pythread-tss.patch [View Less]

7 20

Function arguments in tracebacks
by Ammar Askar Dec. 30, 2016

Dec. 30, 2016

Consider the following similar C and Python code and their tracebacks: C ------- int divide(int x, int y, char* some_string) { return x / y; } int main(...) { divide(2, 0, "Hello World"); } ------- Program received signal SIGFPE, Arithmetic exception. (gdb) bt #0 0x00000000004004c4 in divide (x=2, y=0, some_string=0x4005a8 "Hello World") at test.c:2 #1 0x00000000004004e7 in main (argc=1, argv=0x7fffffffe328) at test.c:6 Python ------- def divide(x, y, … [View More]some_string): return x / y divide(2, 0, "Hello World") ------- Traceback (most recent call last): File "test.py", line 4, in <module> File "test.py", line 2, in divide ZeroDivisionError: division by zero By including the function arguments within the traceback, we can get more information at a glance than we could with just the names of methods. This would be pretty cool and stop the occasional "printf" debugging without cluttering up the traceback too much. There will definitely need to be some reasonable line length limit because the repr() of parameters could be really long. In similar situations gdb replaces the value in the traceback with elipsis, and I believe that's a good solution for python as well. Obviously this isn't a great example since the error is immediately obvious but I think this could be potentially useful in a bunch of situations. I've made a a quick toy implementation in traceback.c, this is what it looks like for the script above. Traceback (most recent call last): File "test.py", line 4, in <module> divide(2, 0, "Hello World") File "test.py", line 2, in divide (x=2, y=0, some_string='Hello World') return x / y ZeroDivisionError: division by zero == Potential Downsides == There's probably a lot more than these, but I could only think of these so far. * Private data might be leaked, imagine a def login(username, password): ... method. While function names/source files/source code are also private, variables can potentially contain all kinds of sensitive data. * A variable that takes a long time to return a string representation may significantly slow down the time it takes to generate a traceback. * We can really only return the state of the variables when the traceback is printed, this might result in some slightly un-intuitive behavior. (Easier to explain with an example) def f(x): x = 2 raise Exception() f(1) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 3, in f(x=2) The fact that x is mutated within the function body means that the value printed in the traceback is the changed value which might be slightly misleading. I'd love to hear your guy's thoughts on the idea. [View Less]

7 6

Importing public symbols and simultainiously privatizing them, is too noisy
by Rick Johnson Dec. 29, 2016

Dec. 29, 2016

Has anyone else found this to be too syntactically noisy? from module import Foo as _Foo, bar as _bar That is horrifically noisy IMO. The problem is, how do we remove the noise without sacrificing intuitiveness? My first idea was to do this: from module import_private Foo, bar And while it's self explanatory, it's also too long. So i thought... from module _import Foo, bar I'm leaning more towards the latter, but i'm not loving it either. Any ideas?

9 17

VT100 style escape codes in Windows
by Joseph Hackman Dec. 29, 2016

Dec. 29, 2016

Hey All! I propose that Windows CPython flip the bit for VT100 support (colors and whatnot) for the stdout/stderr streams at startup time. I believe this behavior is worthwhile because ANSI escape codes are standard across most of Python's install base, and the alternative for Windows (using ctypes/win32 to alter the colors) is non-intuitive and well beyond the scope of most users. Under Linux/Mac, the terminal always supports what it can, and it's up to the application to verify escape … [View More]

3 4

singledispatch for instance methods
by Tim Mitchell Dec. 26, 2016

Dec. 26, 2016

Hi All, We have a modified version of singledispatch at work which works for methods as well as functions. We have open-sourced it as methoddispatch (pypi: https://pypi.python.org/pypi/methoddispatch). IMHO I thought it would make a nice addition to python stdlib. What does everyone else think?

2 2

Re: [Python-ideas] (no subject)
by Victor Stinner Dec. 25, 2016

Dec. 25, 2016

Le 24 déc. 2016 8:42 PM, "Neil Girdhar" <mistersheik(a)gmail.com> a écrit : > Usually, when an exception is hit that will (probably) crash the program, no one cares about less than a microsecond of performance. Just one example. By design, hasattr(obj, name) raises an exception to return False. So it has the cost of building the exception + raise exc + catch it. Victor

2 1

(no subject)
by victor rajewski Dec. 24, 2016

Dec. 24, 2016

I teach a computing subject to high school students using Python as our primary language. One thing that often causes confusion at the start is error messages/exceptions. I think it would lower entry point quite a bit by making error messages more readable to novices. Recent research found reduced frequency of overall errors, errors per student, and repeated errors when error messages were enhanced [1]. This could be implemented by a command-line switch when running python, or a different … [View More]

23 40

Re: [Python-ideas] New PyThread_tss_ C-API for CPython
by Masayuki YAMAMOTO Dec. 24, 2016

Dec. 24, 2016

2016-12-21 19:01 GMT+09:00 Erik Bray <erik.m.bray(a)gmail.com>: > On Wed, Dec 21, 2016 at 2:10 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > Ouch, I'd missed that, and I agree it's not a negligible implementation > > detail - there are definitely applications embedding CPython out there > that > > rely on being able to run multiple Initialize/Finalize cycles in the same > > process and have everything "just work". It also means using the > > … [View More]"PyThread_*" prefix for the initialisation tracking aspect would be > > misleading, since the life cycle details are: > > > > 1. Create the key for the first time if it has never been previously set > in > > the process > > 2. Destroy and reinit if Py_Finalize gets called > > 3. Destroy and reinit if a new subprocess is forked > > > > It also means we can't use pthread_once even in the pthread TLS > > implementation, since it doesn't provide those semantics. > > > > So I see two main alternatives here. > > > > Option 1: Modify the proposed PyThread_tss_create and PyThread_tss_delete > > APIs to accept a "bool *init_flag" pointer in addition to their current > > arguments. > > > > If *init_flag is true, then PyThread_tss_create is a no-op, otherwise it > > sets the flag to true after creating the key. > > If *init_flag is false, then PyThread_tss_delete is a no-op, otherwise it > > sets the flag to false after deleting the key. > > > > Option 2: Similar to option 1, but using a custom type alias, rather than > > using a C99 bool directly > > > > The closest API we have to these semantics at the moment would be > > PyGILState_Ensure, so the following API naming might work for option 2: > > > > Py_ensure_t > > Py_ENSURE_NEEDS_INIT > > Py_ENSURE_INITIALIZED > > > > Respectively, these would just be aliases for bool, false, and true. > > > > And then modify the proposed PyThread_tss_create and PyThread_tss_delete > > APIs to accept a "Py_ensure_t *init_flag" in addition to their current > > arguments. > > That all sounds good--between the two option 2 looks a bit more explicit. > > Though what about this? Rather than adding another type, the original > proposal could be changed slightly so that Py_tss_t *is* partially > defined as a struct consisting of a bool, with whatever the native TLS > key is. E.g. > > typedef struct { > bool init_flag; > #if defined(_POSIX_THREADS) > pthreat_key_t key; > #elif defined (NT_THREADS) > DWORD key; > /* etc... */ > } Py_tss_t; > > Then it's just taking Masayuki's original patch, with the global bool > variables, and formalizing that by combining the initialized flag with > the key, and requiring the semantics you described above for > PyThread_tss_create/delete. > > For Python's purposes it seems like this might be good enough, with > the more general purpose pthread_once-like functionality not required. > > Best, > Erik Above mentioned, In currently TLS API, the thread key uses -1 as defined invalid value. If new TLS API inherits the specifications that the key requires defined invalid value, putting key and flag into one structure seems correct as semantics. In this case, I think TLS API should supply the defined invalid value (like PTHREAD_ONCE_INIT) to API users. Moreover, the structure has an opportunity to assert that the thread key type is the opaque using field name. I think to the suggestion that has effect to improve the understandability of the API because good field name can give that reading and writing to the key seems to be incorrect (even if API users don't read the precautionary statement). Have a nice holiday! Masayuki [View Less]

2 1

PEP 536 – Call for help and improvement
by Philipp A. Dec. 23, 2016

Dec. 23, 2016

Hi Python Ideas, And merry christmas! Once upon a time – in August this year – I started a (somewhat badly titled) thread about improving the f-string grammar: https://mail.python.org/pipermail/python-ideas/2016-August/041727.html Luckily it resulted in an interim grammar change that invalidated a misleading property of the original grammar: To the rejoicing of syntax highlighters and humans everywhere, it’s no longer possible to escape syntactically relevant characters such as the f-string … [View More]

1 0

Re: [Python-ideas] New PyThread_tss_ C-API for CPython
by Masayuki YAMAMOTO Dec. 19, 2016

Dec. 19, 2016

2016-12-17 18:35 GMT+09:00 Stephen J. Turnbull <turnbull.stephen.fw@u. tsukuba.ac.jp>: > I don't understand this. I assume that there are no such platforms > supported at present. I would think that when such a platform becomes > supported, code supporting "key" functions becomes unsupportable > without #ifdefs on that platform, at least directly. So you should > either (1) raise UnimplementedError, or (2) provide the API as a > wrapper over the new API by making the … [View More]

2 1

Enhancing vars()
by Steven D'Aprano Dec. 14, 2016

Dec. 14, 2016

In general, directly accessing dunders is a bit of a code smell. (I exclude writing dunder methods in your classes, of course.) There's usually a built-in or similar to do the job for you, e.g. instead of iterator.__next__() we should use next(iterator). One of the lesser-known ones is vars(obj), which should be used in place of obj.__dict__. Unfortunately, vars() is less useful than it might be, since not all objects have a __dict__. Some objects have __slots__ instead, or even both. … [View More]

12 17

Input characters in strings by decimals (Was: Proposal for default character representation)
by Mikhail V Dec. 8, 2016

Dec. 8, 2016

In past discussion about inputing and printing characters, I was proposing decimal notation instead of hex. Since the discussion was lost in off-topic talks, I'll try to summarise my idea better. I use ASCII only for code input (there are good reasons for that). Here I'll use Python 3.6, and Windows 7, so I can use print() with unicode directly and it works now in system console. Suppose I only start programming and want to do some character manipulation. The vey first thing I would probably … [View More]start with is a simple output for latin and cyrillic capital letters: caps_lat = "" for o in range(65, 91): caps_lat = caps_lat + chr(o) print (caps_lat) caps_cyr = "" for o in range(1040, 1072): caps_cyr = caps_cyr + chr(o) print (caps_cyr) Which prints: ABCDEFGHIJKLMNOPQRSTUVWXYZ АБВГДЕЖЗИЙКЛМНОПРСТУФХЦЧШЩЪЫЬЭЮЯ Say, I want now to input something direct in code: s = "first cyrillic letters: " + chr(1040) + chr(1041) + chr(1042) Which works fine and has clean look. However it is not very convinient because of much typing and also, if I generate such strings, adds a bit more complexity. But in general it is fine, and I use this method currently. ========= Proposal: I would want to have a possibility to input it *by decimals*: s = "first cyrillic letters: \{1040}\{1041}\{1042}" or: s = "first cyrillic letters: \(1040)\(1041)\(1042)" ========= This is more compact and seems not very contradictive with current Python escape characters in string literals. So backslash is a start of some escaping in most cases. For me most important is that in such way I would avoid any presence of hex numbers in strings, which I find very good for readability and for me it is very convinient since I use decimals for processing everywhere (and encourage everyone to do so). So this is my proposal, any comments on this are appreciated. PS: Currently Python 3 supports these in addition to \x: (from https://docs.python.org/3/howto/unicode.html) """ If you can’t enter a particular character in your editor or want to keep the source code ASCII-only for some reason, you can also use escape sequences in string literals. >>> "\N{GREEK CAPITAL LETTER DELTA}" # Using the character name >>> "\u0394" # Using a 16-bit hex value >>> "\U00000394" # Using a 32-bit hex value """ So I have many possibilities and all of them strangely contradicts with my image of intuitive and readable. Well, using charater name is readable, but seriously not much of a practical solution for input, but could be very useful for printing description of a character. Mikhail [View Less]

16 27

PEP: Distributing a Subset of the Standard Library
by Tomas Orsava Dec. 7, 2016

Dec. 7, 2016

Hi! We have written a draft PEP entitled "Distributing a Subset of the Standard Library" that aims to standardize and improve how Python handles omissions from its standard library. This is relevant both to Python itself as well as to Linux and other distributions that are packaging it, as they already separate out parts of the standard library into optionally installable packages. Ideas leading up to this PEP were discussed on the python-dev mailing list: https://mail.python.org/… [View More]pipermail/python-dev/2016-July/145534.html Rendered PEP: https://fedora-python.github.io/pep-drafts/pep-A.html ------------------------------------------------------------------------ Abstract ======== Python is sometimes being distributed without its full standard library. However, there is as of yet no standardized way of dealing with importing a missing standard library module. This PEP proposes a mechanism for identifying which standard library modules are missing and puts forth a method of how attempts to import a missing standard library module should be handled. Motivation ========== There are several use cases for including only a subset of Python's standard library. However, there is so far no formal specification of how to properly implement distribution of a subset of the standard library. Namely, how to safely handle attempts to import a missing *stdlib* module, and display an informative error message. CPython ------- When one of Python standard library modules (such as ``_sqlite3``) cannot be compiled during a Python build because of missing dependencies (e.g. SQLite header files), the module is simply skipped. If you then install this compiled Python and use it to try to import one of the missing modules, Python will go through the ``sys.path`` entries looking for it. It won't find it among the *stdlib* modules and thus it will continue onto ``site-packages`` and fail with a ModuleNotFoundError_ if it doesn't find it. .. _ModuleNotFoundError: https://docs.python.org/3.7/library/exceptions.html#ModuleNotFoundError This can confuse users who may not understand why a cleanly built Python is missing standard library modules. Linux and other distributions ----------------------------- Many Linux and other distributions are already separating out parts of the standard library to standalone packages. Among the most commonly excluded modules are the ``tkinter`` module, since it draws in a dependency on the graphical environment, and the ``test`` package, as it only serves to test Python internally and is about as big as the rest of the standard library put together. The methods of omission of these modules differ. For example, Debian patches the file ``Lib/tkinter/__init__.py`` to envelop the line ``import _tkinter`` in a *try-except* block and upon encountering an ``ImportError`` it simply adds the following to the error message: ``please install the python3-tk package`` [#debian-patch]_. Fedora and other distributions simply don't include the omitted modules, potentially leaving users baffled as to where to find them. Specification ============= When, for any reason, a standard library module is not to be included with the rest, a file with its name and the extension ``.missing.py`` shall be created and placed in the directory the module itself would have occupied. This file can contain any Python code, however, it *should* raise a ModuleNotFoundError_ with a helpful error message. Currently, when Python tries to import a module ``XYZ``, the ``FileFinder`` path hook goes through the entries in ``sys.path``, and in each location looks for a file whose name is ``XYZ`` with one of the valid suffixes (e.g. ``.so``, ..., ``.py``, ..., ``.pyc``). The suffixes are tried in order. If none of them are found, Python goes on to try the next directory in ``sys.path``. The ``.missing.py`` extension will be added to the end of the list, and configured to be handled by ``SourceFileLoader``. Thus, if a module is not found in its proper location, the ``XYZ.missing.py`` file is found and executed, and further locations are not searched. The CPython build system will be modified to generate ``.missing.py`` files for optional modules that were not built. Rationale ========= The mechanism of handling missing standard library modules through the use of the ``.missing.py`` files was chosen due to its advantages both for CPython itself and for Linux and other distributions that are packaging it. The missing pieces of the standard library can be subsequently installed simply by putting the module files in their appropriate location. They will then take precedence over the corresponding ``.missing.py`` files. This makes installation simple for Linux package managers. This mechanism also solves the minor issue of importing a module from ``site-packages`` with the same name as a missing standard library module. Now, Python will import the ``.missing.py`` file and won't ever look for a *stdlib* module in ``site-packages``. In addition, this method of handling missing *stdlib* modules can be implemented in a succinct, non-intrusive way in CPython, and thus won't add to the complexity of the existing code base. The ``.missing.py`` file can be customized by the packager to provide any desirable behaviour. While we strongly recommend that these files only raise a ModuleNotFoundError_ with an appropriate message, there is no reason to limit customization options. Ideas leading up to this PEP were discussed on the `python-dev mailing list`_. .. _`python-dev mailing list`: https://mail.python.org/pipermail/python-dev/2016-July/145534.html Backwards Compatibility ======================= No problems with backwards compatibility are expected. Distributions that are already patching Python modules to provide custom handling of missing dependencies can continue to do so unhindered. Reference Implementation ======================== Reference implementation can be found on `GitHub`_ and is also accessible in the form of a `patch`_. .. _`GitHub`: https://github.com/torsava/cpython/pull/1 .. _`patch`: https://github.com/torsava/cpython/pull/1.patch References ========== .. [#debian-patch] http://bazaar.launchpad.net/~doko/python/pkg3.5-debian/view/head:/patches/t… Copyright ========= This document has been placed in the public domain. ------------------------------------------------------------------------ Regards, Tomas Orsava [View Less]

20 61

Proposal: Tuple of str with w'list of words'
by Gary Godfrey Dec. 7, 2016

Dec. 7, 2016

Hi, I looked around for a while but didn't see this proposed anywhere. I apologize if I missed an existing discussion. I do a fair amount of work with pandas and data munging. This means that I'm often doing things like: mydf = df[ ['field1', 'field2', 'field3' ] ] This is a little ugly, so if the list is long enough, I do: mydf=df[ 'field1 field2 field3'.split() ] This is a little more readable, but still a bit ugly. What I'm proposing here is: mydf = df[ w'field1 field2 field3' ] … [View More]

6 12

Allow random.choice, random.sample to work on iterators
by Random832 Dec. 3, 2016

Dec. 3, 2016

Currently these functions fail if the supplied object has no len(). There are algorithms for this task that can work on any finite iterator (for example, files as a stream of lines), and the functions could fall back to these if there is no len().

7 10

Add optional defaults to namedtuple
by Jelte Fennema Dec. 2, 2016

Dec. 2, 2016

It would be nice to have a supported way to add defaults to namedtuple, so the slightly hacky solution here does not have to be used: http://stackoverflow.com/a/18348004/2570866 Jelte

5 4