Mailman 3 December 2016 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 16 Dec '20

16 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann 22 Feb '20

22 Feb '20

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum 14 Mar '18

14 Mar '18

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

51 165

https://docs.python.org/fr/ ?
by Julien Palard 02 Feb '17

02 Feb '17

o/ The french translation of the Python Documentation [1][2] has translated 20% of the pageviews of docs.python.org. I think it's the right moment to push it do docs.python.org. So there's some questions ! And I'd like feedback. TL;DR (with my personal choices): - URL may be "http://docs.python.org/fr/" - For localized variations of languages we should use dash and lowercase like "docs.python.org/pt-br/" - po files may be hosted on the python's github - existing script to build doc may be patched to build translations - each translations may crosslink to others - untranslated strings may be visually marked as so I also opened: http://bugs.python.org/issue26546. # Chronology, dependencies The only blocking decision here is the URL, (also reviewing my patch ...), with those two, translated docs can be pushed to production, and the other steps can be discussed and applied one by one. # The URL ## CCTLD vs path vs subdomain I think we should use a variation of "docs.python.org/fr/" for simplicity and clarity. I think we should avoid using CCTLDs as they're sometime hard or near impossible to obtain (may cost a lot of time), also some are expensive, so it's time and money we clearly don't need to loose. Last possibility I see is to use a subdomain, like fr.docs.python.org or docs.fr.python.org but I don't think it's the role / responsibility of the sub-domain to do it. So I'm for docs.python.org/LANGUAGE_TAG/ (without moving current documentation inside a /en/). ## Language tag in path ### Dropping the default locale of a language I personally think we should not show the region in case it's redundant: so to use "fr" instead of "fr-FR", "de" instead of "de-DE", but keeping the possibility to use a locale code when it's not redundant like for "pt-br" or "de-AT" (German ('de') as used in Austria ('AT')). I think so because I don't think we'll have a lot of locale variations (like de-AT, fr-CH, fr-CA, ...) so it will be most of the time redundant (visually heavy, longer to type, longer to read) but we'll still need some locale (pt-BR typically). ### gettext VS IETF language tag format gettext goes by using an underscore between language and locale [3] and IETF goes by using a dash [4][5]. As sphinx is using gettext, and gettext uses underscore we may choose underscore too. But URLs are not here to leak the underlying implementation, and the IETF looks like to be the standard way to represent language tags. Also I visually prefer the dash over the underscore, so I'm for the dash here. ### Lower case vs upper case local tag RFC 5646 section-2.1 tells us language tags are not case sensitive, yet ISO3166-1 recommends that country codes (part of the language tag) be capitalized. I personally prefer the all-lowercase one as paths in URLs typically are lowercase. I searched for `inurl:"pt-br"` to see if I'm not too far away from the usage here and usage seems to agree with me, although there's some "pt-BR" in urls. # Where to host the translated files Currently we're hosting the *po* files in the afpy's (Francophone association for python) [6] github [1] but it may make sense to use (in the generation scripts) a more controlled / restricted clone in the python github, at least to have a better view of who can push on the documentation. We may want to choose between aggregating all translations under the same git repository but I don't feel it's useful. # How to Currently, a python script [7] is used to generate `docs.python.org`, I proposed a patch in [8] to make this script clone and build the french translation too, it's a simple and effective way, I don't think we need more ? Any idea welcome. In our side, we have a Makefile [12] to build the translated doc which is only a thin layer on top of the Sphinx Makefile. So my proposed patch to build scripts "just" delegate the build to our Makefile which itself delegate the hard work to the Sphinx Makefile. # Next ? ## Document how to translate Python I think I can (should) write a documentation on "how to start a Python doc translation project" and "how to migrate existing [9][10][11] python doc translation projects to docs.python.org" if french does goes docs.python.org because it may hopefully motivate people to do the same, and I think our structure is a nice way to do it (A Makefile to generate the doc, all versions translated, people mainly working on latest version, scripts to propagating translations to older version, etc...). ## Crosslinking between existing translations Once the translations are on `docs.python.org`, crosslinks may be established so people on a version can be aware of other version, and easily switch to them. I'm not a UI/UX man but I think we may have a select box right before the existing select box about version, on the top-left corner. Right before because it'll reflect the path: /fr/3.5/ -> [select box fr][select box 3.5]. ## Marking as "untranslated, you can help" the untranslated paragraphs The translations will always need work to follow upstream modifications: marking untranslated paragraphs as so may transform the "Oh they suck, this paragraph is not even translated :-(" to "Hey, nice I can help translating that !". There's an opened sphinx-doc ticket to do so [13] but I have not worked on it yet. As previously said I'm real bad at designing user interfaces, so I don't even visualize how I'd like it to be. [1] http://www.afpy.org/doc/python/3.5/ [2] https://github.com/afpy/python_doc_fr [3] https://www.gnu.org/software/gettext/manual/html_node/Locale-Names.html [4] http://tools.ietf.org/html/rfc5646 [5] https://en.wikipedia.org/wiki/IETF_language_tag [6] http://www.afpy.org/ [7] https://github.com/python/docsbuild-scripts/ [8] http://bugs.python.org/issue26546 [9] http://docs.python.jp/3/ [10] https://github.com/python-doc-ja/python-doc-ja [11] http://docs.python.org.ar/tutorial/3/index.html [12] https://github.com/AFPy/python_doc_fr/blob/master/Makefile [13] https://github.com/sphinx-doc/sphinx/issues/1246 -- Julien Palard

7 8

proposal: "python -m foo" should bind sys.modules['foo']
by Cameron Simpson 18 Jan '17

18 Jan '17

Hello all, This is a writeup of a proposal I floated here: https://mail.python.org/pipermail/python-list/2015-August/694905.html last Sunday. If the response is positive I wish to write a PEP. Briefly, it is a natural expectation in users that the command: python -m module_name ... used to invoke modules in "main program" mode on the command line imported the module as "module_name". It does not, it imports it as "__main__". An import within the program of "module_name" makes a new instance of the module, which causes cognitive dissonance and has the side effect that now the program has two instances of the module. What I propose is that the above command line _should_ bind sys.modules['module_name'] as well as binding '__main__' as it does currently. I'm proposing that the python -m option have this effect (python pseudocode): % python -m module.name ... runs: # pseudocode, with values hardwired for clarity import sys M = new_empty_module(name='__main__', qualname='module.name') sys.modules['__main__'] = M sys.modules['module.name'] = M # load the module code from wherever (not necessarily a file - CPython # already must do this phase) M.execfile('/path/to/module/name.py') Specificly, this would have the following two changes to current practice: 1) the module is imported _once_, and bound to both its canonical name and also to __main__. 2) imported modules acquire a new attribute __qualname__ (analogous to the recent __qualname__ on functions). This is always the conanoical name of the module as resolved by the importer. For most modules __name__ will be the same as __qualname__, but for the "main" module __name__ will be '__main__'. This change has the following advantages: The current standard boilerplate: if __name__ == '__main__': ... invoke "main program" here ... continues to work unchanged. Importantly, if the program then issues "import module_name", it is already there and the existing instance is found and used. The thread referenced above outlines my most recent encounter with this and the trouble it caused me. Followup messages include some support for this proposed change, and some criticism. The critiquing article included some workarounds for this multiple module situation, but they were (1) somewhat dependent on modules coming from a file pathname and (2) cumbersome and require every end user to adopt these changes if affected by the situation. I'd like to avoid that. Cheers, Cameron Simpson <cs(a)zip.com.au> The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. - George Bernard Shaw

5 13

api suggestions for the cProfile module
by Thane Brimhall 11 Jan '17

11 Jan '17

I use cProfile a lot, and would like to suggest three backwards-compatible improvements to the API. 1: When using cProfile on a specific piece of code I often use the enable() and disable() methods. It occurred to me that this would be an obvious place to use a context manager. 2: Enhance the `print_stats` method on Profile to accept more options currently available only through the pstats.Stats class. For example, strip_dirs could be a boolean argument, and limit could accept an int. This would reduce the number of cases you'd need to use the more complex API. 3: I often forget which string keys are available for sorting. It would be nice to add an enum for these so a user could have their linter and IDE check that value pre-runtime. Since it would subclass `str` and `Enum` it would still work with all currently existing code. The current documentation contains the following code: import cProfile, pstats, io pr = cProfile.Profile() pr.enable() # ... do something ... pr.disable() s = io.StringIO() sortby = 'cumulative' ps = pstats.Stats(pr, stream=s).sort_stats(sortby) ps.print_stats() print(s.getvalue()) While the code below doesn't exactly match the functionality above (eg. not using StringIO), I envision the context manager working like this, along with some adjustments on how to get the stats from the profiler: import cProfile, pstats with cProfile.Profile() as pr: # ... do something ... pr.print_stats(sort=pstats.Sort.cumulative, limit=10, strip_dirs=True) As you can see, the code is shorter and somewhat more self-documenting. The best thing about these suggestions is that as far as I can tell they would be backwards-compatible API additions. What do you think? Thank you in advance for your time! /Thane

5 6

Re: [Python-ideas] New PyThread_tss_ C-API for CPython
by Masayuki YAMAMOTO 07 Jan '17

07 Jan '17

I have read the discussion and I'm sure that use structure as Py_tss_t instead of platform-specific data type. Just as Steve said that Py_tss_t should be genuinely treated as an opaque type, the key state checking should provide macros or inline functions with name like PyThread_tss_is_created. Well, I'd resolve the specification a bit more :) If PyThread_tss_create is called with the created key, it is no-op but which the function should succeed or fail? In my opinion, It is better to return a failure because it is a high possibility that the code is incorrect for multiple callings of PyThread_tss_create for One key. In this opinion PyThread_tss_is_created should return a value as follows: (A) False while from after defining with Py_tss_NEED_INIT to before calling PyThread_tss_create (B) True after calling PyThread_tss_create succeeded (C) Unchanging before and after calling PyThread_tss_create failed (D) False after calling PyThread_tss_delete regardless of timing (E) For other functions, the return value of PyThread_tss_is_created does not change before and after calling I think that it is better to write a test about the state of the Py_tss_t. Kind regards, Masayuki 2016-12-31 2:38 GMT+09:00 Erik Bray <erik.m.bray(a)gmail.com>: > On Fri, Dec 30, 2016 at 5:05 PM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > On 29 December 2016 at 22:12, Erik Bray <erik.m.bray(a)gmail.com> wrote: > >> > >> 1) CPython's TLS: Defines -1 as an uninitialized key (by fact of the > >> implementation--that the keys are integers starting from zero) > >> 2) pthreads: Does not definite an uninitialized default value for > >> keys, for reasons described at [1] under "Non-Idempotent Data Key > >> Creation". I understand their reasoning, though I can't claim to know > >> specifically what they mean when they say that some implementations > >> would require the mutual-exclusion to be performed on > >> pthread_getspecific() as well. I don't know that it applies here. > > > > > > That section is a little weird, as they describe two requests (one for a > > known-NULL default value, the other for implicit synchronisation of key > > creation to prevent race conditions), and only provide the justification > for > > rejecting one of them (the second one). > > Right, that is confusing to me as well. I'm guessing the reason for > rejecting the first is in part a way to force us to recognize the > second issue. > > > If I've understood correctly, the situation they're worried about there > is > > that pthread_key_create() has to be called at least once-per-process, but > > must be called before *any* call to pthread_getspecific or > > pthread_setspecific for a given key. If you do "implicit init" rather > than > > requiring the use of an explicit mechanism like pthread_once (or our own > > Py_Initialize and module import locks), then you may take a small > > performance hit as either *every* thread then has to call > > pthread_key_create() to ensure the key exists before using it, or else > > pthread_getspecific() and pthread_setspecific() have to become > potentially > > blocking calls. Neither of those is desirable, so it makes sense to leave > > that part of the problem to the API client. > > > > In our case, we don't want the implicit synchronisation, we just want the > > known-NULL default value so the "Is it already set?" check can be moved > > inside the library function. > > Okay, we're on the same page here then. I just wanted to make sure > there wasn't anything else I was missing in Python's case. > > >> 3) windows: The return value of TlsAlloc() is a DWORD (unsigned int) > >> and [2] states that its value should be opaque. > >> > >> So in principle we can cover all cases with an opaque struct that > >> contains, as its first member, an is_initialized flag. The tricky > >> part is how to initialize the rest of the struct (containing the > >> underlying implementation-specific key). For 1) and 3) it doesn't > >> matter--it can just be zero. For 2) it's trickier because there's no > >> defined constant value to initialize a pthread_key_t to. > >> > >> Per Nick's suggestion this can be worked around by relying on C99's > >> initialization semantics. Per [3] section 6.7.8, clause 21: > >> > >> """ > >> If there are fewer initializers in a brace-enclosed list than there > >> are elements or members of an aggregate, or fewer characters in a > >> string literal used to initialize an array of known size than there > >> are elements in the array, the remainder of the aggregate shall be > >> initialized implicitly the same as objects that have static storage > >> duration. > >> """ > >> > >> How objects with static storage are initialized is described in the > >> previous page under clause 10, but in practice it boils down to what > >> you would expect: Everything is initialized to zero, including nested > >> structs and arrays. > >> > >> So as long as we can use this feature of C99 then I think that's the > >> best approach. > > > > > > > > I checked PEP 7 to see exactly which features we've added to the > approved C > > dialect, and designated initialisers are already on the list: > > https://gcc.gnu.org/onlinedocs/gcc/Designated-Inits.html > > > > So I believe that would allow the initializer to be declared as something > > like: > > > > #define Py_tss_NEEDS_INIT {.is_initialized = false} > > Great! One could argue about whether or not the designated > initializer syntax also incorporates omitted fields, but it would seem > strange to insist that it doesn't. > > Have a happy new year, > > Erik >

2 2

incremental hashing in __hash__
by jab＠math.brown.edu 06 Jan '17

06 Jan '17

Suppose you have implemented an immutable Position type to represent the state of a game played on an MxN board, where the board size can grow quite large. Or suppose you have implemented an immutable, ordered collection type. For example, the collections-extended package provides a frozensetlist[1]. One of my own packages provides a frozen, ordered bidirectional mapping type.[2] These types should be hashable so that they can be inserted into sets and mappings. The order-sensitivity of the contents prevents them from using the built-in collections.Set._hash() helper in their __hash__ implementations, to keep from unnecessarily causing hash collisions for objects that compare unequal due only to having a different ordering of the same set of contained items. According to https://docs.python.org/3/reference/datamodel.html#object.__hash__ : """ it is advised to mix together the hash values of the components of the object that also play a part in comparison of objects by packing them into a tuple and hashing the tuple. Example: def __hash__(self): return hash((self.name, self.nick, self.color)) """ Applying this advice to the use cases above would require creating an arbitrarily large tuple in memory before passing it to hash(), which is then just thrown away. It would be preferable if there were a way to pass multiple values to hash() in a streaming fashion, such that the overall hash were computed incrementally, without building up a large object in memory first. Should there be better support for this use case? Perhaps hash() could support an alternative signature, allowing it to accept a stream of values whose combined hash would be computed incrementally in *constant* space and linear time, e.g. "hash(items=iter(self))". In the meantime, what is the best way to incrementally compute a good hash value for such objects using built-in Python routines? (As a library author, it would be preferable to use a routine with explicit support for computing a hash incrementally, rather than having to worry about how to correctly combine results from multiple calls to hash(contained_item) in library code. (Simply XORing such results together would not be order-sensitive, and so wouldn't work.) Using a routine with explicit support for incremental hashing would allow libraries to focus on doing one thing well.[3,4,5]) I know that hashlib provides algorithms that support incremental hashing, but those use at least 128 bits. Since hash() throws out anything beyond sys.hash_info.hash_bits (e.g. 64) bits, anything in hashlib seems like overkill. Am I right in thinking that's the wrong tool for the job? On the other hand, would binascii.crc32 be suitable, at least for 32-bit systems? (And is there some 64-bit incremental hash algorithm available for 64-bit systems? It seems Python has no support for crc64 built in.) For example: import binascii, struct class FrozenOrderedCollection: def __hash__(self): if hasattr(self, '__hashval'): # Computed lazily. return self.__hashval hv = crc32(b'FrozenOrderedCollection') for i in self: hv = binascii.crc32(struct.pack('@l', hash(i)), hv) hv &= 0xffffffff self.__hashval = hv return hv Note that this example illustrates two other common requirements of these use cases: (i) lazily computing the hash value on first use, and then caching it for future use (ii) priming the overall hash value with some class-specific initial value, so that if an instance of a different type of collection, which comprised the same items but which compared unequal, were to compute its hash value out of the same constituent items, we make sure our hash value differs. (On that note, should the documentation in https://docs.python.org/3/reference/datamodel.html#object.__hash__ quoted above be updated to add this advice? The current advice to "return hash((self.name, self.nick, self.color))" would cause a hash collision with a tuple of the same values, even though the tuple should presumably compare unequal with this object.) To summarize these questions: 1. Should hash() add support for incremental hashing? 2. In the meantime, what is the best way to compute a hash of a combination of many values incrementally (in constant space and linear time), using only what's available in the standard library? Ideally there is some routine available that uses exactly hash_info.hash_bits number of bits, and that does the combining of incremental results for you. 3. Should the https://docs.python.org/3/reference/datamodel.html#object.__hash__ documentation be updated to include suitable advice for these use cases, in particular, that the overall hash value should be computed lazily, incrementally, and should be primed with a class-unique value? Thanks in advance for a helpful discussion, and best wishes. Josh References: [1] http://collections-extended.lenzm.net/api.html#collections_extended.frozens… [2] https://bidict.readthedocs.io/en/dev/api.html#bidict.frozenorderedbidict [3] http://stackoverflow.com/questions/2909106/python-whats-a-correct-and-good-… [4] http://stackoverflow.com/a/2909572/161642 [5] http://stackoverflow.com/a/27952689/161642

18 55

AtributeError inside __get__
by Zahari Dim 31 Dec '16

31 Dec '16

Hi, The other day I came across a particularly ugly bug. A simplified case goes like: class X: @property def y(self): return self.nonexisting hasattr(X(),'y') This returns False because hasattr calls the property which in turn raises an AttributeError which is used to determine that the property doesn't exist, even if it does. This is arguably unexpected and surprising and can be very difficult to understand if it happens within a large codebase. Given the precedent with generator_stop, which solves a similar problem for StopIteration, I was wondering if it would be possible to have the __get__ method convert the AttributeErrors raised inside it to RuntimeErrors. The situation with this is a little more complicated because there could be a (possibly strange) where one might want to raise an AttributeError inside __get__. But maybe the specification can be changed so either `raise ForceAttributeError()` or `return NotImplemented` achieves the same effect. Merry Christmas! Zahari.

7 12

New PyThread_tss_ C-API for CPython
by Erik Bray 30 Dec '16

30 Dec '16

Greetings all, I wanted to bring attention to an issue that's been languishing on the bug tracker since last year, which I think would best be addressed by changes to CPython's C-API. The original issue is at http://bugs.python.org/issue25658, but I have made an effort below in a sort of proto-PEP to summarize the problem and the proposed solution. I haven't written this up in the proper PEP format because I want to see if the idea has some broader support first, and it's also not clear to me whether C-API changes (especially to undocumented APIs) even require their own PEP. Abstract ======== The proposal is to add a new Thread Local Storage (TLS) API to CPython which would supersede use of the existing TLS API within the CPython interpreter, while deprecating the existing API. Because the existing TLS API is only used internally (it is not mentioned in the documentation, and the header that defines it, pythread.h, is not included in Python.h either directly or indirectly), this proposal probably only affects CPython, but might also affect other interpreter implementations (PyPy?) that implement parts of the CPython API. Specification ============= The current API for TLS used inside the CPython interpreter consists of 5 functions: PyAPI_FUNC(int) PyThread_create_key(void) PyAPI_FUNC(void) PyThread_delete_key(int key) PyAPI_FUNC(int) PyThread_set_key_value(int key, void *value) PyAPI_FUNC(void *) PyThread_get_key_value(int key) PyAPI_FUNC(void) PyThread_delete_key_value(int key) These would be superseded with a new set of analogous functions: PyAPI_FUNC(int) PyThread_tss_create(Py_tss_t *key) PyAPI_FUNC(void) PyThread_tss_delete(Py_tss_t key) PyAPI_FUNC(int) PyThread_tss_set(Py_tss_t key, void *value) PyAPI_FUNC(void *) PyThread_tss_get(Py_tss_t key) PyAPI_FUNC(void) PyThread_tss_delete_value(Py_tss_t key) and includes the definition of a new type Py_tss_t--any opaque type the specification of which is not given here, and may depend on the underlying TLS implementation. The new PyThread_tss_ functions are almost exactly analogous to their original counterparts with a minor difference: Whereas PyThread_create_key takes no arguments and returns a TLS key as an int, PyThread_tss_create takes a Py_tss_t* as an argument, and returns a Py_tss_t by pointer--the int return value is a status, returning zero on success and non-zero on failure. Further, the old PyThread_*_key* functions will be marked as deprecated. Additionally, the pthread implementations of the old PyThread_*_key* functions will either fail or be no-ops on platforms where sizeof(pythead_t) != sizeof(int). Motivation ========== The primary problem at issue here is the type of the keys (int) used for TLS values, as defined by the original PyThread TLS API. The original TLS API was added to Python by GvR back in 1997, and at the time the key used to represent a TLS value was an int, and so it has been to this day. This used CPython's own TLS implementation, the current generation of which can still be found, largely unchanged, in Python/thread.c. Support for implementation of the API on top of native thread implementations (NT and pthreads) was added much later, and the built-in implementation may still be used on other platforms. The problem with the choice of int to represent a TLS key, is that while it was fine for CPython's internal TLS implementation, and happens to be fine for NT (which uses DWORD), it is not compatible the POSIX standard for the pthreads API, which defines pthread_key_t as an opaque type not further designed by the standard (as with Py_tss_t described above). This leaves it up to the underlying implementation how a pthread_key_t value is used to look thread-specific data. This has not generally been a problem for Python's API, as it just happens that on Linux pthread_key_t is just defined as an unsigned int, and so is fully compatible with Python's TLS API--pthread_key_t's created by pthread_create_key can be freely cast to ints and back (well, not really, even this has issues as pointed out by issue #22206). However, as issue #25658 points out there are at least some platforms (namely Cygwin, CloudABI, but likely others as well) which have otherwise modern and POSIX-compliant pthreads implementations, but are not compatible with Python's API because their pthread_key_t is defined in a way that cannot be safely cast to int. In fact, the possibility of running into this problem was raised by MvL at the time pthreads TLS was added [1]. It could be argued that PEP-11 makes specific requirements for supporting a new, not otherwise officially-support platform (such as CloudABI), and that the status of Cygwin support is currently dubious. However, this places a very barrier to supporting platforms that are otherwise Linux- and/or POSIX-compatible and where CPython might otherwise "just work" except for this one hurdle which Python itself imposes by way of an API that is not compatible with POSIX (and in fact makes invalid assumptions about pthreads). Rationale for Proposed Solution =============================== The use of an opaque type (Py_tss_t) to key TLS values allows the API to be compatible, at least in this regard, with CPython's internal TLS implementation, as well as all present (NT and posix) and future (C11?) native TLS implementations supported by CPython, as it allows the definition of Py_tss_t to depend on the underlying implementation. A new API must be introduced, rather than changing the function signatures of the current API, in order to maintain backwards compatibility. The new API also more clearly groups together these related functions under a single name prefix, "PyThread_tss_". The "tss" in the name stands for "thread-specific storage", and was influenced by the naming and design of the "tss" API that is part of the C11 threads API. However, this is in no way meant to imply compatibility with or support for the C11 threads API, or signal any future intention of supporting C11--it's just the influence for the naming and design. Changing PyThread_create_key to immediately return a failure status on systems using pthreads where sizeof(int) != sizeof(pthread_key_t) is intended as a sanity check: Currently, PyThread_create_key will report initial success on such systems, but attempts to use the returned key are likely to fail. Although in practice this failure occurs quickly during interpreter startup, it's better to fail immediately at the source of failure (PyThread_create_key) rather than sometime later when use of an invalid key is attempted. Rejected Ideas ============== * Do nothing: The status quo is fine because it works on Linux, and platforms wishing to be supported by CPython should follow the requirements of PEP-11. As explained above, while this would be a fair argument if CPython were being to asked to make changes to support particular quirks of a specific platform, in this case the platforms in question are only asking to fix a quirk of CPython that prevents it from being used to its full potential on those platforms. The fact that the current implementation happens to work on Linux is a happy accident, and there's no guarantee that will never change. * Affected platforms should just configure Python --without-threads: This is a possible temporary workaround to the issue, but only that. Python should not be hobbled on affected platforms despite them being otherwise perfectly capable of running multi-threaded Python. * Affected platforms should not define Py_HAVE_NATIVE_TLS: This is a more acceptable alternative to the previous idea, and in fact there is a patch to do just that [2]. However, CPython's internal TLS implementation being "slower and clunkier" in general than native implementations still needlessly hobbles performance on affected platforms. At least one other module (tracemalloc) is also broken if Python is built without Py_HAVE_NATIVE_TLS. * Keep the existing API, but work around the issue by providing a mapping from pthread_key_t values to ints. A couple attempts were made at this [3] [4], but this only injects needless complexity and overhead into performance-critical code on platforms that are not currently affected by this issue (such as Linux). Even if use of this workaround were made conditional on platform compatibility, it introduces platform-specific code to maintain, and still has the problem of the previous rejected ideas of needlessly hobbling performance on affected platforms. Implementation ============== An initial version of a patch [5] is available on the bug tracker for this issue. The patch is proposed and written by Masayuki Yamamoto, who should be considered a co-author of this proto-PEP, though I have not consulted directly with him in writing this. If he's reading, he should chime in in case I've misrepresented anything. If you've made it this far, thanks for reading and thank you for your consideration, Erik [1] https://bugs.python.org/msg116292 [2] http://bugs.python.org/file45548/configure-pthread_key_t.patch [3] http://bugs.python.org/file44269/issue25658-1.patch [4] http://bugs.python.org/file44303/key-constant-time.diff [5] http://bugs.python.org/file45763/pythread-tss.patch

7 20