Mailman 3 November 2016 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 16 Dec '20

16 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann 21 Feb '20

21 Feb '20

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

Implicit string literal concatenation considered harmful?
by Guido van Rossum 14 Mar '18

14 Mar '18

I just spent a few minutes staring at a bug caused by a missing comma -- I got a mysterious argument count error because instead of foo('a', 'b') I had written foo('a' 'b'). This is a fairly common mistake, and IIRC at Google we even had a lint rule against this (there was also a Python dialect used for some specific purpose where this was explicitly forbidden). Now, with modern compiler technology, we can (and in fact do) evaluate compile-time string literal concatenation with the '+' operator, so there's really no reason to support 'a' 'b' any more. (The reason was always rather flimsy; I copied it from C but the reason why it's needed there doesn't really apply to Python, as it is mostly useful inside macros.) Would it be reasonable to start deprecating this and eventually remove it from the language? -- --Guido van Rossum (python.org/~guido)

51 165

https://docs.python.org/fr/ ?
by Julien Palard 02 Feb '17

02 Feb '17

o/ The french translation of the Python Documentation [1][2] has translated 20% of the pageviews of docs.python.org. I think it's the right moment to push it do docs.python.org. So there's some questions ! And I'd like feedback. TL;DR (with my personal choices): - URL may be "http://docs.python.org/fr/" - For localized variations of languages we should use dash and lowercase like "docs.python.org/pt-br/" - po files may be hosted on the python's github - existing script to build doc may be patched to build translations - each translations may crosslink to others - untranslated strings may be visually marked as so I also opened: http://bugs.python.org/issue26546. # Chronology, dependencies The only blocking decision here is the URL, (also reviewing my patch ...), with those two, translated docs can be pushed to production, and the other steps can be discussed and applied one by one. # The URL ## CCTLD vs path vs subdomain I think we should use a variation of "docs.python.org/fr/" for simplicity and clarity. I think we should avoid using CCTLDs as they're sometime hard or near impossible to obtain (may cost a lot of time), also some are expensive, so it's time and money we clearly don't need to loose. Last possibility I see is to use a subdomain, like fr.docs.python.org or docs.fr.python.org but I don't think it's the role / responsibility of the sub-domain to do it. So I'm for docs.python.org/LANGUAGE_TAG/ (without moving current documentation inside a /en/). ## Language tag in path ### Dropping the default locale of a language I personally think we should not show the region in case it's redundant: so to use "fr" instead of "fr-FR", "de" instead of "de-DE", but keeping the possibility to use a locale code when it's not redundant like for "pt-br" or "de-AT" (German ('de') as used in Austria ('AT')). I think so because I don't think we'll have a lot of locale variations (like de-AT, fr-CH, fr-CA, ...) so it will be most of the time redundant (visually heavy, longer to type, longer to read) but we'll still need some locale (pt-BR typically). ### gettext VS IETF language tag format gettext goes by using an underscore between language and locale [3] and IETF goes by using a dash [4][5]. As sphinx is using gettext, and gettext uses underscore we may choose underscore too. But URLs are not here to leak the underlying implementation, and the IETF looks like to be the standard way to represent language tags. Also I visually prefer the dash over the underscore, so I'm for the dash here. ### Lower case vs upper case local tag RFC 5646 section-2.1 tells us language tags are not case sensitive, yet ISO3166-1 recommends that country codes (part of the language tag) be capitalized. I personally prefer the all-lowercase one as paths in URLs typically are lowercase. I searched for `inurl:"pt-br"` to see if I'm not too far away from the usage here and usage seems to agree with me, although there's some "pt-BR" in urls. # Where to host the translated files Currently we're hosting the *po* files in the afpy's (Francophone association for python) [6] github [1] but it may make sense to use (in the generation scripts) a more controlled / restricted clone in the python github, at least to have a better view of who can push on the documentation. We may want to choose between aggregating all translations under the same git repository but I don't feel it's useful. # How to Currently, a python script [7] is used to generate `docs.python.org`, I proposed a patch in [8] to make this script clone and build the french translation too, it's a simple and effective way, I don't think we need more ? Any idea welcome. In our side, we have a Makefile [12] to build the translated doc which is only a thin layer on top of the Sphinx Makefile. So my proposed patch to build scripts "just" delegate the build to our Makefile which itself delegate the hard work to the Sphinx Makefile. # Next ? ## Document how to translate Python I think I can (should) write a documentation on "how to start a Python doc translation project" and "how to migrate existing [9][10][11] python doc translation projects to docs.python.org" if french does goes docs.python.org because it may hopefully motivate people to do the same, and I think our structure is a nice way to do it (A Makefile to generate the doc, all versions translated, people mainly working on latest version, scripts to propagating translations to older version, etc...). ## Crosslinking between existing translations Once the translations are on `docs.python.org`, crosslinks may be established so people on a version can be aware of other version, and easily switch to them. I'm not a UI/UX man but I think we may have a select box right before the existing select box about version, on the top-left corner. Right before because it'll reflect the path: /fr/3.5/ -> [select box fr][select box 3.5]. ## Marking as "untranslated, you can help" the untranslated paragraphs The translations will always need work to follow upstream modifications: marking untranslated paragraphs as so may transform the "Oh they suck, this paragraph is not even translated :-(" to "Hey, nice I can help translating that !". There's an opened sphinx-doc ticket to do so [13] but I have not worked on it yet. As previously said I'm real bad at designing user interfaces, so I don't even visualize how I'd like it to be. [1] http://www.afpy.org/doc/python/3.5/ [2] https://github.com/afpy/python_doc_fr [3] https://www.gnu.org/software/gettext/manual/html_node/Locale-Names.html [4] http://tools.ietf.org/html/rfc5646 [5] https://en.wikipedia.org/wiki/IETF_language_tag [6] http://www.afpy.org/ [7] https://github.com/python/docsbuild-scripts/ [8] http://bugs.python.org/issue26546 [9] http://docs.python.jp/3/ [10] https://github.com/python-doc-ja/python-doc-ja [11] http://docs.python.org.ar/tutorial/3/index.html [12] https://github.com/AFPy/python_doc_fr/blob/master/Makefile [13] https://github.com/sphinx-doc/sphinx/issues/1246 -- Julien Palard

7 8

proposal: "python -m foo" should bind sys.modules['foo']
by Cameron Simpson 17 Jan '17

17 Jan '17

Hello all, This is a writeup of a proposal I floated here: https://mail.python.org/pipermail/python-list/2015-August/694905.html last Sunday. If the response is positive I wish to write a PEP. Briefly, it is a natural expectation in users that the command: python -m module_name ... used to invoke modules in "main program" mode on the command line imported the module as "module_name". It does not, it imports it as "__main__". An import within the program of "module_name" makes a new instance of the module, which causes cognitive dissonance and has the side effect that now the program has two instances of the module. What I propose is that the above command line _should_ bind sys.modules['module_name'] as well as binding '__main__' as it does currently. I'm proposing that the python -m option have this effect (python pseudocode): % python -m module.name ... runs: # pseudocode, with values hardwired for clarity import sys M = new_empty_module(name='__main__', qualname='module.name') sys.modules['__main__'] = M sys.modules['module.name'] = M # load the module code from wherever (not necessarily a file - CPython # already must do this phase) M.execfile('/path/to/module/name.py') Specificly, this would have the following two changes to current practice: 1) the module is imported _once_, and bound to both its canonical name and also to __main__. 2) imported modules acquire a new attribute __qualname__ (analogous to the recent __qualname__ on functions). This is always the conanoical name of the module as resolved by the importer. For most modules __name__ will be the same as __qualname__, but for the "main" module __name__ will be '__main__'. This change has the following advantages: The current standard boilerplate: if __name__ == '__main__': ... invoke "main program" here ... continues to work unchanged. Importantly, if the program then issues "import module_name", it is already there and the existing instance is found and used. The thread referenced above outlines my most recent encounter with this and the trouble it caused me. Followup messages include some support for this proposed change, and some criticism. The critiquing article included some workarounds for this multiple module situation, but they were (1) somewhat dependent on modules coming from a file pathname and (2) cumbersome and require every end user to adopt these changes if affected by the situation. I'd like to avoid that. Cheers, Cameron Simpson <cs(a)zip.com.au> The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself. Therefore all progress depends on the unreasonable man. - George Bernard Shaw

5 13

Importing public symbols and simultainiously privatizing them, is too noisy
by Rick Johnson 28 Dec '16

28 Dec '16

Has anyone else found this to be too syntactically noisy? from module import Foo as _Foo, bar as _bar That is horrifically noisy IMO. The problem is, how do we remove the noise without sacrificing intuitiveness? My first idea was to do this: from module import_private Foo, bar And while it's self explanatory, it's also too long. So i thought... from module _import Foo, bar I'm leaning more towards the latter, but i'm not loving it either. Any ideas?

9 17

singledispatch for instance methods
by Tim Mitchell 26 Dec '16

26 Dec '16

Hi All, We have a modified version of singledispatch at work which works for methods as well as functions. We have open-sourced it as methoddispatch (pypi: https://pypi.python.org/pypi/methoddispatch). IMHO I thought it would make a nice addition to python stdlib. What does everyone else think?

2 2

(no subject)
by victor rajewski 24 Dec '16

24 Dec '16

I teach a computing subject to high school students using Python as our primary language. One thing that often causes confusion at the start is error messages/exceptions. I think it would lower entry point quite a bit by making error messages more readable to novices. Recent research found reduced frequency of overall errors, errors per student, and repeated errors when error messages were enhanced [1]. This could be implemented by a command-line switch when running python, or a different python executable (along the lines of pythonw.exe vs python.exe) such as python-easy. A simpler error message might get rid of excess information which can confuse the novice (such info might be dumped to a file to allow more advanced users to debug), provide a brief description of what the error might mean, and/or how you might fix it. So a SyntaxError might tell the user that they've probably forgotten a :, and a NameError might say that the item has been defined yet, or they've made a typo. A couple of examples follow: Traceback (most recent call last): File "foo.py", line 2, in <module> l[10]=14 IndexError: list assignment index out of range A better message might be: You tried to use l[10] when l is only 4 elements long. You can add items to l using l.append(value), or check your index value to make sure that's really the position you wanted to access. Traceback (most recent call last): File "foo.py", line 2, in <module> while name != "quit" and reponse != "quit": NameError: name 'reponse' is not defined A better message might be: You're trying to use the value of 'reponse', but that variable hasn't got a value yet. You can give it a value earlier in the code, or it could be a typo. You have a variable called 'response' - is that the one you meant? Traceback (most recent call last): File "foo.py", line 2, in <module> print(length(l)) NameError: name 'length' is not defined A better message might be: Python doesn't recognise the function "length". Did you mean len?' File "foo.py", line 2 for i in l ^ SyntaxError: invalid syntax A better message might be: You have a for loop without a : (colon). Try adding a colon at the end of the line. Any thoughts? [1]: http://researchrepository.ucd.ie/handle/10197/7583 -- Victor Rajewski

23 40

PEP: Distributing a Subset of the Standard Library
by Tomas Orsava 07 Dec '16

07 Dec '16

Hi! We have written a draft PEP entitled "Distributing a Subset of the Standard Library" that aims to standardize and improve how Python handles omissions from its standard library. This is relevant both to Python itself as well as to Linux and other distributions that are packaging it, as they already separate out parts of the standard library into optionally installable packages. Ideas leading up to this PEP were discussed on the python-dev mailing list: https://mail.python.org/pipermail/python-dev/2016-July/145534.html Rendered PEP: https://fedora-python.github.io/pep-drafts/pep-A.html ------------------------------------------------------------------------ Abstract ======== Python is sometimes being distributed without its full standard library. However, there is as of yet no standardized way of dealing with importing a missing standard library module. This PEP proposes a mechanism for identifying which standard library modules are missing and puts forth a method of how attempts to import a missing standard library module should be handled. Motivation ========== There are several use cases for including only a subset of Python's standard library. However, there is so far no formal specification of how to properly implement distribution of a subset of the standard library. Namely, how to safely handle attempts to import a missing *stdlib* module, and display an informative error message. CPython ------- When one of Python standard library modules (such as ``_sqlite3``) cannot be compiled during a Python build because of missing dependencies (e.g. SQLite header files), the module is simply skipped. If you then install this compiled Python and use it to try to import one of the missing modules, Python will go through the ``sys.path`` entries looking for it. It won't find it among the *stdlib* modules and thus it will continue onto ``site-packages`` and fail with a ModuleNotFoundError_ if it doesn't find it. .. _ModuleNotFoundError: https://docs.python.org/3.7/library/exceptions.html#ModuleNotFoundError This can confuse users who may not understand why a cleanly built Python is missing standard library modules. Linux and other distributions ----------------------------- Many Linux and other distributions are already separating out parts of the standard library to standalone packages. Among the most commonly excluded modules are the ``tkinter`` module, since it draws in a dependency on the graphical environment, and the ``test`` package, as it only serves to test Python internally and is about as big as the rest of the standard library put together. The methods of omission of these modules differ. For example, Debian patches the file ``Lib/tkinter/__init__.py`` to envelop the line ``import _tkinter`` in a *try-except* block and upon encountering an ``ImportError`` it simply adds the following to the error message: ``please install the python3-tk package`` [#debian-patch]_. Fedora and other distributions simply don't include the omitted modules, potentially leaving users baffled as to where to find them. Specification ============= When, for any reason, a standard library module is not to be included with the rest, a file with its name and the extension ``.missing.py`` shall be created and placed in the directory the module itself would have occupied. This file can contain any Python code, however, it *should* raise a ModuleNotFoundError_ with a helpful error message. Currently, when Python tries to import a module ``XYZ``, the ``FileFinder`` path hook goes through the entries in ``sys.path``, and in each location looks for a file whose name is ``XYZ`` with one of the valid suffixes (e.g. ``.so``, ..., ``.py``, ..., ``.pyc``). The suffixes are tried in order. If none of them are found, Python goes on to try the next directory in ``sys.path``. The ``.missing.py`` extension will be added to the end of the list, and configured to be handled by ``SourceFileLoader``. Thus, if a module is not found in its proper location, the ``XYZ.missing.py`` file is found and executed, and further locations are not searched. The CPython build system will be modified to generate ``.missing.py`` files for optional modules that were not built. Rationale ========= The mechanism of handling missing standard library modules through the use of the ``.missing.py`` files was chosen due to its advantages both for CPython itself and for Linux and other distributions that are packaging it. The missing pieces of the standard library can be subsequently installed simply by putting the module files in their appropriate location. They will then take precedence over the corresponding ``.missing.py`` files. This makes installation simple for Linux package managers. This mechanism also solves the minor issue of importing a module from ``site-packages`` with the same name as a missing standard library module. Now, Python will import the ``.missing.py`` file and won't ever look for a *stdlib* module in ``site-packages``. In addition, this method of handling missing *stdlib* modules can be implemented in a succinct, non-intrusive way in CPython, and thus won't add to the complexity of the existing code base. The ``.missing.py`` file can be customized by the packager to provide any desirable behaviour. While we strongly recommend that these files only raise a ModuleNotFoundError_ with an appropriate message, there is no reason to limit customization options. Ideas leading up to this PEP were discussed on the `python-dev mailing list`_. .. _`python-dev mailing list`: https://mail.python.org/pipermail/python-dev/2016-July/145534.html Backwards Compatibility ======================= No problems with backwards compatibility are expected. Distributions that are already patching Python modules to provide custom handling of missing dependencies can continue to do so unhindered. Reference Implementation ======================== Reference implementation can be found on `GitHub`_ and is also accessible in the form of a `patch`_. .. _`GitHub`: https://github.com/torsava/cpython/pull/1 .. _`patch`: https://github.com/torsava/cpython/pull/1.patch References ========== .. [#debian-patch] http://bazaar.launchpad.net/~doko/python/pkg3.5-debian/view/head:/patches/t… Copyright ========= This document has been placed in the public domain. ------------------------------------------------------------------------ Regards, Tomas Orsava

20 61

Proposal: Tuple of str with w'list of words'
by Gary Godfrey 07 Dec '16

07 Dec '16

Hi, I looked around for a while but didn't see this proposed anywhere. I apologize if I missed an existing discussion. I do a fair amount of work with pandas and data munging. This means that I'm often doing things like: mydf = df[ ['field1', 'field2', 'field3' ] ] This is a little ugly, so if the list is long enough, I do: mydf=df[ 'field1 field2 field3'.split() ] This is a little more readable, but still a bit ugly. What I'm proposing here is: mydf = df[ w'field1 field2 field3' ] This would be identical in all ways (compile-time) to: mydf = df[ ('field1', 'field2', 'field3') ] This should work with all the python quote variations (w''', w""", etc). The only internal escapes are \\ indicating a \ and <backslash><space> indicating a non-splitting space: songs = w'My\ Bloody\ Valentine Blue\ Suede\ Shoes' One question is whether to have w'' be a list or a tuple. I leaned slightly towards tuple because it's faster on internal loops: In [1]: %timeit a=('this','is','a','test') 100000000 loops, best of 3: 11.3 ns per loop In [2]: %timeit a=['this','is','a','test'] 10000000 loops, best of 3: 74.3 ns per loop However, I mostly see lists used in the data science community, so it's a little less convenient: other_fields = df.columns[-3:] new_columns = w'field1 field2' + other_fields # ERROR - can't concatenate list to tuple new_columns = list(w'field1 field2') + other_fields I honestly could go either way with lists or tuples. Other Languages: perl has the qw operator: @a = qw(field1 field2 field3); ruby has %w a=%w{field1 field2} Thanks for reading this far :-) Regards, Gary Godfrey Austin, TX

6 12