Mailman 3 March 2014 - Python-ideas

re-organizing the sys module
by Eric Snow March 14, 2014

March 14, 2014

The sys module is perhaps the most core module to Python and one of the oldest. It has grown relatively organically over the years and its contents could use a little clean up. Similar proposals have come up before but without enough interest to keep them rolling. My motivation for cleaning up sys is a combination of things: * I'm working on a proposal that may include adding a sub-namespace to sys (similar to what we did with sys.implementation) and removing-ish some attributes. * The namespace is large enough that scanning quickly through it, particularly alphabetically a la help() or the docs, isn't very helpful. * Other outstanding proposals like PEP 432 may restructure underlying APIs and having the sys module mirror the resulting APIs would help reduce the cognitive overhead in this space. * It would be nice to support descriptors on the module. * The current structure does not communicate the role of the module very well. Changing the sys module should be done carefully and as little as possible. The status quo has something like a +10 modifier when it comes to sys changes. The current API is pretty stable and undoubtedly a lot of code relies on it. This proposal reflects an understanding of those considerations and addresses them in an entirely backward compatible way (including impact on alternate Python implementations). At the moment I'm not proposing any big changes to sys nor any changes at all to its API. That can come later. <wink> Instead, this proposal is about setting the stage for better flexibility in making changes to sys when the need arises. Consequently I propose just 2.5000000000000001 relatively simple changes: 1. the existing sys module be renamed to _sys (and the stdlib be changed accordingly where appropriate); 2. a pure Python sys module be added to the stdlib that wraps the existing API; 2.5. the new module should use the replace-itself-in-sys.modules technique to provide a module subclass that supports descriptors. Thoughts? -eric

5 7

a standard approach to module-replace-itself-in-sys.modules
by Eric Snow March 14, 2014

March 14, 2014

Not that long ago we had a discussion about the esoteric, yet totally supported and not going anywhere, trick of having a module replace itself in sys.modules. Presently there isn't any convention surrounding the technique. It's simply a matter of sticking something else into sys.modules[__name__]. The problem is that there are some subtle things that can go wrong in the import system if you aren't careful and people generally won't know that. (Then again the whole technique isn't exactly in the mainstream!) To help reduce the risk here, I propose that we provide a helper in importlib.util and an associated convention. The helper will be a class decorator and the convention will be to decorate 1 class at the bottom of the module, where the class will contain any relevant customizations. The decorator will do the following: 1. create a new module subclass; 2. merge the decorated class namespace into the subclass; 3. create an instance of the subclass; 4. merge the original module namespace into the instance; 5. set sys.modules[__name__] to the new instance. For example: @importlib.util.replace_module class CustomModule: CHEDDAR = None @property def limburger (self): return None There's more to it but that captures the gist. I've been exploring API ideas, but ultimately we don't need much to get the ball rolling. In the meantime, I've published an unrefined but mostly working version of replace_module() online: https://bitbucket.org/ericsnowcurrently/odds_and_ends/src/default/replace_m… -eric

3 3

numerical type combining integer and float/decimal properties [Was: Re: Python Numbers as Human Concept Decimal System]
by Wolfgang Maier March 13, 2014

March 13, 2014

Am Montag, 10. März 2014 14:53:42 UTC+1 schrieb Stefan Krah: > [My apologies for being terse, I don't have much time to follow this discussion right now.] > Nick Coghlan <ncoghlan(a)gmail.com> wrote: >> I think users of decimal literals will just need to deal with the risk of unexpected rounding, as the alternatives are even more problematic. > That is why I think we should seriously consider moving to IEEE semantics for a decimal literal. Among other things: > - Always round the literal inputs. > - Supply IEEE contexts. > - Make Decimal64 the default. > - Add the missing rounding modes to sqrt() and exp(). > - Keep the ability to create exact Decimals through the constructor when no context is passed. > - Make the Decimal constructor use the context for rounding if it is passed. > - ... While I find this discussion about decimal literals extremely interesting, in my opinion, such a literal should have an underlying completely new numerical type, if it is really supposed to be for inexperienced users. Most of the discussions right now concern rounding issues that occur after the decimal point, but I think an at least equally big problem is rounding *to the left* of it as in (using current float): >>> 1e50 + 1000 1e+50 Importantly, Decimal is no cure here: >>> Decimal(10**50) + Decimal(100) Decimal('1.000000000000000000000000000E+50') (of course, you can debate context settings to make this particular example work, but, in general, it happens with big enough numbers.) The solution for this example is using ints of course: >>> 10**50 + 100 100000000000000000000000000000000000000000000000100 , but obviously this works only for whole numbers, so there currently is no built-in way to make this example work correctly: >>> 10**50 - 9999999999999999.5 1e+50 (or with Decimal: >>> Decimal(10**50) - Decimal('9999999999999999.5') Decimal('1.000000000000000000000000000E+50') ). If we are discussing a new number literal I would like to have it cope with this and my suggestion is a new numeric type that's sort of a hybrid between int and either Decimal or float, i.e., a type that behaves like int for digits left of the decimal point, but may truncate to the right. In pure Python, something similar (but not a literal form of course) could be implemented as a class that stores the left digits as an int and the right digits as a float/Decimal internally. Calculations involving this class would be slow due to the constant need of shifting digits between the integer´and the float part, and might be too slow for many users even when written in C, but its behavior would meet the expectation of inexperienced people better than the existing types. Going back to Mark Harris' initial proposal of unifying numeric types (which I am not trying to support in any way here), such a type would even allow to unify int and float since an ints could be considered a subset of the new type with a fractional part of zero. Cheers, Wolfgang

6 18

Python Numbers as Human Concept Decimal System
by Mark H. Harris March 12, 2014

March 12, 2014

Greetings, The purpose of this idea is to expand a bit on the decimal default idea which I submitted previously. In this idea I want to suggest the human idea of a *python number.* The concept is very simple, yet may have far reaching implications not only for future python(s) but also for the wider adaptation of python in the greater academic and professional communities. The idea of *python number* means that there are no types, no limits, no constraints, and that all *python numbers *are dynamic. The upper level syntax is also very simple; all* python numbers *are simply human. My influence for this preference is rooted in the Rexx programming language; attributed to Mike Cowlishaw, former IBM fellow. The Rexx programming language is dynamic, and has no types. To put it more succinctly for those of you who have not used Rexx, the only data type is a string of characters (that's it). *Rexx numbers* are simply those strings of characters that may be interpreted as a valid *Rexx number.* http://books.google.com/books?id=cNiVqFmPs8AC&pg=PA100&lpg=PA100&dq=rexx+nu… The Python language might be changed to adopt the *python number* concept for *all math processing*, unless explicitly modified. This goes somewhat beyond using decimal floating point as a default numerical type. It means using human numeric expressions that meet human expectation for numeric processing by default. Under the covers (whatever we mean by that) processing is handled by decimal.Decimal, unless explicitly modified. What does this mean for python users in general? Well, no more worrying about types at all... no ints, no floats, no rationals, no irrationals, no fractions, and certainly no binaries. In short, if its a human number, its a *python number.* I am expecting that (just as in Rexx numbers) defining very clearly what is a *python number* will be key for wide adaptation of the concept. But there should be no surprises for users, particularly average users. Anyone with a middle school expectation of a numeric format should be able to use *python numbers *without surprises. However, for advanced users the full interface should be available (as is the context for Decimal) through coding based on knowledge and experience, yet the default context for Decimal should be based on average users in most environments and use cases. It is my belief that Python should lead the way in the twenty-first century for advanced computation for academic, professional, business, and scientific communities. There is a 40 year momentum for embedded binary floats & doubles, also numeric types generally, but it is time to move forward. The technology is ready, and the need is great. Let's do it. {shameless plug} pdeclib https://pypi.python.org/pypi/pdeclib Thank you for your consideration. Good evening. Mark H Harris marcus

31 271

%-formatting with Decimals
by Oscar Benjamin March 12, 2014

March 12, 2014

During recent discussions I came across something unfortunate with formatting Decimals. Formatting with .format works as I would want: >>> from decimal import Decimal as D >>> '{:1.0e}'.format(D('1.123e+1000')) '1e+1000' >>> '{:1.0e}'.format(D('1.123e-1000')) '1e-1000' >>> '{:.50f}'.format(D('1.1')) '1.10000000000000000000000000000000000000000000000000' But %-formatting coerces to float: >>> '%1.0e' % D('1.123e-1000') '0e+00' >>> '%1.0e' % D('1.123e+1000') 'inf' >>> '%.50f' % D('1.1') '1.10000000000000008881784197001252323389053344726562' Am I doing this wrong? Is this just a limitation of the old-style % formatting or something that could possibly be improved? Oscar

8 13

Parsing ISO 8601 with the standard library
by Tin Tvrtković March 11, 2014

March 11, 2014

Hello, it occured to me today that it's not currently possible to parse ISO 8601 dates using the datetime.datetime.strftime function (I was parsing datetimes generated by Postgres). The problem is in the semicolon in the time zone offset. '1997-07-16T19:20:30.45+01:00' is a valid ISO 8601 date and time representation, the likes of which can be generated by the datetime module itself (the isoformat method). The %z strftime directive only recognizes offsets without the semicolon. The Python docs also direct users to inspect the platform strftime documentation; on my system the man page clearly states %z is "... The +hhmm or -hhmm numeric timezone", again, no semicolon support. Googling around, most common suggestions are to use a third party library (such as dateutil), so either this functionality really doesn't exist in a simple form in the datetime module, or is really undiscoverable. It seems to me ISO 8601 is significant enough (since there are even special methods for it in the datetime module) for it to be parsable, in a non-complex way, by the standard library. I guess it's interesting to point out a new flag was added to Java's SimpleDateFormat's date and time patterns in Java 7 - the X directive stands for "ISO 8601 time zone" (-08; -0800; -08:00, Z). I've worked with this so I happen to know it off the top of my head. Thanks in advance for comments. Tin

6 7

Suggestion for standardized annotations
by Cem Karan March 11, 2014

March 11, 2014

Hi all, I wanted to make a suggestion for a convention on how to write annotations. I know that PEP 8 says that the standard library won't use them in order to avoid setting a convention, but that could lead to confusion and conflicts in how they are used in non-standard libraries. The reason I'm thinking about this is because of how docstrings are currently used. If you use PLY (http://www.dabeaz.com/ply/) and also want to use something like Sphinx (http://sphinx-doc.org/), you're going to have a problem; PLY stores its rules in the docstring, while Sphinx parses it for documentation. I want to prevent this problem for annotations. My thought is that all annotations should be dictionaries. The keys should all be unicode strings that are uppercase UUIDs (basically, what 'uuid.uuid1().hex.upper()' returns), and the values can be anything the programmer wants. Each project can generate one (or more) UUIDs to put into the dictionary, and publicly document what the UUID's meaning is (preferably somewhere where a search engine can find it). The advantage is that since UUIDs are unique, the number of false positives you'll get while searching for it should be low; I've tested this on a different mailing list I'm on, and the UUID I generated for it has 0 false positives, while picking up the complete discussion involving it. As an example, if project A had chosen the UUID B3D2AFE8A45A11E3AE24D49A20C52EF2 and project B chose the UUID C02D7C64A45A11E39DAFD49A20C52EF2, we might annotate a function as follows: def foo(first : { B3D2AFE8A45A11E3AE24D49A20C52EF2 : {"doc" : "English documentation"}, C02D7C64A45A11E39DAFD49A20C52EF2 : {"doc" : "expression : MINUS expression %prec UMINUS"} }, second) -> {B3D2AFE8A45A11E3AE24D49A20C52EF2: {"type" : int, "doc" : "Typechecking for a linter"}}: pass You can already see the downside of this approach; it's really, really verbose. However, at least it avoids outright conflicts in usage that prevent usage of certain tools/projects together. Note that I did consider using names as keys directly (e.g. 'doc'). However, that requires a strong, universal convention on what each key means. Since we can't seem to figure that out for the docstring, I don't see why we should expect to be able to figure it out for any of proposed keys. Moreover, the set of keys would need to be documented somewhere, the documentation kept up to date, etc. It becomes a management nightmare. UUIDs have the advantage the we just tell everyone how to generate their own, and let them go at it. If someone wants to use a given project's docstring for their own purposes, it is up to them to keep the meaning the same. Thoughts/suggestions? Thanks, Cem Karan

8 13

Changing Decimal.__pos__ and Decimal.__neg__ [Was: Re: Python Numbers as Human Concept Decimal System]
by Oscar Benjamin March 10, 2014

March 10, 2014

On 10 March 2014 08:32, Mark Dickinson <dickinsm(a)gmail.com> wrote: > On Sun, Mar 9, 2014 at 8:07 PM, Oscar Benjamin <oscar.j.benjamin(a)gmail.com> > wrote: >> >> The problem though is with things like +3.14d or -3.14d. Python the >> language treats the + and - signs as not being part of the literal but >> as separate unary operators. This is unimportant for int, float and >> imaginary literals because there unary + is a no-op and unary - is >> exact. For decimal this is not the same as +Decimal('3.14') is not the >> same as Decimal('+3.14'): > > Another option would be to change the behaviour of + and - on Decimal > objects by binding `__pos__` and `__neg__` to the specification's 'copy' and > 'copy-negate' operations instead of (as currently) to the 'plus' and 'minus' > operations. Those behaviours are more in keeping with the way that + and - > operate on other objects. Hi Mark and thanks for responding, Do you think it's really feasible to make such a change? The use of unary + for rounding to context has been a feature of the decimal module for as long as I've been using it. It's shown in the docs both in the Recipes section and the FAQ. I just advised Mark Harris to do this and it know appears ~20 times in his pdeclib module: https://code.google.com/p/pythondecimallibrary/source/browse/pdeclib.py#207 I would expect that a change such as this would break quite a lot of code (some of mine would break) so there would have to be some kind of compatibility mechanism or deprecation process or something but I can't think of a good way of doing it. Oscar

7 12

Extend socket.accept to put accepted socket in non-blocking mode
by Saúl Ibarra Corretgé March 9, 2014

March 9, 2014

Hi, (I hope this list is the right place for this) I was browsing around for accept4 usage in Python and I saw it's already used internally in socket.accept (on Linux only), albeit it's not exposed, so accepting an incoming connection and setting it to be non-blocking takes 3 syscalls. accept4 allows us to do this with a single syscall, and it made it into FreeBSD 10, so that's another system that could benefit from this optimization. Would it be desirable to extend socket.accept to something like: socket.accept(set_non_blocking=False) which would use accept4's flags on supported systems and fallback to regular fcntl on others? The idea is to accept an incoming connection and make it non-blocking at the same time, asyncio and other frameworks would benefit from this by doing one function call and 2 syscalls less (on supported systems, that is). Cheers, -- Saúl Ibarra Corretgé bettercallsaghul.com

4 5

Enhance exceptions by attaching some more information to them
by Sebastian Kreft March 8, 2014

March 8, 2014

More than once I've been in a situation where I wish that some of the stdlib exceptions had a better message or some more information to help me diagnose the problem. For example: a = [1, 2, 3, 4, 5] a[5] IndexError: list index out of range In this case there's no reference to neither the length of the array nor to the offending index. I'm of the idea that we could extend the exceptions by adding some more information to them, so 3rd party libraries could use them for debugging/logging. For example, one such use case would be to have a modified test runner, that in case of exceptions automatically prints some debug information. Another would be a logger system that in case of an exception also logs some debug info that could be relevant to understand and solve the issue. I propose extending (at least) the following exceptions with the following attributes: KeyError: key, object IndexError: index, object AttributeError: attribute, object NameError: name Of course that populating these attributes could be controlled by a flag. I know that some of this information is already present in some exceptions, depending on the context. However, I propose adding these attributes, as in this way a tool could easily and reliably extract the information and work with it, as opposed to have to parse the huge variety of different messages there are. For the first use case mentioned above I have a working prototype, although it does not use this proposal, but a series of hacks (I'm modifying the bytecode to save a reference to the key and object :() and parsing of the exception messages. But I want to share what the output of such a tool could be. ====================================================================== ERROR: test_attribute (example.ExampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/skreft/test/debug_exception/example.py.py", line 18, in test_attribute AttributeError: 'str' object has no attribute 'Lower'. Did you mean 'islower', 'lower'? Debug info: Object: '' Type: <type 'str'> ====================================================================== ERROR: test_index (example.ExampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/skreft/test/debug_exception/example.py.py", line 6, in test_index IndexError: list index out of range Debug info: Object: [1, 2] Object len: 2 Index: 2 ====================================================================== ERROR: test_key (example.ExampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/skreft/test/debug_exception/example.py.py", line 10, in test_key KeyError_: 'fooo', did you mean 'foo'? Debug info: Object: {'foo': 1} Key: 'fooo' ====================================================================== ERROR: test_name (example.ExampleTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/skreft/test/debug_exception/example.py.py", line 14, in test_name NameError: global name 'fooo' is not defined. Did you mean 'foo'? ---------------------------------------------------------------------- Ran 4 tests in 0.005s -- Sebastian Kreft

4 4