Mailman 3 August 2016 - Python-ideas

real numbers with SI scale factors: next steps
by Ken Kundert Sept. 1, 2016

Sept. 1, 2016

Okay, let's try to wrap this up. In summary I proposed three things: 1. A change to the Python lexer to accept SI literal as an alternative, but not replacement to, E-notation. As an optional feature, simple units could be added to the end but would be largely ignored. So the following would be accepted: freq = 2.4GHz r = 1k l = 10nm The idea in accepting units was to allow them to be specified when convenient as additional documentation on the meaning of the number. Objections: a. Acceptance of the abbreviation for Exa (E) overlaps with E-notation (1E+1 could represent 1e18 + 1 or 10). A suggestion to change the prefix from E to X conflicts with a proposal to use X, W, and V to represent 10^27, 10^30, and 10^33 (en.wikipedia.org/wiki/Metric_prefix) b. Allowing the units to be specified will lead some users to assume a dimensional analysis is being performed when in fact the units are ignored. This false sense of security could lead to bugs. c. The proposal only supports simple units, not compound units such as m/s. So even if hooks were provided to allow access to the units to support an add-on dimensional analysis capability, an additional mechanism would have to be provided to support compound units. d. Many people objected to allowing the use of naked scale factors as a perversion of the standard. 2. A change to the float() function so that it accepts SI scale factors and units. This extension naturally follows from the first: the float function should accept anything the Python parser accepts. For example: freq = float('2.4GHz') r = float('1k') l = float('10nm') Objections: a. The Exa objection from the above proposal is problematic here as well. b. Things that used to be errors are now no longer errors. This could cause problems if a program was counting on float('1k') to be an error. 3. A change to the various string formatting mechanisms to allow outputting real numbers with SI scale factors: >>> print('Speed of light in a vacuum: {:r}m/s.'.format(2.9979e+08)) Speed of light in a vacuum: 299.79 Mm/s. >>> print('Speed of sound in water: %rm/s.' % 1481 Speed of sound in water: 1.481 km/s. Objections: No objections were raised that I recall, however here is something else to consider: a. Should we also provide mechanism for the binary scale factors (Ki, Mi, ..., Yi)? For example: '{:b}B'.format(2**30) --> 1 GiB. On proposed extension 1 (native support for SI literals) my conclusion is that we did not reach any sense of consensus and there was considerable opposition to my proposal. There was much less discussion on extensions 2 & 3, so it is hard to say whether consensus was reached. So, given all this, I would like to make the following recommendations: 1. No action should be taken. 2. The main justification to modifying float() was to make it consistent with the extended Python language. Without extension 1, this justification goes away. However the need to be able to easily convert strings of numbers with SI scale factors into floats still exists. This should be handled by adding a library or extending an existing library. 3. Allowing numbers to be formatted with SI prefixes is useful and not controversial. The 'r' and 'b' format codes should be added to the various string formatting mechanisms. What do you think? -Ken

14 28

real numbers with SI scale factors
by Ken Kundert Aug. 31, 2016

Aug. 31, 2016

Wow, things have suddenly turned very negative. I understand that this is very normal for these types of forums where it is easy to lose sight of the big picture. So let me try to refocus this discussion again. MOTIVATION The way the scientific and engineering communities predominately write real numbers is by using SI scale factors. These numbers almost always represent physical quantities, so it is common to write the number with scale factor and units. So for example, the distance to Andromeda is 780kpc, the pressure at the bottom of the Mariana Trench is 108MPa, the total power produced by a typical hurricane is 600TW, the number of base pairs in human DNA is 3.2Gb, and the Bohr radius is 53pm. These numbers are the language of science and engineering, but in recent years they have also entered the realm of popular culture. For example, an article by Ars Technica that calculates that the value of the music that can fit on a classic iPod as over $8G (that is the total penalty that can be accessed if they were all stolen copyright works). http://arstechnica.com/tech-policy/2012/06/from-gigabytes-to-petadollars-co… In all of these examples the numbers are either very large or very small, and they employ the use of SI scale factors to make them easy to write and communicate. This way of writing numbers is so well established that it was formally standardized as part of the International System of Units (the Système international d'unités) in 1960. The problem is that most general purpose programming languages do not support this way of writing numbers. Instead they force people to convert the numbers to and from exponential notation, which is rather inconvenient and hard to read, write, say, or hear (it is much easier to say or hear 53 picometers than 5.3e-11 meters). When working with a general purpose programming language, the above numbers become: 780kpc -> 7.8e+05 108MPa -> 1.08e+08 600TW -> 6e+14 3.2Gb -> 3.2e+09 53pm -> 5.3e-11 $8G -> 8e+09 Notice that the numbers become longer, harder to read, harder to type, harder to say, and harder to hear. NEXT STEP So the question is, should Python accommodate this widely used method of writing real numbers? Python already has many ways of writing numbers. For example, Python allows integers to be written in binary, octal, decimal and hex. Any number you can write in one form you can write in another, so the only real reason for providing all these ways is the convenience of the users. Don't Python's users in the scientific and engineering communities deserve the same treatment? These are, after all, core communities for Python. Now, Python is a very flexible language, and it is certainly possible to add simple libraries to make it easy to read and write numbers with SI scale factors. I have written such a library (engfmt). But with such libraries this common form of writing numbers remains a second class citizen. The language itself does not understand SI scale factors, instead any time you want to input or output numbers in their natural form you must manually convert them yourself. Changing Python so that it understands SI scale factors on real numbers as first class citizens innately requires a change to the base language; it cannot be done solely through libraries. The question before you is, should we do it? The same question confronted the developers of Python when it was decided to add binary, octal and hexadecimal number support to Python. You could have done it with libraries, but you didn't because binary, octal and hexadecimal numbers were too common and important to be left as second class citizens. Well, use of binary, octal and hexadecimal numbers is tiny compared to the use of real numbers with SI scale factors. Before we expend any more effort on this topic, let's put aside the question of how it should be done, or how it will be used after its done, and just focus on whether we do it at all. Should Python support real numbers specified with SI scale factors as first class citizens? What do you think? -Ken

19 54

Force UTF-8 option regardless locale
by INADA Naoki Aug. 30, 2016

Aug. 30, 2016

On Tue, Aug 30, 2016 at 8:14 AM, Victor Stinner <victor.stinner(a)gmail.com> wrote: > > I proposed the idea, but I'm not sure that we can have a single option > for Linux and Windows. Moreover, I never really worked on trying to > implement "-X utf8" on Linux, because it looks like the "misconfigured > system" are less and less common nowadays. I see very few user > requests in this direction. Some people loves tiny Linux image for Docker and RasberryPi. They doesn't has any locale other than C. Some OPs loves LANG=C or LC_ALL=C to avoid troubles and unexpected performance regression caused by locale. (e.g. sort command is much slower on ja_JP.utf8). I want to write script using utf-8 for stdio and fsencoding. Sometimes, people runs my script in C locale. And sometimes runs in misconfigured locale because SSH sends LANG that system doesn't have. So I wonder if Python has Force UTF-8" option. And if the option is configure option or site-wide installation option, because: * command line option cannot be set in shebang * Setting environment variable may be forgetten when writing scripts like crontab. The option may make startup bit faster, because it can skip setting locale in startup. Any thoughts? How should the option be set? -- INADA Naoki <songofacandy(a)gmail.com>

4 4

Let’s make escaping in f-literals impossible
by Philipp A. Aug. 30, 2016

Aug. 30, 2016

Hi, I originially posted this via google groups, which didn’t make it through to the list proper, sorry! Read it here please: https://groups.google.com/forum/#!topic/python-ideas/V1U6DGL5J1s My arguments are basically: 1. f-literals are semantically not strings, but expressions. 2. Their escape sequences in the code parts are fundamentally both detrimental and superfluous (they’re only in for convenience, as confirmed by Guido in the quote below) 3. They’re detrimental because Syntax highlighters are (by design) unable to handle this part of Python 3.6a4’s grammar. This will cause code to be highlighted as parts of a string and therefore overlooked. i’m very sure this will cause bugs. 4. The fact that people see the embedded expressions as somehow “part of the string” is confusing. My poposal is to redo their grammar: They shouldn’t be parsed as strings and post-processed, but be their own thing. This also opens the door to potentially extend to with something like JavaScript’s tagged templates) Without the limitations of the string tokenization code/rules, only the string parts would have escape sequences, and the expression parts would be regular python code (“holes” in the literal). Below the mentioned quote and some replies to the original thread: Guido van Rossum <guido(a)python.org> schrieb am Mi., 17. Aug. 2016 um 20:11 Uhr: > The explanation is honestly that the current approach is the most > straightforward for the implementation (it's pretty hard to intercept the > string literal before escapes have been processed) and nobody cares enough > about the edge cases to force the implementation to jump through more hoops. > > I really don't think this discussion should be reopened. If you disagree, > please start a new thread on python-ideas. > I really think it should. Please look at python code with f-literals. if they’re highlighted as strings throughout, you won’t be able to spot which parts are code. if they’re highlighted as code, the escaping rules guarantee that most highlighters can’t correctly highlight python anymore. i think that’s a big issue for readability. Brett Cannon <brett(a)python.org> schrieb am Mi., 17. Aug. 2016 um 20:28 Uhr: > They are still strings, there is just post-processing on the string itself > to do the interpolation. > Sounds hacky to me. I’d rather see a proper parser for them, which of course would make my vision easy. > By doing it this way the implementation can use Python itself to do the > tokenizing of the string, while if you do the string interpolation > beforehand you would then need to do it entirely at the C level which is > very messy and painful since you're explicitly avoiding Python's automatic > handling of Unicode, etc. > of course we reuse the tokenization for the string parts. as said, you can view an f-literal as interleaved sequence of strings and expressions with an attached format specification. <f'> starts the f-literal, string contents follow. the only difference to other strings is <{> which starts expression tokenization. once the expression ends, an optional <formatspec> follows, then a <}> to switch back to string tokenization this repeats until (in string parsing mode) a <'> is encountered which ends the f-literal. You also make it harder to work with Unicode-based variable names (or at > least explain it). If you have Unicode in a variable name but you can't use > \N{} in the string to help express it you then have to say "normal Unicode > support in the string applies everywhere *but* in the string interpolation > part". > i think you’re just proving my point that the way f-literals work now is confusing. the embedded expressions are just normal python. the embedded strings just normal strings. you can simply switch between both using <{> and <[format]}>. unicode in variable names works exactly the same as in all other python code because it is regular python code. Or another reason is you can explain f-strings as "basically > str.format_map(**locals(), **globals()), but without having to make the > actual method call" (and worrying about clashing keys but I couldn't think > of a way of using dict.update() in a single line). But with your desired > change it kills this explanation by saying f-strings aren't like this but > some magical string that does all of this stuff before normal string > normalization occurs. > no, it’s simply the expression parts (that for normal formatting are inside of the braces of .format(...)) are *interleaved* in between string parts. they’re not part of the string. just regular plain python code. Cheers, and i really hope i’ve made a strong case, philipp

20 68

a multiProcess scheduler
by Thales filizola costa Aug. 29, 2016

Aug. 29, 2016

Hey guys, I was recently involved in a job change, and for that I have been doing a lot of programming interviews (white board questions). One common question on those interviews were: "how to implement a scheduler?" follow up by "how to make it multi-processing?". I have to confess that I only had a clue on how to do that. After the interview period, I started searching for a solution for that, and could not find one. The std python implementation for a scheduler says "No multi-threading is implied; you are supposed to hack that yourself, or use a single instance per application." So, I hacked my own implementation of a multi-process scheduler in python: https://github.com/thalesfc/Multprocess-Scheduler What do you guys think? How to improve it? Is it relevant enough to be incorporated to std python ? Thanks, Thales.

4 6

A proposal to rename the term "duck typing"
by ROGER GRAYDON CHRISTMAN Aug. 29, 2016

Aug. 29, 2016

After sending this proposal to the Python list, it was suggested that I redirect it here ("recommended" likely being to strong a word). We have a term in our lexicon "duck typing" that traces its origins, in part to a quote along the lines of "If it walks like a duck, and talks like a duck, ..." But I would assert that it would be far more Pythonic to use this quote instead: BEDEMIR: How do you know she is a witch? VILLAGER #1: She looks like one. In that case, it would be far more appropriate for use to call this sort of type analysis "witch typing" For additional supporting evidence: WITCH: I'm not a witch. I'm not a witch. BEDEMIR: But you are dressed as one. WITCH: They dressed me up like this. CROWD: No, we didn't -- no. WITCH: And this isn't my nose, it's a false one. BEDEMIR: Well? VILLAGER #1: Well, we did do the nose. BEDEMIR: The nose? VILLAGER #1: And the hat -- but she is a witch! The listed operations very much resemble how one defines an iterator in Python. One can take any object add imply add two methods, named __iter__ and __next__ and (voila!), you now have an iterator. And it is known to be such because of these methods. Now, to be intellectually honest, I must acknowledge a very cogent opposing argument from a Mr. Cameron Simpson in Australia, who pointed out that the true identification of a which was not based on these superficial appearances, but instead based on the weight of a duck -- which of course, is very strongly suggestive that "duck typing" is appropriate. After recovering from the confusion that I suffered from him having a name other than "Bruce", I tried to consider more fully the logic of his argument. It seems that the test here is as follows: VILLAGER #1: If... she.. weighs the same as a duck, she's made of wood. BEDEMIR: And therefore--? VILLAGER #1: A witch! Now, "duck typing" strictly applied would seem to me that weighing the same as a duck would imply that she was a duck. This is not explicitly stated in the source material -- but if that was the implicit understanding, then we would have "is-a" relationships relating ducks to wooden things to witches. It's here that I admittedly get a bit befuddled. If being a duck is sufficient evidence of witchcraft, we would be saying that all ducks are witches. On the other hand, if we are to use this test universally, then that sounds like all witches are ducks. I do not have enough samples to fully determine whether the set of witches and the set of ducks are identical sets, but I would be intended to think otherwise, especially if we furthermore have to consider the equivalence with the set of things made of wood. I think I might find it far simpler to accept the villagers' initial intuitions about the type-inferences rules for witches. I certainly would not want to have to set up a three-layer class hierarchy any time I wished to do dynamic type inference, so I'll just go back to the initial question. It was not whether the woman was a duck, but whether she was a witch. Hence, I believe a more Pythonic name for this sort of type inference should be "witch typing" Roger Christman Electrical Engineering and Computer Science Pennsylvania State University

4 3

Re: [Python-ideas] SI scale factors alone, without units or dimensional analysis
by Arek Bulski Aug. 28, 2016

Aug. 28, 2016

SI units are a standard that was kind of imposed top down on the computer science community. But we learned to use KB MB so why no keep the defacto standard we already have? Kibibytes and mibibytes were never really adopted. 1K == 1000 1KB == 1024 1M == 1000**2 1MB == 1024**2 Suffixes, simple. int_value = 8M float_value = 8.0M or float("8M") fraction_value = Fraction(1M, 8) or Fraction("1M/8") decimal_value = Decimal("1.2345M") Suffixes are by definition at the end of a literal. So 1E1E == 1E1 * 1E

3 3

Re: [Python-ideas] SI scale factors in Python
by Steven D'Aprano Aug. 27, 2016

Aug. 27, 2016

On Fri, Aug 26, 2016 at 03:23:24PM -0700, Ken Kundert wrote: > Steven, > This keeps coming up, so let me address it again. > > First, I concede that you are correct that my proposal does not provide > dimensional analysis, so any dimensional errors that exist in this new code will > not be caught by Python itself, as is currently the case. > > However, you should concede that by bringing the units front and center in the > language, they are more likely to be caught by the user themselves. I do not concede any such thing at all. At best this might apply under some circumstances with extremely simple formulae when working directly with literals, in other words when using Python like a souped-up calculator. (Which is a perfectly reasonable way to use Python -- I do that myself. But it's not the only way to use Python.) But counting against that is that there will be other cases where what should be a error will sneak past because it happens to look like a valid scale factor or unit: x += 1Y # oops, I fat-fingered the Y when I meant 16 > It is > my position that dimensional analysis is so difficult and burdensome that there > is no way it should be in the base Python language. If available, it should be > as an add on. This is a strange and contradictory position to take. If dimensional analysis is so "difficult and burdensome", how do you expect the user to do it in their head by just looking at the source code? It is your argument above that users will be able to catch dimensional errors just by looking at the units in the source code, but here, just one sentence later, you claim that dimensional analysis is so difficult and burdensome that users cannot deal with it even with the assistence of the interpreter. I cannot reconcile those two beliefs. If you think that dimensional analysis is both important and "difficult and burdensome", then surely we should want to automate as much of it as possible? Of course the easy cases are easy: torque = 45_N * 18_m is obviously correct, but the hard cases are not. As far as I can tell, your suggested syntax doesn't easily support compound units, let alone more realistic cases of formulae from sciences other than electrical engineering: # Van der Waals equation pressure = (5_mol * 6.022140857e23/1_mol * 1.38064852e−23_J/1_K * 340_K / (2.5_m**3 - 5_mol * 0.1281_m**3/1_mol) - (5_mol)**2*(19.7483_L*1_L*1_bar/(1_mol)**2) /(2.5_m**3)**2) ) I'm not even sure if I've got that right after checking it three times. I believe it is completely unrealistic to expect the reader to spot dimensional errors by eye in anything but the most trivial cases. Here is how I would do the same calculation in sympy. For starters, rather than using a bunch of "magic constants" directly in the formula, I would set them up as named variables. That's just good programming practice whether there are units involved or not. # Grab the units we need. from sympy.physics.units import mol, J, K, m, Pa, bar, liter as L # And some constants. from sympy.physics.units import avogadro_constant as N_a, boltzmann as k R = N_a*k # Define our variables. n = 5*mol T = 340*K V = 2.5*m**3 # Van der Waals constants for carbon tetrachloride. a = 19.7483*L**2*bar/mol**2 b = 0.1281*m**3/mol # Apply Van der Waals equation to calculate the pressure p = n*R*T/(V - n*b) - n**2*a/V**2 # Print the result in Pascal. print p/Pa Sympy (apparently) doesn't warn you if your units are incompatible, it just treats them as separate terms in an expression: py> 2*m + 3*K 3*K + 2*m which probably makes sense from the point of view of a computer algebra system (adding two metres and three degrees Kelvin is no weirder than adding x and y). But from a unit conversion point of view, I think sympy is the wrong solution. Nevertheless, it still manages to give the right result, and in a form that is easy to understand, easy to read, and easy to confirm is correct. (If p/Pa is not a pure number, then I know the units are wrong. That's not ideal, but it's better than having to track the units myself. There are better solutions than sympy, I just picked this because I happened to have it already installed.) > This proposal is more about adding capabilities to be base > language that happen to make dimensional analysis easier and more attractive > than about providing dimensional analysis itself. I think it is an admirable aim to want to make unit tracking easier in Python. That doesn't imply that this is the right way to go about it. Perhaps you should separate your suggested syntax from your ultimate aim. Instead of insisting that your syntax is the One Right Way to get units into Python, how about thinking about what other possible syntax might work? Here's a possibility, thrown out just to be shot down: # Van der Waals constants for carbon tetrachloride. a = 19.7483 as L**2*bar/mol**2 b = 0.1281 as m**3/mol I think that's better than: a = 19.7483_L * (1_L) * (1_bar) / (1_mol)**2 b = 0.1281_m * (1_m)**2 / 1_mol and *certainly* better than trying to have the intrepreter guess whether: 19.7483_L**2*bar/mol**2 means 19.7483 with units L**2*bar/mol**2 or 19.7483_L squared, times bar, divided by mol**2 -- Steve

3 2

Re: [Python-ideas] SI scale factors in Python
by Steven D'Aprano Aug. 27, 2016

Aug. 27, 2016

On Fri, Aug 26, 2016 at 03:23:24PM -0700, Ken Kundert wrote: > Second, I concede that there is some chance that users may be lulled into > a false sense of complacency and that some dimensional errors would get missed > by these otherwise normally very diligent users. But I would point out that > I have been intensively using and supporting languages that provide this feature > for 40 years and have never seen it. In your first post, you said that there were no languages at all that supported units as a language feature, and suggested that Python should lead the way here: I find it a little shocking that no programming languages offer this feature yet Now you say you've been using these "languages" plural for forty years. Would you like to rephrase your claim? I am unable to reconcile the discrepency. (There are three languages that I know of that support units as a first class language feature, RPL, Frink and Fortress. None of them are 40 years old.) -- Steve

4 3

Re: [Python-ideas] SI scale factors alone, without units or dimensional analysis
by Stephen J. Turnbull Aug. 27, 2016

Aug. 27, 2016

Ken Kundert writes: > The rule is you cannot give unit without a scale factor, and the > unity scale factor is _, so if you wanted to say 1 mol you would > use 1_mol. 1mol means one milli ol. These look a little strange, > but that is because the use they unit scale factor, which is the > one that is currently not in heavy use. One reason I like Python is that it has relatively few of these irregularities and ambiguities (in comparison to other languages and non-programming contexts with similar usage). For me, that counts against this proposal. BTW, where is "_" used as the unit scale prefix? > I suggest that we do not support the h (=100), da (=10), d (=0.1), > or c (=0.01) scale factors. I don't think it's reasonable to exclude those. Around me, cm, dB, and ha (centimeters, decibels, and hectares) are in common use. What happened to "support with a capital S"? I don't speak for anybody but myself, but I think this proposal has gotten less interesting/acceptable with each post. I'm going wait and see if the "units are types" approach goes anywhere. I think it's probably the only one that has wings, but that's because it requires no change to the language.

1 0