Mailman 3 February 2014 - Python-ideas

int('0x3241fca1')
by Ram Rachum Feb. 7, 2014

Feb. 7, 2014

What do you think about letting the `int` constructor automatically understand the number type without specifying base if given a prefix, so int('0x3414fa') would immediately work without specifying a base of 16?

11 13

Inline Functions - idea
by Alex Rodrigues Feb. 6, 2014

Feb. 6, 2014

Hi everyone, This is my first time on the Python mailing lists. I've been learning a lot about how python is run recently and today I thought I ran across an idea which might create in interesting discussion.Today I was writing a piece of software where I had a block of code that would take a bunch of local variables, apply some transformations to them and then output them as a a string to a log. Then I realized that I actually wanted to reuse this code in multiple locations - there were multiple cases where I might need to do this. My natural inclination was to write a function in order to maintain DRY programming. This was prohibitively challenging, however, since the code itself interacted with lots of variables in the namespace. The number of arguments to the function would have to be very large and possibly would change on a fairly regular basis.This seems like a fairly common problem in programming, having a piece of code which is both reused and heavily integrated with the namespace making it necessary to use copy-paste. As a solution to this I propose the idea of an inline function. An inline function would run in it's parent's namespace instead of creating a new one. This would allow you to avoid passing back and forth tons of values while still maintaining DRY code. It might look something like this if implemented in a log for a traction control system: # First without an inline functiondef main(): file = open('file.txt') counter = 0 while True: counter += 1 frontL, frontR, backL, backR = getWheelSpeeds() if counter > 100: # Log at least every 10 seconds slipL = abs(frontL - backL) slipR = abs(frontR - backR) file.write('Speeds: ('+frontL+', '+frontR+', '+backL+', '+backR+'), \n Slip: '+slipL+', '+slipR) counter = 0 elif abs(frontR-backR) > 1 or abs(frontL-backL) > 1: # Also log if the wheels are slipping slipL = abs(frontL - backL) slipR = abs(frontR - backR) file.write('Speeds: ('+frontL+', '+frontR+', '+backL+', '+backR+'), \n Slip: '+slipL+', '+slipR) counter = 0 elif average([frontL, frontR, backL, backR]) > 60: # Also log if we're going really fast slipL = abs(frontL - backL) slipR = abs(frontR - backR) file.write('Speeds: ('+frontL+', '+frontR+', '+backL+', '+backR+'), \n Slip: '+slipL+', '+slipR) counter = 0 time.sleep(.1) # And now with an inline functiondef main(): file = open('file.txt') counter = 0 while True: counter += 1 frontL, frontR, backL, backR = getWheelSpeeds() if counter > 100: # Log every 10 seconds no matter what saveLine() elif abs(frontR-backR) > 1 or abs(frontL-backL) > 1: # Also log if the wheels are slipping saveLine() elif average([frontL, frontR, backL, backR]) > 60: # Also log if we're going really fast saveLine() time.sleep(.1) inline def saveLine(): slipL = abs(frontL - backL) slipR = abs(frontR - backR) file.write('Speeds: ('+frontL+', '+frontR+', '+backL+', '+backR+'), \n Slip: '+slipL+', '+slipR) counter = 0 What do you think? - Alex

21 33

Fwd: int('0x3241fca1')
by Bruce Leban Feb. 6, 2014

Feb. 6, 2014

On Thu, Feb 6, 2014 at 2:24 AM, Ram Rachum <ram.rachum(a)gmail.com> wrote: > What do you think about letting the `int` constructor automatically > understand the number type without specifying base if given a prefix, so > int('0x3414fa') would immediately work without specifying a base of 16? > What's the use case? Do you expect prefixes in user input? In data you're reading from files and parsing where you don't know the base? I don't see it. This would change the behavior of current code and potentially introduce bugs. For example, this allows attackers to trick programs into accepting aliases of numbers they would not otherwise accept. Search [overlong utf-8 sequences] for an example of this kind of attack. And it would be quite strange that a base of 10 means sometimes 10 and sometimes look at a prefix while all other bases mean exactly what they say. >>> int('0xa', 16) 10 >>> int('0xa', 36) 1198 >>> int('1e1', 16) 481 EIBTI: If there's a compelling use case, a better idea IMHO would be to allow int(x, None) to mean to use prefixes or even a different function, like parse_int. I think e suffixes have long implied float values and I see little benefit in allowing them here. ASIDE: int accepts a prefix when it matches the base you're parsing and rejects invalid prefixes. Most of the time. >>> int('0x111', 16) 273 >>> int('0b111', 2) 7 >>> int('0x111', 2) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: invalid literal for int() with base 2: '0x111' >>> int('0b111', 16) 45329 The last case might be surprising for a moment but hardly wrong, and it's a good illustration of the problem of trying to guess what things mean. --- Bruce

1 0

Re: [Python-ideas] int('0x3241fca1')
by Andrew Barnert Feb. 6, 2014

Feb. 6, 2014

On Feb 6, 2014, at 4:44, Ram Rachum <ram.rachum(a)gmail.com> wrote: > Ah, you're talking about non-programmers. I can't imagine though a non-programmer seriously complaining that "0x0" is a number while "0z" isn't. Can you please read the whole sentence before replying to it? I specifically said "Would a newbie--or a non-programmer using a program--understand..." And you can't imagine a non-programmer complaining that "0123" is either not a number, or the number 83? > On Thu, Feb 6, 2014 at 2:41 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: >> On Feb 6, 2014, at 4:35, Ram Rachum <ram.rachum(a)gmail.com> wrote: >> >>> On Thu, Feb 6, 2014 at 2:31 PM, Andrew Barnert <abarnert(a)yahoo.com> wrote: >>>> On Feb 6, 2014, at 4:19, Ram Rachum <ram.rachum(a)gmail.com> wrote: >>>> >>>>> I don't understand... The newbie will pass '0x13412' to the int constructor by mistake and be surprised when it's parsed as hex? Doesn't make sense does it? >>>> >>>> Would a newbie--or a non-programmer using a program--understand why, say, "0X0" counts as a valid number but "0Z" doesn't? >>> >>> Since they're likely to get the same confusion while feeding literals to the Python shell, I think this is acceptable. >> >> You think it's acceptable that a non-programmer should have to understand the python shell to use any program written in Python? >

3 3

Inline Functions - idea
by Alex Rodrigues Feb. 5, 2014

Feb. 5, 2014

I like the idea of requiring that inline functions be called in a special way that denotes them. Otherwise it does add a much bigger area that a maintainer must look for possible bugs. Also on the topic of locals(), note this from the python docs: Note The contents of this dictionary [the locals() dict] should not be modified; changes may not affect the values of local and free variables used by the interpreter.

2 1

Make all builtin functions accept None for optional arguments
by Ram Rachum Feb. 5, 2014

Feb. 5, 2014

Here is something that always annoys me. I was going to write my own rreplace function, like this: def rreplace(s, old, new, count=None): return new.join(s.rsplit(old, count)) But lo and behold, I have to write it like this: def rreplace(s, old, new, count=None): return new.join(s.rsplit(old, count) if count is not None else s.rsplit(old)) Why? Because the `str.rsplit` can't handle a count of `None`. That is quite annoying. There are many more builtin functions except `str.rsplit` that behave like this. What do you think about going over all such functions and making them able to accept `None`? Thanks, Ram.

4 3

Re: [Python-ideas] str.rreplace
by Nick Coghlan Feb. 4, 2014

Feb. 4, 2014

On 25 Jan 2014 04:29, "Andrew Barnert" <abarnert(a)yahoo.com> wrote: > > On Jan 24, 2014, at 10:20, Antoine Pitrou <solipsis(a)pitrou.net> wrote: > > > On Fri, 24 Jan 2014 20:13:26 +0200 > > Serhiy Storchaka <storchaka(a)gmail.com> > > wrote: > >> 24.01.14 19:36, Antoine Pitrou написав(ла): > >>> On Fri, 24 Jan 2014 19:30:00 +0200 > >>> Serhiy Storchaka <storchaka(a)gmail.com> > >>> wrote: > >>>> 24.01.14 18:56, Antoine Pitrou написав(ла): > >>>>> On Fri, 24 Jan 2014 08:47:14 -0800 (PST) > >>>>> Ram Rachum <ram.rachum(a)gmail.com> wrote: > >>>>>> I propose implementing str.rreplace. (It'll be to str.replace what > >>>>>> str.rsplit is to str.split.) > >>>>> > >>>>> I suppose it only differs when the count parameter is supplied? > >>>>> > >>>>> I don't think it can hurt, except for the funny looks of its name. > >>>>> In any case, if str.rreplace is added then so should bytes.rreplace and > >>>>> bytearray.rreplace. > >>>> > >>>> bytearray.rremove, tuple.rindex, list.rindex, list.rremove. > >>> > >>> Not sure what those have to do with rreplace(). Overgeneralization > >>> doesn't help. > >> > >> If open a door for rreplace, it would be not easy to close it for rindex > >> and rremove. > > > > Perhaps you underestimate our collective door closing skills ;) > > While we're speculatively overgeneralizing, couldn't all of the index/find/remove/replace/etc. methods take a negative n to count from the end, making r variants unnecessary? Strings already provide rfind and rindex (they're just not part of the general sequence API). Since strings are immutable, there's also no call for an "rremove". rreplace (pronounced as 'ar-replace", like "ar-split" et al) is more obvious than a negative count, and seems like an almost exact parallel to rsplit. On the other hand, I don't recall ever lamenting its absence. Call me +0 on the idea. Cheers, Nick. > _______________________________________________ > Python-ideas mailing list > Python-ideas(a)python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/

10 18

Unify global and nonlocal
by Manuel Cerón Feb. 4, 2014

Feb. 4, 2014

Hello, The 'global' and 'nonlocal' statements serve very similar purposes. They both allow to rebind names in an outer scope. While 'nonlocal' allows to rebind a name in the closest outer scope, *except* the global module scope, 'global' is used for the later case. Is there a reason why 'nonlocal' has to stop at the scope immediately after the global module scope? If 'nonlocal' could continue until the global scope, then it would become (almost) a generalized case of 'global'. Having both statements goes against the "There should be one - and preferably only one - obvious way to do it" principle. It's also confusing for beginners and people used to a functional style. I think 'nonlocal' should be changed to support all the enclosing scopes, including the global module scope, and the 'global' statement should be marked as deprecated. There is one specific use case that breaks though, which is rebinding to global names from a deep function skipping names in the middle, for example: x = 'global' def outer1(): x = 'outer1' def outer2(): global x x = 'outer2' # binds the global module name But you know "special cases aren't special enough to break the rules". My apologizes if this topic was already discussed in the past. Manuel.

6 5

statistics module in Python3.4
by Wolfgang Feb. 2, 2014

Feb. 2, 2014

Dear all, I am still testing the new statistics module and I found two cases were the behavior of the module seems suboptimal to me. My most important concern is the module's internal _sum function and its implications, the other one about passing Counter objects to module functions. As for the first subject: Specifically, I am not happy with the way the function handles different types. Currently _coerce_types gets called for every element in the function's input sequence and type conversion follows quite complicated rules, and - what is worst - make the outcome of _sum() and thereby mean() dependent on the order of items in the input sequence, e.g.: >>> mean((1,Fraction(2,3),1.0,Decimal(2.3),2.0, Decimal(5))) 1.9944444444444445 >>> mean((1,Fraction(2,3),Decimal(2.3),1.0,2.0, Decimal(5))) Traceback (most recent call last): File "<pyshell#7>", line 1, in <module> mean((1,Fraction(2,3),Decimal(2.3),1.0,2.0, Decimal(5))) File "C:\Python33\statistics.py", line 369, in mean return _sum(data)/n File "C:\Python33\statistics.py", line 157, in _sum T = _coerce_types(T, type(x)) File "C:\Python33\statistics.py", line 327, in _coerce_types raise TypeError('cannot coerce types %r and %r' % (T1, T2)) TypeError: cannot coerce types <class 'fractions.Fraction'> and <class 'decimal.Decimal'> (this is because when _sum iterates over the input type Fraction wins over int, then float wins over Fraction and over everything else that follows in the first example, but in the second case Fraction wins over int, but then Fraction vs Decimal is undefined and throws an error). Confusing, isn't it? So here's the code of the _sum function: def _sum(data, start=0): """_sum(data [, start]) -> value Return a high-precision sum of the given numeric data. If optional argument ``start`` is given, it is added to the total. If ``data`` is empty, ``start`` (defaulting to 0) is returned. Examples -------- >>> _sum([3, 2.25, 4.5, -0.5, 1.0], 0.75) 11.0 Some sources of round-off error will be avoided: >>> _sum([1e50, 1, -1e50] * 1000) # Built-in sum returns zero. 1000.0 Fractions and Decimals are also supported: >>> from fractions import Fraction as F >>> _sum([F(2, 3), F(7, 5), F(1, 4), F(5, 6)]) Fraction(63, 20) >>> from decimal import Decimal as D >>> data = [D("0.1375"), D("0.2108"), D("0.3061"), D("0.0419")] >>> _sum(data) Decimal('0.6963') """ n, d = _exact_ratio(start) T = type(start) partials = {d: n} # map {denominator: sum of numerators} # Micro-optimizations. coerce_types = _coerce_types exact_ratio = _exact_ratio partials_get = partials.get # Add numerators for each denominator, and track the "current" type. for x in data: T = _coerce_types(T, type(x)) n, d = exact_ratio(x) partials[d] = partials_get(d, 0) + n if None in partials: assert issubclass(T, (float, Decimal)) assert not math.isfinite(partials[None]) return T(partials[None]) total = Fraction() for d, n in sorted(partials.items()): total += Fraction(n, d) if issubclass(T, int): assert total.denominator == 1 return T(total.numerator) if issubclass(T, Decimal): return T(total.numerator)/total.denominator return T(total) Internally, the function uses exact ratios for its calculations (which I think is very nice) and only goes through all the pain of coercing types to return T(total.numerator)/total.denominator where T is the final type resulting from the chain of conversions. I think a much cleaner (and probably faster) implementation would be to gather first all the types in the input sequence, then decide what to return in an input order independent way. My tentative implementation: def _sum2(data, start=None): if start is not None: t = set((type(start),)) n, d = _exact_ratio(start) else: t = set() n = 0 d = 1 partials = {d: n} # map {denominator: sum of numerators} # Micro-optimizations. exact_ratio = _exact_ratio partials_get = partials.get # Add numerators for each denominator, and build up a set of all types. for x in data: t.add(type(x)) n, d = exact_ratio(x) partials[d] = partials_get(d, 0) + n T = _coerce_types(t) # decide which type to use based on set of all types if None in partials: assert issubclass(T, (float, Decimal)) assert not math.isfinite(partials[None]) return T(partials[None]) total = Fraction() for d, n in sorted(partials.items()): total += Fraction(n, d) if issubclass(T, int): assert total.denominator == 1 return T(total.numerator) if issubclass(T, Decimal): return T(total.numerator)/total.denominator return T(total) this leaves the re-implementation of _coerce_types. Personally, I'd prefer something as simple as possible, maybe even: def _coerce_types (types): if len(types) == 1: return next(iter(types)) return float , but that's just a suggestion. In this case then: >>> _sum2((1,Fraction(2,3),1.0,Decimal(2.3),2.0, Decimal(5)))/6 1.9944444444444445 >>> _sum2((1,Fraction(2,3),Decimal(2.3),1.0,2.0, Decimal(5)))/6 1.9944444444444445 lets check the examples from the _sum docstring just to be sure: >>> _sum2([3, 2.25, 4.5, -0.5, 1.0], 0.75) 11.0 >>> _sum2([1e50, 1, -1e50] * 1000) # Built-in sum returns zero. 1000.0 >>> from fractions import Fraction as F >>> _sum2([F(2, 3), F(7, 5), F(1, 4), F(5, 6)]) Fraction(63, 20) >>> from decimal import Decimal as D >>> data = [D("0.1375"), D("0.2108"), D("0.3061"), D("0.0419")] >>> _sum2(data) Decimal('0.6963') Now the second issue: It is maybe more a matter of taste and concerns the effects of passing a Counter() object to various functions in the module. I know this is undocumented and it's probably the user's fault if he tries that, but still: >>> from collections import Counter >>> c=Counter((1,1,1,1,2,2,2,2,2,3,3,3,3)) >>> c Counter({1: 4, 2: 5, 3: 4}) >>> mode(c) 2 Cool, mode knows how to work with Counters (interpreting them as frequency tables) >>> median(c) 2 Looks good >>> mean(c) 2.0 Very well But the truth is that only mode really works as you may think and we were just lucky with the other two: >>> c=Counter((1,1,2)) >>> mean(c) 1.5 oops >>> median(c) 1.5 hmm >From a quick look at the code you can see that mode actually converts your input to a Counter behind the scenes anyway, so it has no problem. mean and median, on the other hand, are simply iterating over their input, so if that input happens to be a mapping, they'll use just the keys. I think there are two simple ways to avoid this pitfall: 1) add an explicit warning to the docs explaining this behavior or 2) make mean and median do the same magic with Counters as mode does, i.e. make them check for Counter as the input type and deal with it as if it were a frequency table. I'd favor this behavior because it looks like little extra code, but may be very useful in many situations. I'm not quite sure whether maybe even all mappings should be treated that way? Ok, that's it for now I guess. Opinions anyone? Best, Wolfgang

11 19

Re: [Python-ideas] statistics module in Python3.4
by Wolfgang Maier Feb. 2, 2014

Feb. 2, 2014

Oscar Benjamin <oscar.j.benjamin@...> writes: > > On 1 February 2014 20:10, Wolfgang Maier > <wolfgang.maier@...> wrote: > > Greg Ewing <greg.ewing <at> ...> writes: > > > >> > >> Wolfgang Maier wrote: > >> > Mappings may be an excellent way of specifying frequencies and weights in an > >> > elegant way. > >> > >> That may be, but I think I'd rather have a separate > >> function, or a mode argument, explicitly indicating > >> that this is what you want. Detecting whether something > >> is a mapping in a duck-typed way is dodgy in general. > >> > > > > There should be nothing dodgy about this with the abstract base class Mapping. > > I agree with Greg about this. I dislike APIs that try to be too clever > about accepting different types of input. The mode() function is > clearly intended to accept an iterable not a Counter and I consider it > a bug that it does. It can be fixed to treat a Counter as an iterable > by changing the _counts function to do > > collections.Counter(data) > > to > > collections.Counter(iter(data)) > > If you want the option for doing statistics with Counter style data > formats then they should be invoked explicitly as Greg says. > I would accept this as a bug-fix, but I do not agree with you and Greg about an API trying to be too clever here. A Mapping is a clearly defined term and if the module doc stated that for Mappings passed to functions in statistics their values will be interpreted as weights/frequencies of the corresponding keys that should be clear enough. If what you really want is to do statistics just on the keys, you can easily pass just these (e.g., mean(mydict.keys()). On the other hand, separate functions would complicate the API because you would end up with a weighted counterpart for almost every function in the module. The alternative of passing weights in the form of a second sequence sounds more attractive, but I guess both specifications could coexist peacefully (just like there are several ways of constructing a dictionary). Then if you have data and weights in a Mapping already, you can just go ahead and use it, and likewise, if you have them in two separate sequences. Best, Wolfgang

3 4