Mailman 3 May 2005 - Python-Dev

Re: [Python-Dev] PEP 344: Explicit vs. Implicit Chaining
by Michael Chermside May 23, 2005

May 23, 2005

James Knight writes: > I still don't see why people think the python interpreter should be > automatically providing __context__. To me it seems like it'll just > clutter things up for no good reason. If you really want the other > exception, you can access it via the local variable in the frame > where it was first caught. No you can't, because you didn't know the second exception was going to happen! I write something like this: db_connection = get_db_connection() try: do_some_stuff(db_connection) except DatabaseException, err: log_the_problem(err) cleanup(db_connection) If something goes wrong inside of do_some_stuff, I enter the exception handler. But then if an error occurs within log_the_problem() or cleanup(), then I lose the original exception. It's just GONE. I didn't expect log_the_problem() or cleanup() to fail, but sometimes things DO fail. An example of this happens to me in in Java (which has the same problem. I have code like this: db_connection = get_db_connection() try: do_some_stuff(db_connection) finally: db_connection.close() For instance, when I want to do unit testing, I create a mock database connection that raises an exception if you don't use it as the test expects. So I get exceptions like this all the time: Error: did not expect call to "close()" Of course, what REALLY happened was that we tried to update a row that didn't exist, which created an exception: Error: tried to update row with key "100", but it does not exist. But then it entered the finally clause, and tried to close the connection. That wasn't expected either, and the new exception replaces the old one... and we lose information about what REALLY caused the problem. In Java, I had to fix this by making my mock objects very smart. They have to keep track of whether any problem has occurred during this test (in any of the cooperating mock objects) and if so, then they have to re-report the original problem whenever something new goes wrong. This is the only way I've found to work around the problem in Java. Wouldn't it be nice if Python could do better? -- Michael Chermside

1 0

Re: [Python-Dev] Adventures with Decimal
by Michael Chermside May 23, 2005

May 23, 2005

I'd like to respond to a few people, I'll start with Greg Ewing: Greg writes: > I don't see how it > helps significantly to have just the very first > step -- turning the input into numbers -- be > exempt from this behaviour. If anything, people > are going to be even more confused. "But it > can obviously cope with 1.1000000000000000001, > so why does it give the wrong answer when I add > something to it?" As I see it, there is a meaningful distinction between constructing Decimal instances and performing arithmatic with them. I even think this distinction is easy to explain to users, even beginners. See, it's all about the program "doing what you tell it to". If you type in this: x = decimal.Decimal("1.100000000000000000000000000003") as a literal in your program, then you clearly intended for that last decimal place to mean something. By contrast, if you were to try passing a float to the Decimal constructor, it would raise an exception expressly to protect users from "accidently" entering something slightly off from what they meant. On the other hand, in Python, if you type this: z = x + y then what it does is completely dependent on the types of x and y. In the case of Decimal objects, it performs a "perfect" arithmetic operation then rounds to the current precision. The simple explanation for users is "Context affects *operations*, but not *instances*." This explains the behavior of operations, of constructors, and also explains the fact that changing precision doesn't affect the precision of existing instances. And it's only 6 words long. > But I also found it interesting that, while the spec > requires the existence of a context for each operation, > it apparently *doesn't* mandate that it must be kept > in a global variable, which is the part that makes me > uncomfortable. > > Was there any debate about this choice when the Decimal > module was being designed? It shouldn't make you uncomfortable. Storing something in a global variable is a BAD idea... it is just begging for threads to mess each other up. The decimal module avoided this by storing a SEPARATE context for each thread, so different threads won't interfere with each other. And there *is* a means for easy access to the context objects... decimal.getcontext(). Yes, it was debated, and the debate led to changing from a global variable to the existing arrangement. ------ As long as I'm writing, let me echo Nick Coghlan's point: > The fact that the BDFL (and others, me included) were at least temporarily > confused by the ability to pass a context in to the constructor suggests there > is an interface problem here. > > The thing that appears to be confusing is that you *can* pass a context in to > the Decimal constructor, but that context is then almost completely ignored. Yeah... I agree. If you provide a Context, it should be used. I favor changing the behavior of the constructor as follows: def Decimal(data, context=None): result = Existing_Version_Of_Decimal(data) if context is None: result = +result return result In other words, make FULL use of the context in the constructor if a context is provided, but make NO use of the thread context when no context is provided. ------ One final point... Thanks to Mike Cowlishaw for chiming in with a detailed and well-considered explanation of his thoughts on the matter. -- Michael Chermside

1 0

Re: [Python-Dev] Adventures with Decimal
by Nick Coghlan May 23, 2005

May 23, 2005

Raymond Hettinger wrote: >>Py> decimal.Decimal("a", context) >>Decimal("NaN") >> >>I'm tempted to suggest deprecating the feature, and say if you want >>invalid >>strings to produce NaN, use the create_decimal() method of Context >>objects. > > > The standard does require a NaN to be produced. In that case, I'd prefer to see the behaviour of the Decimal constructor (InvalidOperation exception, or NaN result) always governed by the current context. If you want to use a different context (either to limit the precision, or to alter the way malformed strings are handled), you invoke creation via that context, not via the standard constructor. > Unless something is shown to be wrong with the current implementation, I > don't think we should be in a hurry to make a post-release change. The fact that the BDFL (and others, me included) were at least temporarily confused by the ability to pass a context in to the constructor suggests there is an interface problem here. The thing that appears to be confusing is that you *can* pass a context in to the Decimal constructor, but that context is then almost completely ignored. It gives me TOOWTDI concerns, even though passing the context to the constructor does, in fact, differ slightly from using the create_decimal() method (the former does not apply the precision, as Guido discovered). Cheers, NIck. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia --------------------------------------------------------------- http://boredomandlaziness.blogspot.com

1 0

Re: [Python-Dev] Decimal FAQ
by Tony Meyer May 23, 2005

May 23, 2005

> Q. I'm writing a fixed-point application to two decimal places. > Some inputs have many places and needed to be rounded. Others > are not supposed to have excess digits and need to be validated. > What methods should I use? > > A. The quantize() method rounds to a fixed number of decimal > places. If the Inexact trap is set, it is also useful for > validation: > > >>> TWOPLACES = Decimal(10) ** -2 > >>> # Round to two places > >>> Decimal("3.214").quantize(TWOPLACES) > Decimal("3.21") > >>> # Validate that a number does not exceed two places > >>> Decimal("3.21").quantize(TWOPLACES, > context=Context(traps=[Inexact])) > Decimal("3.21") I think an example of what happens when it does exceed two places would make this example clearer. For example, adding this to the end of that: >>> Decimal("3.214").quantize(TWOPLACES, context=Context(traps=[Inexact])) Traceback (most recent call last): [...] Inexact: Changed in rounding =Tony.Meyer

1 0

Adventures with Decimal
by Mike Cowlishaw May 22, 2005

May 22, 2005

Several people have pointed me at this interesting thread, and both Tim and Raymond have sent me summaries of their arguments. Thank you all! I see various things I have written have caused some confusion, for which I apologise. The 'right' answer might, in fact, depend somewhat on the programming language, as I'll try and explain below, but let me first try and summarize the background of the decimal specification which is on my website at: http://www2.hursley.ibm.com/decimal/#arithmetic Rexx ---- Back in 1979/80, I was writing the Rexx programming language, which has always had (only) decimal arithmetic. In 1980, it was used within IBM in over 40 countries, and had evolved a decimal arithmetic which worked quite well, but had some rather quirky arithmetic and rounding rules -- in particular, the result of an operation had a number of decimal places equal to the larger of the number of decimal places of its operands. Hence 1.23 + 1.27 gave 2.50 and 1.230000 + 1.27 gave 2.500000. This had some consequences that were quite predictable, but were unexpected by most people. For example, 1.2 x 1.2 gave 1.4, and you had to suffix a 0 to one of the operands (easy to do in Rexx) to get an exact result: 1.2 x 1.20 => 1.44. By 1981, much of the e-mail and feedback I was getting was related to various arithmetic quirks like this. My design strategy for the language was more-or-less to 'minimise e-mail' (I was getting 350+ every day, as there were no newsgroups or forums then) -- and it was clear that the way to minimise e-mail was to make the language work the way people expected (not just in arithmetic). I therefore 'did the research' on arithmetic to find out what it was that people expected (and it varies in some cases, around the world), and then changed the arithmetic to match that. The result was that e-mail on the subject dropped to almost nothing, and arithmetic in Rexx became a non-issue: it just did what people expected. It's strongest feature is, I think, that "what you see is what you've got" -- there are no hidden digits, for example. Indeed, in at least one Rexx interpreter the numbers are, literally, character strings, and arithmetic is done directly on those character strings (with no conversions or alternative internal representation). I therefore feel, quite strongly, that the value of a literal is, and must be, exactly what appears on the paper. And, in a language with no constructors (such as Rexx), and unlimited precision, this is straightforward. The assignment a = 1.10000001 is just that; there's no operation involved, and I would argue that anyone reading that and knowing the syntax of a Rexx assignment would expect the variable a to have the exact value of the literal (that is, "say a" would then display 1.10000001). The Rexx arithmetic does have the concept of 'context', which mirrors the way people do calculations on paper -- there are some implied rules (how many digits to work to, etc.) beyond the sum that is written down. This context, in Rexx, "is used to change the way in which arithmetic operations are carried out", and does not affect other operations (such as assignment). Java ---- So what should one do in an object-oriented language, where numbers are objects? Java is perhaps a good model, here. The Java BigDecimal class originally had only unlimited precision arithmetic (the results of multiplies just got longer and longer) and only division had a mechanism to limit (round) the result in some way, as it must. By 1997, it became obvious that the original BigDecimal, though elegant in its simplicity, was hard to use. We (IBM) proposed various improvements and built a prototype: http://www2.hursley.ibm.com/decimalj/ and this eventually became a formal Java Specification Request: http://jcp.org/aboutJava/communityprocess/review/jsr013/index.html which led to the extensive enhancements in BigDecimal that were shipped last year in Java 5: http://java.sun.com/j2se/1.5.0/docs/api/java/math/BigDecimal.html In summary, for each operation (such as a.add(b)) a new method was added which takes a context: a.add(b, context). The context supplies the rounding precision and rounding mode. Since the arguments to an operation can be of any length (precision), the rounding rule is simple: the operation is carried out as though to infinite precision and is then rounded (if necessary). This rule avoids double-rounding. Constructors were not a point of debate. The constructors in the original BigDecimal always gave an exact result (even when constructing from a binary double) so those were not going to change. We did, however, almost as an afterthought, add versions of the constructors that took a context argument. The model, therefore, is essentially the same as the Rexx one: what you see is what you get. In Java, the assignment: BigDecimal a = new BigDecimal("1.10000001"); ends up with a having an object with the value you see in the string, and for it to be otherwise one would have to write: BigDecimal a = new BigDecimal("1.10000001", context); which gives a very nice clue that something may happen to the value. This, to me, seems a clean design. So why does my specification appear to say something different? --------------------------------------------------------------- Both the languages described so far support arbitrary-length decimal numbers. Over the past five years, however, I have been concentrating more on fixed-length decimals, as in languages such as C# and as will be in C and C++ and in hardware. When the representation of a decimal number has a fixed length, then the nice clean model of a one-to-one mapping of a literal to the internal representation is no longer always possible. For example, the IEEE 754r proposed decimal32 format can represent a maximum of 7 decimal digits in the significand. Hence, the assignment: decimal32 d = 1.10000001; (in some hypothetical C-like language) cannot result in d having the value shown in the literal. This is the point where language history or precedent comes in: some languages might quietly round at this point, others might give a compile-time warning or error (my preference, at least for decimal types). Similar concerns apply when the conversion to internal form causes overflow or underflow. The wording in the specification was intended to allow for these kinds of behaviors, and to allow for explicit rounding, using a context, when a string is converted to some internal representation. It was not intended to restrict the behavior of (for example) the Java constructor: one might consider that constructor to be working with an implied context which has an infinite precision. In other words, the specification does not attempt to define where the context comes from, as this would seem to be language-dependent. In Java, any use of a programmer- supplied context is explicit and visible, and if none is supplied then the implied context has UNLIMITED precision. In Rexx, the context is always implicit -- but there is no 'conversion from string to number' because numbers _are_ strings. So what should Python do? ------------------------- Since your Decimal class has the ability to preserve the value of a literal exactly, my recommendation is that that should be the default behavior. Changing the value supplied as a literal without some explicit syntax is likely to surprise, given the knowledge that there are no length restrictions in the class. My view is that only in the case of a fixed-length destination or an explicit context might such a rounding be appropriate. Given that Python has the concept of an implicit decimal context, I can see why Tim can argue that the implicit context is the context which applies for a constructor. However, perhaps you can define that the implicit context 'only applies to arithmetic operations', or some such definition, much as in Rexx? And I should clarify my specification to make it clear that preserving the value, if possible, is preferable to rounding -- any suggestions for wording? Mike Cowlishaw [I'll try and follow this thread in the mailing list for a while, but I am flying to the USA on Monday so my e-mail access will be erratic for the next week or so.]

1 0

Re: [Python-Dev] Adventures with Decimal
by Michael Chermside May 21, 2005

May 21, 2005

[Tim and Raymond are slugging it out about whether Decimal constructors should respect context precision] Tim, I find Raymond's arguments to be much more persuasive. (And that's even BEFORE I read his 11-point missive.) I understood the concept that *operations* are contex-dependent, but decimal *objects* are not, and thus it made sense to me that *constructors* were not context-dependent. On the other hand, I am NOT a floating-point expert. Can you educate me some? What is an example of a case where users would get "wrong" results because constructors failed to respect context precision? (By the way... even if other constructors begin to respect context precision, the constructor from tuple should NOT -- it exists to provide low-level access to the implementation. I'll express no opinion on the constructor from Decimal, because I don't understand the issues.) -- Michael Chermside

7 8

Re: [Python-Dev] [Python-checkins] python/nondist/peps pep-0343.txt, 1.11, 1.12
by Raymond Hettinger May 21, 2005

May 21, 2005

> + def sin(x): > + "Return the sine of x as measured in radians." > + do with_extra_precision(): > + i, lasts, s, fact, num, sign = 1, 0, x, 1, x, 1 > + while s != lasts: > + lasts = s > + i += 2 > + fact *= i * (i-1) > + num *= x * x > + sign *= -1 > + s += num / fact * sign > + return +s One more change: The final "return +s" should be unindented. It should be at the same level as the "do with_extra_precision()". The purpose of the "+s" is to force the result to be rounded back to the *original* precision. This nuance is likely to be the bane of folks who shift back and forth between different levels of precision. The following example shows the kind of oddity that can arise when working with quantities that have not been rounded to the current precision: >>> from decimal import getcontext, Decimal as D >>> getcontext().prec = 3 >>> D('3.104') + D('2.104') Decimal("5.21") >>> D('3.104') + D('0.000') + D('2.104') Decimal("5.20") Raymond

7 18

Re: [Python-Dev] Adventures with Decimal
by Michael Chermside May 21, 2005

May 21, 2005

Guido writes: > It looks like if you pass in a context, the Decimal constructor still > ignores that context No, you just need to use the right syntax. The correct syntax for converting a string to a Decimal using a context object is to use the create_decimal() method of the context object: >>> import decimal >>> decimal.getcontext().prec = 4 >>> decimal.getcontext().create_decimal("1.234567890") Decimal("1.235") Frankly, I have no idea WHAT purpose is served by passing a context to the decimal constructor... I didn't even realize it was allowed! -- Michael Chermside

3 2

Re: [Python-Dev] Adventures with Decimal
by Michael Chermside May 21, 2005

May 21, 2005

Tim Peters writes: > Other important > implementations of the standard didn't make this mistake; for example, > Java's BigDecimal|(java.lang.String) constructor follows the rules > here: [...] > Hmm -- or maybe it doesn't! The text says: [...] Here is the actual behavior: Jython 2.1 on java1.5.0_03 (JIT: null) Type "copyright", "credits" or "license" for more information. >>> import java.math.BigDecimal as BigDecimal >>> import java.math.MathContext as MathContext >>> BigDecimal("0.33333333333333333333333333333333333") 0.33333333333333333333333333333333333 >>> BigDecimal("0.33333333333333333333333333333333333", MathContext.DECIMAL32) 0.3333333 >>> In other words, Java's behavior is much closer to the current behavior of Python, at least in terms of features that are user-visible. The default behavior in Java is to have infinite precision unless a context is supplied that says otherwise. So the constructor that takes a string converts it faithfully, while the constructor that takes a context obeys the context. One could argue that they are "following the rules" appropriately and it just happens that their default context has infinite precision. But from the point of view of the typical, non-floating-point-aware user, Java's constructor gives "all the digits you told it to", and so does Python's current string constructor. Using "+decimal.Decimal(s)" today "rounds off the constant" (in the nieve user's viewpoint). Of course, the user is going to be surprised in the NEXT step since the Python *operations* respect context while the Java ones use infinite precision for +, -, and * (and require you to specify the behavior for /). (PS: No, I don't think we should design decimal.Decimal to match the behavior of Java... but I don't think that the Java example really helps make your point.) Elsewhere, Tim writes: > Sorry, I can't make more time for this now. I understand! > The short course is that > a module purporting to implement an external standard should not > deviate from that standard without very good reasons Yes, but should we think of the constructor-from-string as an implementation-specific means of creating Decimal objects, which is separate from string-to-Decimal converter that the standard requires (and which is provided by the Context objects)? -- Michael Chermside

5 7

First PyPy (preview) release
by Armin Rigo May 20, 2005

May 20, 2005

The PyPy 0.6 release -------------------- *The PyPy Development Team is happy to announce the first public release of PyPy after two years of spare-time and half a year of EU funded development. The 0.6 release is eminently a preview release.* What it is and where to start ----------------------------- Getting started: http://codespeak.net/pypy/index.cgi?doc/getting_started.html PyPy Documentation: http://codespeak.net/pypy/index.cgi?doc PyPy Homepage: http://codespeak.net/pypy/ PyPy is a MIT-licensed reimplementation of Python written in Python itself. The long term goals are an implementation that is flexible and easy to experiment with and retarget to different platforms (also non-C ones) and such that high performance can be achieved through high-level implementations of dynamic optimisation techniques. The interpreter and object model implementations shipped with 0.6 can be run on top of CPython and implement the core language features of Python as of CPython 2.3. PyPy passes around 90% of the Python language regression tests that do not depend deeply on C-extensions. Some of that functionality is still made available by PyPy piggy-backing on the host CPython interpreter. Double interpretation and abstractions in the code-base make it so that PyPy running on CPython is quite slow (around 2000x slower than CPython ), this is expected. This release is intended for people that want to look and get a feel into what we are doing, playing with interpreter and perusing the codebase. Possibly to join in the fun and efforts. Interesting bits and highlights --------------------------------- The release is also a snap-shot of our ongoing efforts towards low-level translation and experimenting with unique features. * By default, PyPy is a Python version that works completely with new-style-classes semantics. However, support for old-style classes is still available. Implementations, mostly as user-level code, of their metaclass and instance object are included and can be re-made the default with the ``--oldstyle`` option. * In PyPy, bytecode interpretation and object manipulations are well separated between a bytecode interpreter and an *object space* which implements operations on objects. PyPy comes with experimental object spaces augmenting the standard one through delegation: * an experimental object space that does extensive tracing of bytecode and object operations; * the 'thunk' object space that implements lazy values and a 'become' operation that can exchange object identities. These spaces already give a glimpse in the flexibility potential of PyPy. See demo/fibonacci.py and demo/sharedref.py for examples about the 'thunk' object space. * The 0.6 release also contains a snapshot of our translation-efforts to lower level languages. For that we have developed an annotator which is capable of infering type information across our code base. The annotator right now is already capable of successfully type annotating basically *all* of PyPy code-base, and is included with 0.6. * From type annotated code, low-level code needs to be generated. Backends for various targets (C, LLVM,...) are included; they are all somehow incomplete and have been and are quite in flux. What is shipped with 0.6 is able to deal with more or less small/medium examples. Ongoing work and near term goals --------------------------------- Generating low-level code is the main area we are hammering on in the next months; our plan is to produce a PyPy version in August/September that does not need to be interpreted by CPython anymore and will thus run considerably faster than the 0.6 preview release. PyPy has been a community effort from the start and it would not have got that far without the coding and feedback support from numerous people. Please feel free to give feedback and raise questions. contact points: http://codespeak.net/pypy/index.cgi?contact contributor list: http://codespeak.net/pypy/index.cgi?doc/contributor.html have fun, Armin Rigo, Samuele Pedroni, Holger Krekel, Christian Tismer, Carl Friedrich Bolz PyPy development and activities happen as an open source project and with the support of a consortium funded by a two year EU IST research grant. Here is a list of partners of the EU project: Heinrich-Heine University (Germany), AB Strakt (Sweden) merlinux GmbH (Germany), tismerysoft GmbH(Germany) Logilab Paris (France), DFKI GmbH (Germany) ChangeMaker (Sweden)

1 0