On Oct 20, 2009, at 9:43 AM, Stefan Krah wrote:

ssteinerX@gmail.com wrote:

...

Shouldn't this be on python-ideas?

I found previous discussions about "Decimal in C" on python-dev, that's why used this list.

python-ideas: This list is to contain discussion of speculative language ideas for Python for possible inclusion into the language. If an idea gains traction it can then be discussed and honed to the point of becoming a solid proposal to put to either python-dev or python-3000 as appropriate. I guess it's a fine line...and matter of opinion. No worries. S

On Tue, 20 Oct 2009 at 09:27, ssteinerX@gmail.com wrote:

Shouldn't this be on python-ideas?

IMO this question is appropriate for python-dev, not python-ideas. --David

On Tue, Oct 20, 2009 at 3:00 PM, Paul Moore wrote:

2009/10/20 Stefan Krah :

...

Hi,

as some of you know, recently I've released an arbitrary precision C library for decimal arithmetic together with a Python module:

http://www.bytereef.org/libmpdec.html http://www.bytereef.org/fastdec.html

Both the library and the module have been tested extensively. Fastdec currently differs from decimal.py in a couple of ways that could be fixed. The license is AGPL, but if there is interest in integrating it into Python I'd release it under a Python-compatible license.

There have been several approaches towards getting C decimal arithmetic into Python:

http://bugs.python.org/issue2486

Fastdec follows Raymond Hettinger's suggestion to provide wrappers for an independent C implementation. Arguments in favour of fastdec are:

 * Complete implementation of Mike Cowlishaw's specification

 * C library can be tested independently

 * Redundant arithmetic module for tests against decimal.py

 * Faster than Java BigDecimal

 * Compares relatively well in speed against gmpy

To be clear, I would not want to _replace_ decimal.py. Rather I'd like to see a cdecimal module alongside decimal.py.

Why? If it's 100% compatible with decimal.py, just replace it. All the user should see is improved speed. Let's not do another pickle/cpickle.

For example other python implementations might decide to use python version as long as builtin version does not appear. Python versions are usually also better targets for jit than mixed versions. C level versions also usually have more bugs (just statistics), so some people might want to choose pure-python version. In general - some people have some reasons. Cheers, fijal

On Wed, Oct 21, 2009 at 11:37 AM, Nick Coghlan wrote:

As far as decimal.py in particular goes, there are significant maintenance gains in keeping a lot of the non-performance critical context management code in pure Python. So we're likely to wait and see how much speed Mark can wring out of a simple C decimal coefficient object

No need to wait for this. :-) I don't really expect to get much speed gain at all in normal, low-precision (= precisions less than 100 digits, say) use. Even doubling the speed would be way too much to hope for here. The only real gain from a decimal integer coefficient would be fixing the asymptotics for high-precision calculations. Mark

Mark Dickinson writes:

There are certainly some open issues (e.g., what to do with the existing Python module; what should other Python implementations do).

The existing module could be kept as a fallback. Also, the test suite should be careful to test both implementations (like what is currently done for the io module).

BTW, does anyone know the current SLOC count for py3k?

Here you are, generated using David A. Wheeler's 'SLOCCount': SLOC Directory SLOC-by-Language (Sorted) 261496 Lib python=261451,sh=45 186769 Modules ansic=173279,asm=9561,sh=3929 53258 Objects ansic=53258 40257 Python ansic=40229,python=28 27220 Tools python=27117,ansic=67,sh=36 18892 Demo python=18511,ansic=377,sh=4 9168 PC ansic=8465,python=703 5840 Include ansic=5840 5799 Parser ansic=3914,python=1885 3485 Misc lisp=2948,python=242,sh=185,ansic=110 3101 Doc python=2306,ansic=795 3030 Mac python=2138,objc=775,sh=109,ansic=8 1666 top_dir python=1140,ansic=286,sh=240 349 PCbuild python=279,ansic=70 337 build ansic=295,python=42 0 Grammar (none) Totals grouped by language (dominant language first): python: 315842 (50.89%) ansic: 286993 (46.24%) asm: 9561 (1.54%) sh: 4548 (0.73%) lisp: 2948 (0.47%) objc: 775 (0.12%) Total Physical Source Lines of Code (SLOC) = 620,667

Mark Dickinson wrote:

Thanks, Antoine! With SLOCCount I can revise my earlier numbers, as well: Here's Stefan Krah's mpdecimal, version 0.80:

SLOC Directory SLOC-by-Language (Sorted) 21445 top_dir ansic=21267,sh=105,python=55,asm=18 6238 python python=6177,java=43,sh=18 1403 tests ansic=1356,sh=47 476 literature lisp=476 274 cmd ansic=274 11 tools sh=11 0 doc (none)

I would say that the relevant code is less than that: The module code is counted twice (fastdec2.c, fastdec3.c), and sloccount counts the inline functions in the mpdecimal*.h header files. So, after removing fastdec3.c, mpdecimal32.h, mpdecimal32vc.h, mpdecimal64vc.h, I get: SLOC Directory SLOC-by-Language (Sorted) 13702 top_dir ansic=13524,sh=105,python=55,asm=18 6238 python python=6177,java=43,sh=18 1403 tests ansic=1356,sh=47 476 literature lisp=476 274 cmd ansic=274 11 tools sh=11 0 doc (none) Totals grouped by language (dominant language first): ansic: 15154 (68.56%) python: 6232 (28.19%) lisp: 476 (2.15%) sh: 181 (0.82%) java: 43 (0.19%) asm: 18 (0.08%) Therefore, my estimate is between 12660 and 13702 lines, depending on whether to count the remaining mpdecimal64.h (Most inline functions in this header file are signaling wrappers around the quiet functions and are not necessary for the module). If one takes out all library functions that the module does not use, I'm sure it could be condensed to ~11500 lines. This is comparable to the expat directory in Modules/: SLOC Directory SLOC-by-Language (Sorted) 11406 expat ansic=11406 Stefan Krah

Mark Dickinson wrote:

...

2. Would fastdec - after achieving full decimal.py compatibility - be   a serious candidate?

Definitely. As far as I know it's the only real candidate for a full C version of decimal right now. Other possibilities that I'm aware of:

* I think Raymond Hettinger is working on a C version of decimal. I'm not sure what its current status is. Raymond?

* There's a partially complete rewrite of decimal in C in the sandbox, dating from the Need for Speed sprint in 2006:

* Wrapping the decNumber library directly would make some sense, but I think licensing issues rule this out.

I hope I'm not spreading misinformation, but I see two problems with the use of decNumber as an arbitrary precision library. First, the algorithms are not optimized for large numbers. Then, you have to define the maximum length of the coefficient *at compile time* by setting DECNUMDIGITS.

It's difficult to gauge the likelihood of eventual acceptance in advance, though. Maybe writing a PEP would be an efficient use of time in this regard? There are certainly some open issues (e.g., what to do with the existing Python module; what should other Python implementations do).

Good, I'll do that. I mainly wanted to find out if there is fundamental opposition against C-decimal in the standard library, but this does not seem to be the case.

I think my biggest concern is maintenance: we'd be replacing 8500 lines of Python code in a single file, that several of the current core developers understand well, with 30000 (Stefan, is that about accurate?) lines of C in several files, that presumably none of the current core devs is familiar with right now.

Ok, I think we could say that the sloc-counts are around 2600 lines of Python vs. around 11500-12500 lines of C.

What happens when (e.g.,) the number-theoretic transform needs updating for one reason or another? Stefan, do you see yourself having time to spend on maintenance of this code for the forseeable future?

Yes. Generally speaking, I expect the code to be low-maintenance. It would be even easier if all major compilers supported exact width C99 types and __uint128_t, but this is wishful thinking. Stefan Krah