Shouldn't this be on python-ideas?

S

On Oct 20, 2009, at 9:15 AM, Stefan Krah wrote:

Hi,

as some of you know, recently I've released an arbitrary precision C library for decimal arithmetic together with a Python module:

http://www.bytereef.org/libmpdec.html http://www.bytereef.org/fastdec.html

Both the library and the module have been tested extensively. Fastdec currently differs from decimal.py in a couple of ways that could be fixed. The license is AGPL, but if there is interest in integrating it into Python I'd release it under a Python-compatible license.

There have been several approaches towards getting C decimal arithmetic into Python:

http://bugs.python.org/issue2486

Fastdec follows Raymond Hettinger's suggestion to provide wrappers for an independent C implementation. Arguments in favour of fastdec are:

• Complete implementation of Mike Cowlishaw's specification

• C library can be tested independently

• Redundant arithmetic module for tests against decimal.py

• Faster than Java BigDecimal

• Compares relatively well in speed against gmpy

To be clear, I would not want to _replace_ decimal.py. Rather I'd like to see a cdecimal module alongside decimal.py.

I know that ultimately there should be a PEP for module inclusions. The purpose of this post is to gauge interest. Specifically:

1. Are you generally in favour of a C decimal module in Python?

2. Would fastdec - after achieving full decimal.py compatibility - be

a serious candidate?

3. Could I use this list to settle a couple of questions, or would

perhaps a Python developer be willing to work with me to make it compatible? I'm asking this to avoid doing work that would not find acceptance afterwards.

Thanks,

Stefan Krah

---

Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/ssteinerx%40gmail.com

On Tue, 20 Oct 2009 at 09:27, ssteinerX@gmail.com wrote:

Shouldn't this be on python-ideas?

IMO this question is appropriate for python-dev, not python-ideas.

--David

On Tue, 20 Oct 2009 at 09:55, ssteinerX@gmail.com wrote:

On Oct 20, 2009, at 9:43 AM, Stefan Krah wrote:

...

ssteinerX@gmail.com ssteinerx@gmail.com wrote:

...

Shouldn't this be on python-ideas?

I found previous discussions about "Decimal in C" on python-dev, that's why used this list.

python-ideas:

This list is to contain discussion of speculative language ideas for Python for possible inclusion into the language. If an idea gains traction it can then be discussed and honed to the point of becoming a solid proposal to put to either python-dev or python-3000 as appropriate.

I guess it's a fine line...and matter of opinion. No worries.

In this case it isn't a language idea under discussion, but the possible inclusion of an implementation of an idea (and moreover an idea that is already included in Python in another, less efficient, form).

(I'm expecting that both Mark Dickinson and Raymond Hettinger will comment on this thread eventually...)

--David (RDM)

On Tue, Oct 20, 2009 at 3:00 PM, Paul Moore p.f.moore@gmail.com wrote:

2009/10/20 Stefan Krah stefan-usenet@bytereef.org:

...

Hi,

as some of you know, recently I've released an arbitrary precision C library for decimal arithmetic together with a Python module:

http://www.bytereef.org/libmpdec.html http://www.bytereef.org/fastdec.html

Both the library and the module have been tested extensively. Fastdec currently differs from decimal.py in a couple of ways that could be fixed. The license is AGPL, but if there is interest in integrating it into Python I'd release it under a Python-compatible license.

There have been several approaches towards getting C decimal arithmetic into Python:

http://bugs.python.org/issue2486

Fastdec follows Raymond Hettinger's suggestion to provide wrappers for an independent C implementation. Arguments in favour of fastdec are:

* Complete implementation of Mike Cowlishaw's specification

* C library can be tested independently

* Redundant arithmetic module for tests against decimal.py

* Faster than Java BigDecimal

* Compares relatively well in speed against gmpy

To be clear, I would not want to _replace_ decimal.py. Rather I'd like to see a cdecimal module alongside decimal.py.

Why? If it's 100% compatible with decimal.py, just replace it. All the user should see is improved speed. Let's not do another pickle/cpickle.

For example other python implementations might decide to use python version as long as builtin version does not appear. Python versions are usually also better targets for jit than mixed versions. C level versions also usually have more bugs (just statistics), so some people might want to choose pure-python version.

In general - some people have some reasons.

Cheers, fijal

On Tue, Oct 20, 2009 at 2:15 PM, Stefan Krah stefan-usenet@bytereef.org wrote:

1. Are you generally in favour of a C decimal module in Python?

I'm certainly interested in the general idea; whether I'd be in favour of replacing decimal.py with a particular C version would depend on a lot of factors, with code quality, interchangeability with the current decimal module, and maintainability by Python core developers high on the list. There's also the question of what IronPython and Jython would do.

2. Would fastdec - after achieving full decimal.py compatibility - be

a serious candidate?

Definitely. As far as I know it's the only real candidate for a full C version of decimal right now. Other possibilities that I'm aware of:

* I think Raymond Hettinger is working on a C version of decimal. I'm not sure what its current status is. Raymond?

* There's a partially complete rewrite of decimal in C in the sandbox, dating from the Need for Speed sprint in 2006:

* Wrapping the decNumber library directly would make some sense, but I think licensing issues rule this out.

http://svn.python.org/view/sandbox/trunk/decimal-c/

Last time I looked at this it wasn't up to date with the decimal specification: I'm not sure that functions like exp, log and log10 are currently working. Georg Brandl might know better than I do.

3. Could I use this list to settle a couple of questions, or would perhaps

a Python developer be willing to work with me to make it compatible? I'm   asking this to avoid doing work that would not find acceptance afterwards.

I don't see why not. Working closely with one of the core developers on this sounds like a good idea, as well: if the code were to go into Python core, at least one (and preferably more than one) core dev should be familiar enough with it for maintenance. (BTW, I think that *if* fastdec went in it's likely you'd be granted commit privileges at some point.)

It's difficult to gauge the likelihood of eventual acceptance in advance, though. Maybe writing a PEP would be an efficient use of time in this regard? There are certainly some open issues (e.g., what to do with the existing Python module; what should other Python implementations do).

I think my biggest concern is maintenance: we'd be replacing 8500 lines of Python code in a single file, that several of the current core developers understand well, with 30000 (Stefan, is that about accurate?) lines of C in several files, that presumably none of the current core devs is familiar with right now. What happens when (e.g.,) the number-theoretic transform needs updating for one reason or another? Stefan, do you see yourself having time to spend on maintenance of this code for the forseeable future?

BTW, does anyone know the current SLOC count for py3k?

Mark

Mark Dickinson <dickinsm <at> gmail.com> writes:

There are certainly some open issues (e.g., what to do with the existing Python module; what should other Python implementations do).

The existing module could be kept as a fallback. Also, the test suite should be careful to test both implementations (like what is currently done for the io module).

BTW, does anyone know the current SLOC count for py3k?

Here you are, generated using David A. Wheeler's 'SLOCCount':

SLOC Directory SLOC-by-Language (Sorted) 261496 Lib python=261451,sh=45 186769 Modules ansic=173279,asm=9561,sh=3929 53258 Objects ansic=53258 40257 Python ansic=40229,python=28 27220 Tools python=27117,ansic=67,sh=36 18892 Demo python=18511,ansic=377,sh=4 9168 PC ansic=8465,python=703 5840 Include ansic=5840 5799 Parser ansic=3914,python=1885 3485 Misc lisp=2948,python=242,sh=185,ansic=110 3101 Doc python=2306,ansic=795 3030 Mac python=2138,objc=775,sh=109,ansic=8 1666 top_dir python=1140,ansic=286,sh=240 349 PCbuild python=279,ansic=70 337 build ansic=295,python=42 0 Grammar (none)

Totals grouped by language (dominant language first): python: 315842 (50.89%) ansic: 286993 (46.24%) asm: 9561 (1.54%) sh: 4548 (0.73%) lisp: 2948 (0.47%) objc: 775 (0.12%)

Total Physical Source Lines of Code (SLOC) = 620,667

Mark Dickinson schrieb:

• There's a partially complete rewrite of decimal in C in the sandbox,

dating from the Need for Speed sprint in 2006:

http://svn.python.org/view/sandbox/trunk/decimal-c/

Last time I looked at this it wasn't up to date with the decimal specification: I'm not sure that functions like exp, log and log10 are currently working. Georg Brandl might know better than I do.

I haven't touched that code since the sprint, but a student named Mateusz Rukowicz worked on it for a past Summer of Code, I think. I never heard of the outcome of that particular project.

Georg

Mark Dickinson dickinsm@gmail.com wrote:

...

2. Would fastdec - after achieving full decimal.py compatibility - be

a serious candidate?

Definitely. As far as I know it's the only real candidate for a full C version of decimal right now. Other possibilities that I'm aware of:

• I think Raymond Hettinger is working on a C version of decimal.

I'm not sure what its current status is. Raymond?

• There's a partially complete rewrite of decimal in C in the sandbox,

dating from the Need for Speed sprint in 2006:

• Wrapping the decNumber library directly would make some sense,

but I think licensing issues rule this out.

I hope I'm not spreading misinformation, but I see two problems with the use of decNumber as an arbitrary precision library. First, the algorithms are not optimized for large numbers. Then, you have to define the maximum length of the coefficient *at compile time* by setting DECNUMDIGITS.

It's difficult to gauge the likelihood of eventual acceptance in advance, though. Maybe writing a PEP would be an efficient use of time in this regard? There are certainly some open issues (e.g., what to do with the existing Python module; what should other Python implementations do).

Good, I'll do that. I mainly wanted to find out if there is fundamental opposition against C-decimal in the standard library, but this does not seem to be the case.

I think my biggest concern is maintenance: we'd be replacing 8500 lines of Python code in a single file, that several of the current core developers understand well, with 30000 (Stefan, is that about accurate?) lines of C in several files, that presumably none of the current core devs is familiar with right now.

Ok, I think we could say that the sloc-counts are around 2600 lines of Python vs. around 11500-12500 lines of C.

What happens when (e.g.,) the number-theoretic transform needs updating for one reason or another? Stefan, do you see yourself having time to spend on maintenance of this code for the forseeable future?

Yes. Generally speaking, I expect the code to be low-maintenance. It would be even easier if all major compilers supported exact width C99 types and __uint128_t, but this is wishful thinking.

Stefan Krah