Libraries implementation in C or in Python? Evolution of Python, Jython...

[Tim Peters]

[Ron Stephens]
> Are Python's libraries generally written in C or in Python,

Both; e.g., look in your Lib/ directory and you'll find a couple megabytes
of Python library source.

> and why?

Complex coding is easier in Python; low-level fiddling is easier and runs
faster in C.

> Someone commented on another thread that the re.search() function  ran
> faster in its 1.5.2 version than in its 2.1 version, because in 1.5.2 it
> was implemented in C and in 2.1 it was implemented in Python. It was
> suggested to instead use the "pre" module, but I see it is old and may
> become obsolete? I am curious as to the reasoning behind why these kinds
> of changes and decisions are made?

sre was primarily motivated by the new need to support Unicode (pre couldn't
handle it).  Fredrik also thought he could write a matching engine faster
than pre, and turns out he was right.  But any project has to be completed
in finite time, and there can't be a guarantee that every case will be no
slower than before.  In the way I use regexps, I almost always find that sre
is faster.

There are no "general reasons" here:  each thing has its own reasons.

> In general, as Python evolves from version to version, is Python code
> likely to get slightly faster in execution time, or slightly slower?

Neither, really, although the trend since 1.5.2 has been to get a bit slower
each time.  Two things account for that:

1. New features can cost.  For example, rich comparisons added many
   more possibilities for the runtime code to sort through, and *simple*
   comparisons got significantly slower as a more-than-less direct
   result.  Or cyclic gc removes worries about trash cycles leaking,
   but it takes time to find those cycles (before they simply leaked).

2. Fixing ancient bugs.  For example, it's possible to crash any Python
   through 2.1 by filling a dict with class instance keys, where the
   class overrides comparison with a function that mutates the dict
   while it's comparing.   This isn't sane, but it is possible.  Plugging
   this hole required additional code in the dict lookup routines, and
   that slows them down.  Plugging holes often costs a little, and then
   plugging many holes adds up to more than a little.

OTOH, current CVS Python is significantly quicker than 2.1 by several
accounts, as people have found 2.1 hotspots and added patches to cool them
off.  On the third hand, Python's speed is also significantly affected by
the compiler and OS; for example, on identical hardware, pystone runs
significantly faster under Win2K+MSVC than under Linux+gcc, but test_bufio
runs 10x faster under Linux; etc.  "The speed" of a x-platform app is a
slippery concept, because it's not independent of platform.

> Mind you, I am a Pythonista because of the numerous other beauties of
> the language, not because of any speed issues; and I can certainly see
> why Python should be and is optimized for other criteria, not speed, and
> if this sometimes means a little step backwards in speed of code
> execution then so be it. But I am just curious about the trade-offs, and
> why such decisions are made.

Decisions to accept or reject a new feature don't generally take speed into
account, because it's generally impossible to predict in advance.  What
happens instead is that if an accepted feature is "too slow", someone who
cares enough will speed it over the next release cycle.  It's actually rare
that anyone cares enough to do that, though.

> There have been a few threads about 2.x versions of Python being a
> little slower, in terms of the execution speed of similar code, than
> 1.5.2. Is this speed decrement likely to be permanent?

It's impossible to say without specific code to time on a specific platform.
The only way you'll ever get a clear answer to something as vague as
"similar code is a little slower -- permanent?" is if a developer chimes in
with "Doh! I *knew* I shouldn't have left sleep(1) in the inner loop"
<wink>.

> ...
> I hesitate to ask any of this, because my extreme state of newbieness
> and lack of knowledge do not give me any right to even speak up and ask
> such questions. I only rush in where I should fear to tread because the
> tangible beauty of the evolutionary process of a work of art such as the
> Python programming language excites my curiosity and sense of wonder.

Rest assured (or appalled <wink>) that Python is pretty much the same as any
large project in this respect:  the speed freaks worry about speed, the
feature freaks push for new features, the good-old-days freaks complain
about change, and nobody has enough time to do all they want to do.

> And then there is Jython. Is Python the mother and Java the father, or
> is it the other way around?

They're more like teenage siblings, who can't believe their parents ever had
sex so each suspects the other of being adopted.

> Does anyone else see the evolution of Python, Jython and future
> offspring as a living work of art?

Undoubtedly.  But they usually keep it to themselves <wink>.