Zooko O'Whielacronx wrote:
> I'm sorry, I think I didn't make my concern clear. My users, and lots
> of other users, are having a problem with incompatibility between
> Python binary extension modules. One way to improve the situation
> would be if the Python devs would use their "bully pulpit" -- their
> unique position as a source respected by all Linux distributions --
> and say "We recommend that Linux distributions use UCS4 for
> compatibility with one another". This would not abrogate anyone's
> ability to choose their preferred setting nor, as far as I can tell,
> would it interfere with the ongoing development of Python.
Please note that we did not choose to ship Python as UCS4 binary
on Linux - the Linux distributions did.
The Python default is UCS2 for a good reason: it's a good trade-off
between memory consumption, functionality and performance.
As already mentioned, I also don't understand how the changing
the Python default on Linux would help your users in any way -
if you let distutils compile your extensions, it's automatically
going to use the right Unicode setting for you (as well as your
Unfortunately, this automatic support doesn't help you when
shipping e.g. setuptools eggs, but this is a tool problem,
not one of Python: setuptools completely ignores the fact
that there are two ways to build Python.
I'd suggest you ask the tool maintainers to adjust their tools
to support the Python Unicode option.
> Here are the details:
> I'm the maintainer of several Python packages. I work hard to make it
> easy for users, even users who don't know anything about Python, to
> use my software. There have been many pain points in this process and
> I've spent a lot of time on it for about three years now working on
> packaging, including the tools such as setuptools and distutils and
> the new "distribute" tool. Python packaging has been improving during
> these years -- things are looking up.
> One of the remaining pain points is that I can distribute binaries of
> my Python extension modules for Windows or Mac, but if I distribute a
> binary Python extension module on Linux, then if the user has a
> different UCS2/UCS4 setting then they won't be able to use the
> extension module. The current de facto standard for Linux is UCS4 --
> it is used by Debian, Ubuntu, Fedora, RHEL, OpenSuSE, etc. etc.. The
> vast majority of Linux users in practice have UCS4, and most binary
> Python modules are compiled for UCS4.
> That means that a few folks will get left out. Those folks, from my
> experience, are people who built their python executable themselves
> without specifying an override for the default, and the smaller Linux
> distributions who insist on doing whatever upstream Python devs
> recommend instead of doing whatever the other Linux distros are doing.
> One of the data points that I reported was a Python interpreter that
> was built locally on an Ubuntu server. Since the person building it
> didn't know to override the default setting of --enable-unicode, he
> ended up with a Python interpreter built for UCS2, even though all the
> Python extension modules shipped by Ubuntu were built with UCS4.
People building their own Python version will usually also build
their own extensions, so I don't really believe that the above
scenario is very common.
Also note that Python will complain loudly when you try to load
a UCS2 extension in a UCS4 build and vice-versa. We've made sure
that any extension using the Python Unicode C API has to be built
for the same UCS version of Python. This is done by using different
names for the C APIs at the C level.
> These are not isolated incidents. The following google searches
> suggest that a number of people spend time trying to figure out why
> Python extension modules fail on their linux systems:
Perhaps we should add a FAQ entry for these linker errors
(which are caused by the mentioned C API changes to prevent
mixing UCS version) ?!
Here's a quick way to determine you Python Unicode build type:
python -c "import sys;print((sys.maxunicode<66000)and'UCS2'or'UCS4')"
Perhaps we should include this info as well as an 32/64-bit indicator
and the processor type in the Python startup line:
Python 2.6 (r26:66714, Feb 3 2009, 20:49:49, UCS4, 64-bit, x86_64)
[GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2
Type "help", "copyright", "credits" or "license" for more information.
This would help users find the right binaries to install as
> Another data point is the Mandriva Linux distribution. It is probably
> much smaller than Debian, Ubuntu, or RedHat, but it is still one of
> the major, well-known distributions. I requested of the Python
> maintainer for Mandriva, Michael Scherer, that they switch from UCS2
> to UCS4 in order to reduce compatibility problems like these. His
> answer as I understood it was that it is best to follow the
> recommendations of the upstream Python devs by using the default
> setting instead of choosing a setting for himself.
Which is IMHO what all Linux distributions should have done.
Distributions should really not be put in charge of upstream
coding design decisions.
Professional Python Services directly from the Source (#1, Sep 28 2009)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
::: Try our new mxODBC.Connect Python Database Interface for free ! ::::
eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
Registered at Amtsgericht Duesseldorf: HRB 46611
At 01:14 PM 10/6/2009 -0700, Guido van Rossum wrote:
>suggest nobody hold their breath waiting for setuptools 0.7.
I've never suggested or implied otherwise.
But, if you like Distribute so much, why not just add it directly to
the stdlib? ;-)
AFAIK, the only reason they've had multiple releases of it is to
address the issues with its hijacking of setuptools; in a stdlib
version all that stuff could be dropped. Otherwise, it'd already be "mature".
There are many wins to be had from such a move, not the least of
which is that it allows something to go into the stdlib that isn't
(branding-wise) associated with me or setuptools, and lets it become
owned/supported by an even wider community.
Less political bickering, and the some of the technical results I
hoped for all along are achieved. Yay, open source.
I accidentally sent this letter just to MAL when I intended it to
python-dev. Please read it, as it explains why the issue I'm raising
is not just the "we should switch to ucs4 because it is better" issue
that was previously settled by GvR. This is a current, practical
problem that is preventing people from distributing and using Python
packages with binary extension modules on Linux.
---------- Forwarded message ----------
From: Zooko O'Whielacronx <zookog(a)gmail.com>
Date: Sun, Sep 27, 2009 at 11:43 AM
Subject: Re: [Python-Dev] please consider changing --enable-unicode
default to ucs4
To: "M.-A. Lemburg" <mal(a)egenix.com>
I'm sorry, I think I didn't make my concern clear. My users, and lots
of other users, are having a problem with incompatibility between
Python binary extension modules. One way to improve the situation
would be if the Python devs would use their "bully pulpit" -- their
unique position as a source respected by all Linux distributions --
and say "We recommend that Linux distributions use UCS4 for
compatibility with one another". This would not abrogate anyone's
ability to choose their preferred setting nor, as far as I can tell,
would it interfere with the ongoing development of Python.
Here are the details:
I'm the maintainer of several Python packages. I work hard to make it
easy for users, even users who don't know anything about Python, to
use my software. There have been many pain points in this process and
I've spent a lot of time on it for about three years now working on
packaging, including the tools such as setuptools and distutils and
the new "distribute" tool. Python packaging has been improving during
these years -- things are looking up.
One of the remaining pain points is that I can distribute binaries of
my Python extension modules for Windows or Mac, but if I distribute a
binary Python extension module on Linux, then if the user has a
different UCS2/UCS4 setting then they won't be able to use the
extension module. The current de facto standard for Linux is UCS4 --
it is used by Debian, Ubuntu, Fedora, RHEL, OpenSuSE, etc. etc.. The
vast majority of Linux users in practice have UCS4, and most binary
Python modules are compiled for UCS4.
That means that a few folks will get left out. Those folks, from my
experience, are people who built their python executable themselves
without specifying an override for the default, and the smaller Linux
distributions who insist on doing whatever upstream Python devs
recommend instead of doing whatever the other Linux distros are doing.
One of the data points that I reported was a Python interpreter that
was built locally on an Ubuntu server. Since the person building it
didn't know to override the default setting of --enable-unicode, he
ended up with a Python interpreter built for UCS2, even though all the
Python extension modules shipped by Ubuntu were built with UCS4.
These are not isolated incidents. The following google searches
suggest that a number of people spend time trying to figure out why
Python extension modules fail on their linux systems:
Another data point is the Mandriva Linux distribution. It is probably
much smaller than Debian, Ubuntu, or RedHat, but it is still one of
the major, well-known distributions. I requested of the Python
maintainer for Mandriva, Michael Scherer, that they switch from UCS2
to UCS4 in order to reduce compatibility problems like these. His
answer as I understood it was that it is best to follow the
recommendations of the upstream Python devs by using the default
setting instead of choosing a setting for himself.
(Now we could implement a protocol which would show whether a given
Python package was compiled for UCS2 or UCS4. That would be good.
Hopefully it would make incompatibility more explicit and
understandable to users. Here is a ticket for that -- which project I
am contributing to: http://bugs.python.org/setuptools/issue78 .
However, even if we implement that feature in the distribute tool (the
successor to setuptools), users who build their own python or who use
a Linux distribution that follows upstream configuration defaults will
still be unable to use most Python packages with compiled extension
In a message on this thread, MvL wrote:
> "For that reason I think it's also better that the configure script
> continues to default to UTF-16 -- this will give the UTF-16 support
> code the necessary exercise."
> This is effectively a BDFL pronouncement. Nothing has changed the
> validity of the premise of the statement, so the conclusion remains
> valid, as well.
My understand of the earlier thread was that someone suggested that
UCS4 would be technically better and GvR decided that there were
technical reasons to continue actively maintaining the UCS2 code.
This thread is different: I'm saying that users are suffering
packaging problems and asking for help with that. The way that
python-dev can help is to make it so that people who choose "Whatever
the upstream devs prefer" (--enable-unicde) on Linux get the de facto
In the earlier thread, GvR wrote: "I think we should continue to leave
this up to the distribution. AFAIK many Linux distros already use UCS4
for everything anyway.". But at least some distributions are asking
the upstream Python devs to choose for them, by leaving the setting at
Hm, pondering GvR's words: "I think we should continue to leave this
up to the distribution", I have a new proposal: make it so that *on
Linux only* "--enable-unicode" errors out with an error message saying
"please choose either --enable-unicode=ucs2 or --enable-unicode=ucs4.
ucs4 is the most widely used setting on Linux. See
http://python.org/wiki/UCS2_vs_UCS4 for details.". This would force
those Linux distributions who are *not* currently deciding to decide.
Thank you for your attention.
There has been some discussion on the distutils-sig list over the past
few days about a change to distutils released with 2.6.3 which
inadvertently causes a number (all?) packages with C extensions that use
setuptools/easy_install to fail during builds/installs with a rather
cryptic error message. A number of popular packages are affected
directly or indirectly through dependencies.
The problem is summarized in the issue tracker here;
which includes links to the distutils list archive and the setuptools
and Distribute project bug trackers.
Assuming that distutils is not changed in a forthcoming 2.6.4, users
and, possibly package maintainers, will need to take some action: either
migrate from setuptools to Distribute (an easy migration), or wait for a
fix to setuptools or the package or to the distribution (if there is
I'm not proposing to debate the merits of all of the options here.
However, if a 2.6.4 release is to be pushed out quickly for other
reasons, a one-time window of opportunity would be available and it
would be prudent to at least consider the possibility of a distutils fix.
In any case, it seems like there should be some effort to make package
maintainers aware of the problem so they can take whatever steps
necessary to update their packages or help files or web sites etc.
I am working on deploying Python on VxWorks platform as a part of project
for my company. Accordingly, I would like to know couple of things from
python's core-developers. Although I think I already know the answers for
most of the questions, we need a confirmation from the community
1) Is the byte-compiled .pyc file and optimized .pyo file
platform-independent?(including python versions 3.x) *If yes, is it
guaranteed to stay that way in future?*
2) If the the generation of .pyc file fails (say, due to write protected
area), does that stop the script execution?
3) Is it possible to redirect the location of the generation of .pyc files
to other than that of the corresponding .py files?
Tue Jul 3 10:14:17 CEST 2007, Guido van Rossum wrote:
> On 6/30/07, Matt Chisholm <matt-python at theory.org> wrote:
> > I've created and submitted a new PEP proposing support for labels in
> > Python's break and continue statements. Georg Brandl has graciously
> > added it to the PEP list as PEP 3136:
> > http://www.python.org/dev/peps/pep-3136/
> I think this is a good summary of various proposals that have been
> floated in the past, plus some new ones. As a PEP, it falls short
> because it doesn't pick a solution but merely offers a large menu of
> possible options. Also, there is nothing about implementation yet.
> However, I'm rejecting it on the basis that code so complicated to
> require this feature is very rare. In most cases there are existing
> work-arounds that produce clean code, for example using 'return'.
I agree that this feature will only serve as a quick hack and in many
cases it would be misused and ugly code will be the result. However,
it seems that there are some shortcuts that have sneaked into python
(I am a fairly new Python programmer, only since late 2.4, so don't
shoot me). The specific one of which I speak about is:
while_stmt ::= "while" expression ":" suite
["else" ":" suite]
for_stmt ::= "for" target_list "in" expression_list ":" suite
["else" ":" suite]
try1_stmt ::= "try" ":" suite
("except" [expression ["as" target]] ":" suite)+
["else" ":" suite]
["finally" ":" suite]
All these else's seem like shortcuts to me. I did find a use for them,
once I found out they existed, but I get butterflies whenever I do.
At 07:00 PM 9/23/2009 +0200, Tarek Ziadé wrote:
>While it's great to have Philipp being part of our distutils design
>for his experience, I am not concerned of not having him in this "internal
>consensus" since Setuptools is not maintained anymore.
>He said some months ago, he would work on a brand new setuptools
>version with zero
>dependency to distutils. But it's still vaporware (from his own
>words), and the previous version is unmaintained for a year, so it was
Here's what actually happened, if anyone cares. Tarek and friends
announced a fork of setuptools. I reviewed the work and saw that --
for the most part -- I was happy with it, and opined as how I might
be willing to bless the the "package inquisition" team as official
maintainers of the 0.6 branch of setuptools, so that I could work on
the fun bits I've long planned for 0.7, but never felt free to start
on while there was so much still needing to be done on 0.6.
However, just as I mentioned this, and suggested an option for what I
could do that would be helpful to his Distribute 0.7 project as well
as various other tools (e.g. implementing some of Jim Fulton's
long-requested features for better modularization of setuptools),
Tarek accused me of somehow trying to undermine his plans.
In addition, it appears Tarek was also offended by my earlier
statement that there were only a few people in the Python community
who had *already* earned my implicit trust to not only hack on
setuptools unsupervised, but also to take over its *future* direction
and BDFL-ship. (For example, Jim Fulton and Ian Bicking.)
Tarek, however, appears to have taken this to mean that I personally
thought he was an incompetent programmer or something (when I
actually had no opinion one way or the other), and ever since he has
taken to levelling potshots like the above at me on a semi-regular basis.
I've tried to ignore this and play nice, because he is actually
working on this stuff and I am not. But it's hard for me to actually
give any help in practice, if Tarek is too busy projecting hidden
plots onto everything I say and do.
If you read Tarek's distutils-sig posts, it appears my
already-existing trust in Ian and Jim was not only a personal insult
to Tarek, but also a plot to ensure that nobody with any time to do
so would ever work on setuptools, just as my excitement about working
on setuptools again was a plot to steal thunder from his fork.
All I want is for good stuff to happen for setuptools users and
Python users in general, so I don't think all the suspicion and
backbiting is merited. I certainly don't appreciate it, and I would
like it to stop. It also isn't even relevant to the thread, since my
lack of work on setuptools says exactly zero about the merits or lack
thereof of Tarek's proposals for the distutils!
Hell, I *support* the bulk of Tarek's setup.cfg proposal, and don't
even object to him Pronouncing it or cutting off the discussion! My
only issue on Python-Dev was his inaccurate implication that it was a
SIG consensus rather than a pronouncement on it. There is and was no
need for any of this to get personal, and I have continually strived
to keep my posts here and distutils-sig civil, even when I didn't
feel like being civil in response to Tarek's jabs. I have in fact
bent over backwards to be *nice* to Tarek, because he seemed so damn
sensitive about everything. Apparently, however, this does not
actually help things. :-(
At 02:22 PM 10/5/2009 +0200, Tarek Ziadé wrote:
>Setuptools development has been discontinued for a year, and does
>patches on Distutils code. Some of these patches are sensitive to any change
>made on Distutils, wether those changes are internal or not.
Setuptools is also not the only thing out there that subclasses
distutils commands in general, or the build_ext command in
particular. Numpy, Twisted, the mx extensions... there are plenty
of things out there that subclass distutils commands, quite in
adherence to the rules. (Note too that subclassing != patching, and
the ability to subclass and substitute distutils commands is documented.)
It's therefore not appropriate to treat the issue as if it were
setuptools-specific; it could have broken any other major (or minor)
package's subclassing of the build_ext command.