Python-Dev October 2009

python-dev@python.org

146 participants
92 discussions

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

by M.-A. Lemburg

Zooko O'Whielacronx wrote: > Folks: > > I'm sorry, I think I didn't make my concern clear. My users, and lots > of other users, are having a problem with incompatibility between > Python binary extension modules. One way to improve the situation > would be if the Python devs would use their "bully pulpit" -- their > unique position as a source respected by all Linux distributions -- > and say "We recommend that Linux distributions use UCS4 for > compatibility with one another". This would not abrogate anyone's > ability to choose their preferred setting nor, as far as I can tell, > would it interfere with the ongoing development of Python. -1 Please note that we did not choose to ship Python as UCS4 binary on Linux - the Linux distributions did. The Python default is UCS2 for a good reason: it's a good trade-off between memory consumption, functionality and performance. As already mentioned, I also don't understand how the changing the Python default on Linux would help your users in any way - if you let distutils compile your extensions, it's automatically going to use the right Unicode setting for you (as well as your users). Unfortunately, this automatic support doesn't help you when shipping e.g. setuptools eggs, but this is a tool problem, not one of Python: setuptools completely ignores the fact that there are two ways to build Python. I'd suggest you ask the tool maintainers to adjust their tools to support the Python Unicode option. > Here are the details: > > I'm the maintainer of several Python packages. I work hard to make it > easy for users, even users who don't know anything about Python, to > use my software. There have been many pain points in this process and > I've spent a lot of time on it for about three years now working on > packaging, including the tools such as setuptools and distutils and > the new "distribute" tool. Python packaging has been improving during > these years -- things are looking up. > > One of the remaining pain points is that I can distribute binaries of > my Python extension modules for Windows or Mac, but if I distribute a > binary Python extension module on Linux, then if the user has a > different UCS2/UCS4 setting then they won't be able to use the > extension module. The current de facto standard for Linux is UCS4 -- > it is used by Debian, Ubuntu, Fedora, RHEL, OpenSuSE, etc. etc.. The > vast majority of Linux users in practice have UCS4, and most binary > Python modules are compiled for UCS4. > > That means that a few folks will get left out. Those folks, from my > experience, are people who built their python executable themselves > without specifying an override for the default, and the smaller Linux > distributions who insist on doing whatever upstream Python devs > recommend instead of doing whatever the other Linux distros are doing. > One of the data points that I reported was a Python interpreter that > was built locally on an Ubuntu server. Since the person building it > didn't know to override the default setting of --enable-unicode, he > ended up with a Python interpreter built for UCS2, even though all the > Python extension modules shipped by Ubuntu were built with UCS4. People building their own Python version will usually also build their own extensions, so I don't really believe that the above scenario is very common. Also note that Python will complain loudly when you try to load a UCS2 extension in a UCS4 build and vice-versa. We've made sure that any extension using the Python Unicode C API has to be built for the same UCS version of Python. This is done by using different names for the C APIs at the C level. > These are not isolated incidents. The following google searches > suggest that a number of people spend time trying to figure out why > Python extension modules fail on their linux systems: > > http://www.google.com/search?q=PyUnicodeUCS4_FromUnicode+undefined+symbol > http://www.google.com/search?q=+PyUnicodeUCS2_FromUnicode+undefined+symbol > http://www.google.com/search?q=_PyUnicodeUCS2_AsDefaultEncodedString+undefi… Perhaps we should add a FAQ entry for these linker errors (which are caused by the mentioned C API changes to prevent mixing UCS version) ?! Here's a quick way to determine you Python Unicode build type: python -c "import sys;print((sys.maxunicode<66000)and'UCS2'or'UCS4')" Perhaps we should include this info as well as an 32/64-bit indicator and the processor type in the Python startup line: # python Python 2.6 (r26:66714, Feb 3 2009, 20:49:49, UCS4, 64-bit, x86_64) [GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2 Type "help", "copyright", "credits" or "license" for more information. This would help users find the right binaries to install as extension. > Another data point is the Mandriva Linux distribution. It is probably > much smaller than Debian, Ubuntu, or RedHat, but it is still one of > the major, well-known distributions. I requested of the Python > maintainer for Mandriva, Michael Scherer, that they switch from UCS2 > to UCS4 in order to reduce compatibility problems like these. His > answer as I understood it was that it is best to follow the > recommendations of the upstream Python devs by using the default > setting instead of choosing a setting for himself. Which is IMHO what all Linux distributions should have done. Distributions should really not be put in charge of upstream coding design decisions. Regards, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 28 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

11 years, 6 months

Re: [Python-Dev] a new setuptools release?

by P.J. Eby

At 01:14 PM 10/6/2009 -0700, Guido van Rossum wrote: >suggest nobody hold their breath waiting for setuptools 0.7. I've never suggested or implied otherwise. But, if you like Distribute so much, why not just add it directly to the stdlib? ;-) AFAIK, the only reason they've had multiple releases of it is to address the issues with its hijacking of setuptools; in a stdlib version all that stuff could be dropped. Otherwise, it'd already be "mature". There are many wins to be had from such a move, not the least of which is that it allows something to go into the stdlib that isn't (branding-wise) associated with me or setuptools, and lets it become owned/supported by an even wider community. Less political bickering, and the some of the technical results I hoped for all along are achieved. Yay, open source.

11 years, 6 months

please consider changing --enable-unicode default to ucs4

by Zooko O'Whielacronx

Folks: I accidentally sent this letter just to MAL when I intended it to python-dev. Please read it, as it explains why the issue I'm raising is not just the "we should switch to ucs4 because it is better" issue that was previously settled by GvR. This is a current, practical problem that is preventing people from distributing and using Python packages with binary extension modules on Linux. Regards, Zooko ---------- Forwarded message ---------- From: Zooko O'Whielacronx <zookog(a)gmail.com> Date: Sun, Sep 27, 2009 at 11:43 AM Subject: Re: [Python-Dev] please consider changing --enable-unicode default to ucs4 To: "M.-A. Lemburg" <mal(a)egenix.com> Folks: I'm sorry, I think I didn't make my concern clear. My users, and lots of other users, are having a problem with incompatibility between Python binary extension modules. One way to improve the situation would be if the Python devs would use their "bully pulpit" -- their unique position as a source respected by all Linux distributions -- and say "We recommend that Linux distributions use UCS4 for compatibility with one another". This would not abrogate anyone's ability to choose their preferred setting nor, as far as I can tell, would it interfere with the ongoing development of Python. Here are the details: I'm the maintainer of several Python packages. I work hard to make it easy for users, even users who don't know anything about Python, to use my software. There have been many pain points in this process and I've spent a lot of time on it for about three years now working on packaging, including the tools such as setuptools and distutils and the new "distribute" tool. Python packaging has been improving during these years -- things are looking up. One of the remaining pain points is that I can distribute binaries of my Python extension modules for Windows or Mac, but if I distribute a binary Python extension module on Linux, then if the user has a different UCS2/UCS4 setting then they won't be able to use the extension module. The current de facto standard for Linux is UCS4 -- it is used by Debian, Ubuntu, Fedora, RHEL, OpenSuSE, etc. etc.. The vast majority of Linux users in practice have UCS4, and most binary Python modules are compiled for UCS4. That means that a few folks will get left out. Those folks, from my experience, are people who built their python executable themselves without specifying an override for the default, and the smaller Linux distributions who insist on doing whatever upstream Python devs recommend instead of doing whatever the other Linux distros are doing. One of the data points that I reported was a Python interpreter that was built locally on an Ubuntu server. Since the person building it didn't know to override the default setting of --enable-unicode, he ended up with a Python interpreter built for UCS2, even though all the Python extension modules shipped by Ubuntu were built with UCS4. These are not isolated incidents. The following google searches suggest that a number of people spend time trying to figure out why Python extension modules fail on their linux systems: http://www.google.com/search?q=PyUnicodeUCS4_FromUnicode+undefined+symbol http://www.google.com/search?q=+PyUnicodeUCS2_FromUnicode+undefined+symbol http://www.google.com/search?q=_PyUnicodeUCS2_AsDefaultEncodedString+undefi… Another data point is the Mandriva Linux distribution. It is probably much smaller than Debian, Ubuntu, or RedHat, but it is still one of the major, well-known distributions. I requested of the Python maintainer for Mandriva, Michael Scherer, that they switch from UCS2 to UCS4 in order to reduce compatibility problems like these. His answer as I understood it was that it is best to follow the recommendations of the upstream Python devs by using the default setting instead of choosing a setting for himself. (Now we could implement a protocol which would show whether a given Python package was compiled for UCS2 or UCS4. That would be good. Hopefully it would make incompatibility more explicit and understandable to users. Here is a ticket for that -- which project I am contributing to: http://bugs.python.org/setuptools/issue78 . However, even if we implement that feature in the distribute tool (the successor to setuptools), users who build their own python or who use a Linux distribution that follows upstream configuration defaults will still be unable to use most Python packages with compiled extension modules.) In a message on this thread, MvL wrote: > "For that reason I think it's also better that the configure script > continues to default to UTF-16 -- this will give the UTF-16 support > code the necessary exercise." > > This is effectively a BDFL pronouncement. Nothing has changed the > validity of the premise of the statement, so the conclusion remains > valid, as well. My understand of the earlier thread was that someone suggested that UCS4 would be technically better and GvR decided that there were technical reasons to continue actively maintaining the UCS2 code. This thread is different: I'm saying that users are suffering packaging problems and asking for help with that. The way that python-dev can help is to make it so that people who choose "Whatever the upstream devs prefer" (--enable-unicde) on Linux get the de facto standard setting. In the earlier thread, GvR wrote: "I think we should continue to leave this up to the distribution. AFAIK many Linux distros already use UCS4 for everything anyway.". But at least some distributions are asking the upstream Python devs to choose for them, by leaving the setting at the default. Hm, pondering GvR's words: "I think we should continue to leave this up to the distribution", I have a new proposal: make it so that *on Linux only* "--enable-unicode" errors out with an error message saying "please choose either --enable-unicode=ucs2 or --enable-unicode=ucs4. ucs4 is the most widely used setting on Linux. See http://python.org/wiki/UCS2_vs_UCS4 for details.". This would force those Linux distributions who are *not* currently deciding to decide. Thank you for your attention. Regards, Zooko

11 years, 6 months

Package install failures in 2.6.3

by Ned Deily

There has been some discussion on the distutils-sig list over the past few days about a change to distutils released with 2.6.3 which inadvertently causes a number (all?) packages with C extensions that use setuptools/easy_install to fail during builds/installs with a rather cryptic error message. A number of popular packages are affected directly or indirectly through dependencies. The problem is summarized in the issue tracker here; http://bugs.python.org/issue7064 which includes links to the distutils list archive and the setuptools and Distribute project bug trackers. Assuming that distutils is not changed in a forthcoming 2.6.4, users and, possibly package maintainers, will need to take some action: either migrate from setuptools to Distribute (an easy migration), or wait for a fix to setuptools or the package or to the distribution (if there is one). I'm not proposing to debate the merits of all of the options here. However, if a 2.6.4 release is to be pushed out quickly for other reasons, a one-time window of opportunity would be available and it would be prudent to at least consider the possibility of a distutils fix. In any case, it seems like there should be some effort to make package maintainers aware of the problem so they can take whatever steps necessary to update their packages or help files or web sites etc. -- Ned Deily, nad(a)acm.org

11 years, 6 months

Python byte-compiled and optimized code

by Swapnil Talekar

I am working on deploying Python on VxWorks platform as a part of project for my company. Accordingly, I would like to know couple of things from python's core-developers. Although I think I already know the answers for most of the questions, we need a confirmation from the community 1) Is the byte-compiled .pyc file and optimized .pyo file platform-independent?(including python versions 3.x) *If yes, is it guaranteed to stay that way in future?* 2) If the the generation of .pyc file fails (say, due to write protected area), does that stop the script execution? 3) Is it possible to redirect the location of the generation of .pyc files to other than that of the corresponding .py files? Regards, Swapnil

11 years, 6 months

Announcing PEP 3136

by Hatem Nassrat

Tue Jul 3 10:14:17 CEST 2007, Guido van Rossum wrote: > On 6/30/07, Matt Chisholm <matt-python at theory.org> wrote: > > I've created and submitted a new PEP proposing support for labels in > > Python's break and continue statements. Georg Brandl has graciously > > added it to the PEP list as PEP 3136: > > > > http://www.python.org/dev/peps/pep-3136/ > > I think this is a good summary of various proposals that have been > floated in the past, plus some new ones. As a PEP, it falls short > because it doesn't pick a solution but merely offers a large menu of > possible options. Also, there is nothing about implementation yet. > > However, I'm rejecting it on the basis that code so complicated to > require this feature is very rare. In most cases there are existing > work-arounds that produce clean code, for example using 'return'. I agree that this feature will only serve as a quick hack and in many cases it would be misused and ugly code will be the result. However, it seems that there are some shortcuts that have sneaked into python (I am a fairly new Python programmer, only since late 2.4, so don't shoot me). The specific one of which I speak about is: while_stmt ::= "while" expression ":" suite ["else" ":" suite] for_stmt ::= "for" target_list "in" expression_list ":" suite ["else" ":" suite] try1_stmt ::= "try" ":" suite ("except" [expression ["as" target]] ":" suite)+ ["else" ":" suite] ["finally" ":" suite] All these else's seem like shortcuts to me. I did find a use for them, once I found out they existed, but I get butterflies whenever I do. -- Hatem Nassrat

11 years, 6 months

Re: [Python-Dev] Distutils ML wrap-up: setup.cfg new format

by P.J. Eby

At 07:00 PM 9/23/2009 +0200, Tarek Ziadé wrote: >While it's great to have Philipp being part of our distutils design >discussions, >for his experience, I am not concerned of not having him in this "internal >consensus" since Setuptools is not maintained anymore. > >He said some months ago, he would work on a brand new setuptools >version with zero >dependency to distutils. But it's still vaporware (from his own >words), and the previous version is unmaintained for a year, so it was >forked. Here's what actually happened, if anyone cares. Tarek and friends announced a fork of setuptools. I reviewed the work and saw that -- for the most part -- I was happy with it, and opined as how I might be willing to bless the the "package inquisition" team as official maintainers of the 0.6 branch of setuptools, so that I could work on the fun bits I've long planned for 0.7, but never felt free to start on while there was so much still needing to be done on 0.6. However, just as I mentioned this, and suggested an option for what I could do that would be helpful to his Distribute 0.7 project as well as various other tools (e.g. implementing some of Jim Fulton's long-requested features for better modularization of setuptools), Tarek accused me of somehow trying to undermine his plans. In addition, it appears Tarek was also offended by my earlier statement that there were only a few people in the Python community who had *already* earned my implicit trust to not only hack on setuptools unsupervised, but also to take over its *future* direction and BDFL-ship. (For example, Jim Fulton and Ian Bicking.) Tarek, however, appears to have taken this to mean that I personally thought he was an incompetent programmer or something (when I actually had no opinion one way or the other), and ever since he has taken to levelling potshots like the above at me on a semi-regular basis. I've tried to ignore this and play nice, because he is actually working on this stuff and I am not. But it's hard for me to actually give any help in practice, if Tarek is too busy projecting hidden plots onto everything I say and do. If you read Tarek's distutils-sig posts, it appears my already-existing trust in Ian and Jim was not only a personal insult to Tarek, but also a plot to ensure that nobody with any time to do so would ever work on setuptools, just as my excitement about working on setuptools again was a plot to steal thunder from his fork. All I want is for good stuff to happen for setuptools users and Python users in general, so I don't think all the suspicion and backbiting is merited. I certainly don't appreciate it, and I would like it to stop. It also isn't even relevant to the thread, since my lack of work on setuptools says exactly zero about the merits or lack thereof of Tarek's proposals for the distutils! Hell, I *support* the bulk of Tarek's setup.cfg proposal, and don't even object to him Pronouncing it or cutting off the discussion! My only issue on Python-Dev was his inaccurate implication that it was a SIG consensus rather than a pronouncement on it. There is and was no need for any of this to get personal, and I have continually strived to keep my posts here and distutils-sig civil, even when I didn't feel like being civil in response to Tarek's jabs. I have in fact bent over backwards to be *nice* to Tarek, because he seemed so damn sensitive about everything. Apparently, however, this does not actually help things. :-(

11 years, 6 months

Old libc's isspace() bug on FreeBSD brings back on Mac OS X?

by INADA Naoki

I found this hg's issue. http://mercurial.selenic.com/bts/msg8375 I think below fix is not enabled on Mac OS X. http://svn.python.org/view/python/trunk/Include/pyport.h?view=diff&pathrev=… I can't confirm it because I am not Mac OS X user. Can anyone confirm it? -- Naoki INADA <songofacandy(a)gmail.com>

11 years, 6 months

eggs now mandatory for pypi?

by Fredrik Lundh

it's revews like this that makes me wonder if releasing open source is a good idea: no egg - worst seen ever, remove it from pypi or provide an egg (jensens, 2009-10-05, 0 points) </F>

11 years, 6 months

Re: [Python-Dev] Package install failures in 2.6.3

by P.J. Eby

At 02:22 PM 10/5/2009 +0200, Tarek Ziadé wrote: >Setuptools development has been discontinued for a year, and does >patches on Distutils code. Some of these patches are sensitive to any change >made on Distutils, wether those changes are internal or not. Setuptools is also not the only thing out there that subclasses distutils commands in general, or the build_ext command in particular. Numpy, Twisted, the mx extensions... there are plenty of things out there that subclass distutils commands, quite in adherence to the rules. (Note too that subclassing != patching, and the ability to subclass and substitute distutils commands is documented.) It's therefore not appropriate to treat the issue as if it were setuptools-specific; it could have broken any other major (or minor) package's subclassing of the build_ext command.

11 years, 6 months

← Newer
1
2
3
4
5
6
7
8
9
10
Older →

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev October 2009