Mailman 3 October 2009 - Python-Dev

please consider changing --enable-unicode default to ucs4
by Zooko O'Whielacronx Oct. 8, 2009

Oct. 8, 2009

Dear Pythonistas: This issue causes serious problems. Users occasionally get binaries built for a compatible Linux and Python version but with a different UCS2-vs-UCS4 setting, and those users get mysterious memory corruption errors which are hard to diagnose. It is possible that these situations also open up security vulnerabilities. A couple such instances are documented on http://bugs.python.org/setuptools/issue78, but you can find more by googling. I would like to get this problem fixed!… [View More] In order to help address this issue I sampled what UCS size is used by python executables in the wild. I instrumented a few buildslaves that are contributed by various people to the Tahoe-LAFS project to print out their platform, python version, and sys.maxunicode. The full results are appended below. maxunicode: 1114111 means that python executable was configured with --enable-unicode=ucs4, and maxunicode: 65535 means that python executable was configured with --enable-unicode=ucs2 or just with --enable-unicode . The only incompatibilities that I found are because some packagers have deliberately set UCS4 configuration and other packagers have left the default setting. In the three cases where someone configured python with UCS2, one of the three is certainly an accident (a custom-built python executable on an Ubuntu server) and the other two just use the default instead of specifically configuring ucs2 in their configurations of Python and I suspect that they don't know the difference and that it was an accident that they built a Python incompatible with other distributions of their operating system. In sum, while it would be good to add the unicode setting to the platform's ABI (as discussed in setuptools ticket #78), it would also be good to make the default value be UCS4 instead of UCS2. This would fix all three of the potential incompatibilities that I found (listed below), and once we have proper inclusion of the unicode setting in the ABI in order to prevent the memory corruption, defaulting to UCS4 would increase the likelihood that a binary built on one distribution would be usable on another. I'm sure that someone can come up with a reason why UCS2 is better than UCS4, but I'm also sure that the benefits of compatibility outweigh any benefits of UCS2 encoding, and that the widespread use of UCS4 demonstrates that there is nothing fatally wrong with it, and that people who really value UCS2 encoding more than compatibility can choose that for themselves by explicitly setting UCS2. Let me restate that I am not suggesting taking away anyone's options, only making the setting for people who don't specify default to the compatible option. Hm, I guess that means that it should default to UCS2 on Windows and Mac and to UCS4 on Linux and Solaris. Regards, Zooko Ubuntu 6.10 "edgy" i386: python: 2.4.4c1 (#2, Mar 7 2008, 03:03:38) [GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)], maxunicode: 1114111 Ubuntu 7.04 "feisty": python: 2.5.1 (r251:54863, Jul 31 2008, 22:53:39) [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)], maxunicode: 1114111 Ubuntu 7.10 "gutsy" i386: python: 2.5.1 (r251:54863, Jul 31 2008, 23:17:40) [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)], maxunicode: 1114111 Ubuntu 8.04 "hardy" amd64: python: 2.5.2 (r252:60911, Jul 22 2009, 15:33:10) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)], maxunicode: 1114111 Ubuntu 8.04 "hardy" i386: *custom* python: 2.6 (r26:66714, Oct 2 2008, 13:40:28) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)], maxunicode: 65535 Ubuntu 8.04 "hardy" i386: python: 2.5.2 (r252:60911, Jul 22 2009, 15:35:03) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)], maxunicode: 1114111 Ubuntu 9.04 "jaunty" amd64: *custom* python: 2.6.2 (release26-maint, Apr 19 2009, 01:58:18) [GCC 4.3.3], maxunicode: 1114111 Debian 4.0 "etch" i386: python: 2.4.4 (#2, Oct 22 2008, 19:52:44) [GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)], maxunicode: 1114111 Debian 5.0 "lenny" i386: python: 2.5.2 (r252:60911, Jan 4 2009, 17:40:26) [GCC 4.3.2], maxunicode: 1114111 Debian 5.0 "lenny" amd64: python: 2.5.2 (r252:60911, Jan 4 2009, 21:59:32) [GCC 4.3.2], maxunicode: 1114111 Debian 5.0 "lenny" armv5tel: python: 2.5.2 (r252:60911, Jan 5 2009, 02:00:00) [GCC 4.3.2], maxunicode: 1114111 Debian unstable "squeeze/sid" i386: python: 2.5.4 (r254:67916, Feb 17 2009, 20:16:45) [GCC 4.3.3], maxunicode: 1114111 Fedora 11 "leonidas" amd64: python: 2.6 (r26:66714, Jul 4 2009, 17:37:13) [GCC 4.4.0 20090506 (Red Hat 4.4.0-4)], maxunicode: 1114111 ArchLinux: python: 2.6.2 (r262:71600, Jul 20 2009, 02:23:30) [GCC 4.4.0 20090630 (prerelease)], maxunicode: 65535 NetBSD 4: python: 2.5.2 (r252:60911, Mar 20 2009, 14:00:07) [GCC 4.1.2 20060628 prerelease (NetBSD nb2 20060711)], maxunicode: 65535 OpenSolaris SunOS-5.11-i86pc-i386-32bit: python: 2.4.4 (#1, Mar 10 2009, 09:35:36) [C], maxunicode: 65535 Nexenta NCP1 SunOS-5.11-i86pc-i386-32bit: python: 2.4.3 (#2, May 3 2006, 19:12:42) [GCC 4.0.3 (GNU_OpenSolaris 4.0.3-1nexenta4)], maxunicode: 1114111 Mac OS 10.6 "snow leopard" i386: python: 2.6.1 (r261:67515, Jul 7 2009, 23:51:51) [GCC 4.2.1 (Apple Inc. build 5646)], maxunicode: 65535 Mac OS 10.5 "leopard" i386: python: 2.5.1 (r251:54863, Feb 6 2009, 19:02:12) [GCC 4.0.1 (Apple Inc. build 5465)], maxunicode: 65535 Mac OS 10.4 "tiger" *custom* python: 2.5.4 (release25-maint:72153M, Apr 30 2009, 12:28:20) [GCC 4.0.1 (Apple Computer, Inc. build 5367)], maxunicode: 65535 Cygwin CYGWIN_NT-5.1-1.5.25-0.156-4-2-i686-32bit-WindowsPE: python: 2.5.2 (r252:60911, Dec 2 2008, 09:26:14) [GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)], maxunicode: 65535 Windows: python: 2.6.2 (r262:71600, Apr 21 2009, 15:05:37) [MSC v.1500 32 bit (Intel)], maxunicode: 65535 Windows: python: 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)], maxunicode: 65535 [View Less]

7 13

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4
by M.-A. Lemburg Oct. 7, 2009

Oct. 7, 2009

Zooko O'Whielacronx wrote: > Folks: > > I'm sorry, I think I didn't make my concern clear. My users, and lots > of other users, are having a problem with incompatibility between > Python binary extension modules. One way to improve the situation > would be if the Python devs would use their "bully pulpit" -- their > unique position as a source respected by all Linux distributions -- > and say "We recommend that Linux distributions use UCS4 for > compatibility with … [View More]one another". This would not abrogate anyone's > ability to choose their preferred setting nor, as far as I can tell, > would it interfere with the ongoing development of Python. -1 Please note that we did not choose to ship Python as UCS4 binary on Linux - the Linux distributions did. The Python default is UCS2 for a good reason: it's a good trade-off between memory consumption, functionality and performance. As already mentioned, I also don't understand how the changing the Python default on Linux would help your users in any way - if you let distutils compile your extensions, it's automatically going to use the right Unicode setting for you (as well as your users). Unfortunately, this automatic support doesn't help you when shipping e.g. setuptools eggs, but this is a tool problem, not one of Python: setuptools completely ignores the fact that there are two ways to build Python. I'd suggest you ask the tool maintainers to adjust their tools to support the Python Unicode option. > Here are the details: > > I'm the maintainer of several Python packages. I work hard to make it > easy for users, even users who don't know anything about Python, to > use my software. There have been many pain points in this process and > I've spent a lot of time on it for about three years now working on > packaging, including the tools such as setuptools and distutils and > the new "distribute" tool. Python packaging has been improving during > these years -- things are looking up. > > One of the remaining pain points is that I can distribute binaries of > my Python extension modules for Windows or Mac, but if I distribute a > binary Python extension module on Linux, then if the user has a > different UCS2/UCS4 setting then they won't be able to use the > extension module. The current de facto standard for Linux is UCS4 -- > it is used by Debian, Ubuntu, Fedora, RHEL, OpenSuSE, etc. etc.. The > vast majority of Linux users in practice have UCS4, and most binary > Python modules are compiled for UCS4. > > That means that a few folks will get left out. Those folks, from my > experience, are people who built their python executable themselves > without specifying an override for the default, and the smaller Linux > distributions who insist on doing whatever upstream Python devs > recommend instead of doing whatever the other Linux distros are doing. > One of the data points that I reported was a Python interpreter that > was built locally on an Ubuntu server. Since the person building it > didn't know to override the default setting of --enable-unicode, he > ended up with a Python interpreter built for UCS2, even though all the > Python extension modules shipped by Ubuntu were built with UCS4. People building their own Python version will usually also build their own extensions, so I don't really believe that the above scenario is very common. Also note that Python will complain loudly when you try to load a UCS2 extension in a UCS4 build and vice-versa. We've made sure that any extension using the Python Unicode C API has to be built for the same UCS version of Python. This is done by using different names for the C APIs at the C level. > These are not isolated incidents. The following google searches > suggest that a number of people spend time trying to figure out why > Python extension modules fail on their linux systems: > > http://www.google.com/search?q=PyUnicodeUCS4_FromUnicode+undefined+symbol > http://www.google.com/search?q=+PyUnicodeUCS2_FromUnicode+undefined+symbol > http://www.google.com/search?q=_PyUnicodeUCS2_AsDefaultEncodedString+undefi… Perhaps we should add a FAQ entry for these linker errors (which are caused by the mentioned C API changes to prevent mixing UCS version) ?! Here's a quick way to determine you Python Unicode build type: python -c "import sys;print((sys.maxunicode<66000)and'UCS2'or'UCS4')" Perhaps we should include this info as well as an 32/64-bit indicator and the processor type in the Python startup line: # python Python 2.6 (r26:66714, Feb 3 2009, 20:49:49, UCS4, 64-bit, x86_64) [GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2 Type "help", "copyright", "credits" or "license" for more information. This would help users find the right binaries to install as extension. > Another data point is the Mandriva Linux distribution. It is probably > much smaller than Debian, Ubuntu, or RedHat, but it is still one of > the major, well-known distributions. I requested of the Python > maintainer for Mandriva, Michael Scherer, that they switch from UCS2 > to UCS4 in order to reduce compatibility problems like these. His > answer as I understood it was that it is best to follow the > recommendations of the upstream Python devs by using the default > setting instead of choosing a setting for himself. Which is IMHO what all Linux distributions should have done. Distributions should really not be put in charge of upstream coding design decisions. Regards, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 28 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ [View Less]

8 14

Re: [Python-Dev] a new setuptools release?
by P.J. Eby Oct. 7, 2009

Oct. 7, 2009

At 01:14 PM 10/6/2009 -0700, Guido van Rossum wrote: >suggest nobody hold their breath waiting for setuptools 0.7. I've never suggested or implied otherwise. But, if you like Distribute so much, why not just add it directly to the stdlib? ;-) AFAIK, the only reason they've had multiple releases of it is to address the issues with its hijacking of setuptools; in a stdlib version all that stuff could be dropped. Otherwise, it'd already be "mature". There are many wins to be had from … [View More]

10 17

please consider changing --enable-unicode default to ucs4
by Zooko O'Whielacronx Oct. 7, 2009

Oct. 7, 2009

Folks: I accidentally sent this letter just to MAL when I intended it to python-dev. Please read it, as it explains why the issue I'm raising is not just the "we should switch to ucs4 because it is better" issue that was previously settled by GvR. This is a current, practical problem that is preventing people from distributing and using Python packages with binary extension modules on Linux. Regards, Zooko ---------- Forwarded message ---------- From: Zooko O'Whielacronx <zookog(a)… [View More]gmail.com> Date: Sun, Sep 27, 2009 at 11:43 AM Subject: Re: [Python-Dev] please consider changing --enable-unicode default to ucs4 To: "M.-A. Lemburg" <mal(a)egenix.com> Folks: I'm sorry, I think I didn't make my concern clear. My users, and lots of other users, are having a problem with incompatibility between Python binary extension modules. One way to improve the situation would be if the Python devs would use their "bully pulpit" -- their unique position as a source respected by all Linux distributions -- and say "We recommend that Linux distributions use UCS4 for compatibility with one another". This would not abrogate anyone's ability to choose their preferred setting nor, as far as I can tell, would it interfere with the ongoing development of Python. Here are the details: I'm the maintainer of several Python packages. I work hard to make it easy for users, even users who don't know anything about Python, to use my software. There have been many pain points in this process and I've spent a lot of time on it for about three years now working on packaging, including the tools such as setuptools and distutils and the new "distribute" tool. Python packaging has been improving during these years -- things are looking up. One of the remaining pain points is that I can distribute binaries of my Python extension modules for Windows or Mac, but if I distribute a binary Python extension module on Linux, then if the user has a different UCS2/UCS4 setting then they won't be able to use the extension module. The current de facto standard for Linux is UCS4 -- it is used by Debian, Ubuntu, Fedora, RHEL, OpenSuSE, etc. etc.. The vast majority of Linux users in practice have UCS4, and most binary Python modules are compiled for UCS4. That means that a few folks will get left out. Those folks, from my experience, are people who built their python executable themselves without specifying an override for the default, and the smaller Linux distributions who insist on doing whatever upstream Python devs recommend instead of doing whatever the other Linux distros are doing. One of the data points that I reported was a Python interpreter that was built locally on an Ubuntu server. Since the person building it didn't know to override the default setting of --enable-unicode, he ended up with a Python interpreter built for UCS2, even though all the Python extension modules shipped by Ubuntu were built with UCS4. These are not isolated incidents. The following google searches suggest that a number of people spend time trying to figure out why Python extension modules fail on their linux systems: http://www.google.com/search?q=PyUnicodeUCS4_FromUnicode+undefined+symbol http://www.google.com/search?q=+PyUnicodeUCS2_FromUnicode+undefined+symbol http://www.google.com/search?q=_PyUnicodeUCS2_AsDefaultEncodedString+undefi… Another data point is the Mandriva Linux distribution. It is probably much smaller than Debian, Ubuntu, or RedHat, but it is still one of the major, well-known distributions. I requested of the Python maintainer for Mandriva, Michael Scherer, that they switch from UCS2 to UCS4 in order to reduce compatibility problems like these. His answer as I understood it was that it is best to follow the recommendations of the upstream Python devs by using the default setting instead of choosing a setting for himself. (Now we could implement a protocol which would show whether a given Python package was compiled for UCS2 or UCS4. That would be good. Hopefully it would make incompatibility more explicit and understandable to users. Here is a ticket for that -- which project I am contributing to: http://bugs.python.org/setuptools/issue78 . However, even if we implement that feature in the distribute tool (the successor to setuptools), users who build their own python or who use a Linux distribution that follows upstream configuration defaults will still be unable to use most Python packages with compiled extension modules.) In a message on this thread, MvL wrote: > "For that reason I think it's also better that the configure script > continues to default to UTF-16 -- this will give the UTF-16 support > code the necessary exercise." > > This is effectively a BDFL pronouncement. Nothing has changed the > validity of the premise of the statement, so the conclusion remains > valid, as well. My understand of the earlier thread was that someone suggested that UCS4 would be technically better and GvR decided that there were technical reasons to continue actively maintaining the UCS2 code. This thread is different: I'm saying that users are suffering packaging problems and asking for help with that. The way that python-dev can help is to make it so that people who choose "Whatever the upstream devs prefer" (--enable-unicde) on Linux get the de facto standard setting. In the earlier thread, GvR wrote: "I think we should continue to leave this up to the distribution. AFAIK many Linux distros already use UCS4 for everything anyway.". But at least some distributions are asking the upstream Python devs to choose for them, by leaving the setting at the default. Hm, pondering GvR's words: "I think we should continue to leave this up to the distribution", I have a new proposal: make it so that *on Linux only* "--enable-unicode" errors out with an error message saying "please choose either --enable-unicode=ucs2 or --enable-unicode=ucs4. ucs4 is the most widely used setting on Linux. See http://python.org/wiki/UCS2_vs_UCS4 for details.". This would force those Linux distributions who are *not* currently deciding to decide. Thank you for your attention. Regards, Zooko [View Less]

2 1

Package install failures in 2.6.3
by Ned Deily Oct. 7, 2009

Oct. 7, 2009

There has been some discussion on the distutils-sig list over the past few days about a change to distutils released with 2.6.3 which inadvertently causes a number (all?) packages with C extensions that use setuptools/easy_install to fail during builds/installs with a rather cryptic error message. A number of popular packages are affected directly or indirectly through dependencies. The problem is summarized in the issue tracker here; http://bugs.python.org/issue7064 which includes … [View More]

16 32

Python byte-compiled and optimized code
by Swapnil Talekar Oct. 7, 2009

Oct. 7, 2009

I am working on deploying Python on VxWorks platform as a part of project for my company. Accordingly, I would like to know couple of things from python's core-developers. Although I think I already know the answers for most of the questions, we need a confirmation from the community 1) Is the byte-compiled .pyc file and optimized .pyo file platform-independent?(including python versions 3.x) *If yes, is it guaranteed to stay that way in future?* 2) If the the generation of .pyc file fails (say,… [View More]

5 4

Announcing PEP 3136
by Hatem Nassrat Oct. 7, 2009

Oct. 7, 2009

Tue Jul 3 10:14:17 CEST 2007, Guido van Rossum wrote: > On 6/30/07, Matt Chisholm <matt-python at theory.org> wrote: > > I've created and submitted a new PEP proposing support for labels in > > Python's break and continue statements. Georg Brandl has graciously > > added it to the PEP list as PEP 3136: > > > > http://www.python.org/dev/peps/pep-3136/ > > I think this is a good summary of various proposals that have been > floated in the past, … [View More]

7 7

Re: [Python-Dev] Distutils ML wrap-up: setup.cfg new format
by P.J. Eby Oct. 6, 2009

Oct. 6, 2009

At 07:00 PM 9/23/2009 +0200, Tarek Ziadé wrote: >While it's great to have Philipp being part of our distutils design >discussions, >for his experience, I am not concerned of not having him in this "internal >consensus" since Setuptools is not maintained anymore. > >He said some months ago, he would work on a brand new setuptools >version with zero >dependency to distutils. But it's still vaporware (from his own >words), and the previous version is unmaintained for a … [View More]year, so it was >forked. Here's what actually happened, if anyone cares. Tarek and friends announced a fork of setuptools. I reviewed the work and saw that -- for the most part -- I was happy with it, and opined as how I might be willing to bless the the "package inquisition" team as official maintainers of the 0.6 branch of setuptools, so that I could work on the fun bits I've long planned for 0.7, but never felt free to start on while there was so much still needing to be done on 0.6. However, just as I mentioned this, and suggested an option for what I could do that would be helpful to his Distribute 0.7 project as well as various other tools (e.g. implementing some of Jim Fulton's long-requested features for better modularization of setuptools), Tarek accused me of somehow trying to undermine his plans. In addition, it appears Tarek was also offended by my earlier statement that there were only a few people in the Python community who had *already* earned my implicit trust to not only hack on setuptools unsupervised, but also to take over its *future* direction and BDFL-ship. (For example, Jim Fulton and Ian Bicking.) Tarek, however, appears to have taken this to mean that I personally thought he was an incompetent programmer or something (when I actually had no opinion one way or the other), and ever since he has taken to levelling potshots like the above at me on a semi-regular basis. I've tried to ignore this and play nice, because he is actually working on this stuff and I am not. But it's hard for me to actually give any help in practice, if Tarek is too busy projecting hidden plots onto everything I say and do. If you read Tarek's distutils-sig posts, it appears my already-existing trust in Ian and Jim was not only a personal insult to Tarek, but also a plot to ensure that nobody with any time to do so would ever work on setuptools, just as my excitement about working on setuptools again was a plot to steal thunder from his fork. All I want is for good stuff to happen for setuptools users and Python users in general, so I don't think all the suspicion and backbiting is merited. I certainly don't appreciate it, and I would like it to stop. It also isn't even relevant to the thread, since my lack of work on setuptools says exactly zero about the merits or lack thereof of Tarek's proposals for the distutils! Hell, I *support* the bulk of Tarek's setup.cfg proposal, and don't even object to him Pronouncing it or cutting off the discussion! My only issue on Python-Dev was his inaccurate implication that it was a SIG consensus rather than a pronouncement on it. There is and was no need for any of this to get personal, and I have continually strived to keep my posts here and distutils-sig civil, even when I didn't feel like being civil in response to Tarek's jabs. I have in fact bent over backwards to be *nice* to Tarek, because he seemed so damn sensitive about everything. Apparently, however, this does not actually help things. :-( [View Less]

7 8

Old libc's isspace() bug on FreeBSD brings back on Mac OS X?
by INADA Naoki Oct. 6, 2009

Oct. 6, 2009

I found this hg's issue. http://mercurial.selenic.com/bts/msg8375 I think below fix is not enabled on Mac OS X. http://svn.python.org/view/python/trunk/Include/pyport.h?view=diff&pathrev=… I can't confirm it because I am not Mac OS X user. Can anyone confirm it? -- Naoki INADA <songofacandy(a)gmail.com>

3 6

eggs now mandatory for pypi?
by Fredrik Lundh Oct. 6, 2009

Oct. 6, 2009

it's revews like this that makes me wonder if releasing open source is a good idea: no egg - worst seen ever, remove it from pypi or provide an egg (jensens, 2009-10-05, 0 points) </F>

17 31