Python-Dev September 2009

python-dev@python.org

129 participants
82 discussions

by Guido van Rossum

I've received some enthusiastic emails from someone who wants to revive restricted mode. He started out with a bunch of patches to the CPython runtime using ctypes, which he attached to an App Engine bug: http://code.google.com/p/googleappengine/issues/detail?id=671 Based on his code (the file secure.py is all you need, included in secure.tar.gz) it seems he believes the only security leaks are __subclasses__, gi_frame and gi_code. (I have since convinced him that if we add "restricted" guards to these attributes, he doesn't need the functions added to sys.) I don't recall the exploits that Samuele once posted that caused the death of rexec.py -- does anyone recall, or have a pointer to the threads? -- --Guido van Rossum (home page: http://www.python.org/~guido/)

5 years, 10 months

PEP 3145 (With Contents)

by Eric Pruitt

Alright, I will re-submit with the contents pasted. I never use double backquotes as I think them rather ugly; that is the work of an editor or some automated program in the chain. Plus, it also messed up my line formatting and now I have lines with one word on them... Anyway, the contents of PEP 3145: PEP: 3145 Title: Asynchronous I/O For subprocess.Popen Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson Type: Standards Track Content-Type: text/plain Created: 04-Aug-2009 Python-Version: 3.2 Abstract: In its present form, the subprocess.Popen implementation is prone to dead-locking and blocking of the parent Python script while waiting on data from the child process. Motivation: A search for "python asynchronous subprocess" will turn up numerous accounts of people wanting to execute a child process and communicate with it from time to time reading only the data that is available instead of blocking to wait for the program to produce data [1] [2] [3]. The current behavior of the subprocess module is that when a user sends or receives data via the stdin, stderr and stdout file objects, dead locks are common and documented [4] [5]. While communicate can be used to alleviate some of the buffering issues, it will still cause the parent process to block while attempting to read data when none is available to be read from the child process. Rationale: There is a documented need for asynchronous, non-blocking functionality in subprocess.Popen [6] [7] [2] [3]. Inclusion of the code would improve the utility of the Python standard library that can be used on Unix based and Windows builds of Python. Practically every I/O object in Python has a file-like wrapper of some sort. Sockets already act as such and for strings there is StringIO. Popen can be made to act like a file by simply using the methods attached the the subprocess.Popen.stderr, stdout and stdin file-like objects. But when using the read and write methods of those options, you do not have the benefit of asynchronous I/O. In the proposed solution the wrapper wraps the asynchronous methods to mimic a file object. Reference Implementation: I have been maintaining a Google Code repository that contains all of my changes including tests and documentation [9] as well as blog detailing the problems I have come across in the development process [10]. I have been working on implementing non-blocking asynchronous I/O in the subprocess.Popen module as well as a wrapper class for subprocess.Popen that makes it so that an executed process can take the place of a file by duplicating all of the methods and attributes that file objects have. There are two base functions that have been added to the subprocess.Popen class: Popen.send and Popen._recv, each with two separate implementations, one for Windows and one for Unix based systems. The Windows implementation uses ctypes to access the functions needed to control pipes in the kernel 32 DLL in an asynchronous manner. On Unix based systems, the Python interface for file control serves the same purpose. The different implementations of Popen.send and Popen._recv have identical arguments to make code that uses these functions work across multiple platforms. When calling the Popen._recv function, it requires the pipe name be passed as an argument so there exists the Popen.recv function that passes selects stdout as the pipe for Popen._recv by default. Popen.recv_err selects stderr as the pipe by default. "Popen.recv" and "Popen.recv_err" are much easier to read and understand than "Popen._recv('stdout' ..." and "Popen._recv('stderr' ..." respectively. Since the Popen._recv function does not wait on data to be produced before returning a value, it may return empty bytes. Popen.asyncread handles this issue by returning all data read over a given time interval. The ProcessIOWrapper class uses the asyncread and asyncwrite functions to allow a process to act like a file so that there are no blocking issues that can arise from using the stdout and stdin file objects produced from a subprocess.Popen call. References: [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess http://mail.python.org/pipermail/python-bugs-list/2006-December/ 036524.html [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess http://ivory.idyll.org/blog/feb-07/problems-with-subprocess [3] How can I run an external command asynchronously from Python? - Stack Overflow http://stackoverflow.com/questions/636561/how-can-i-run-an-external- command-asynchronously-from-python [4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation http://docs.python.org/library/subprocess.html#subprocess.Popen.wait [5] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation http://docs.python.org/library/subprocess.html#subprocess.Popen.kill [6] Issue 1191964: asynchronous Subprocess - Python tracker http://bugs.python.org/issue1191964 [7] Module to allow Asynchronous subprocess use on Windows and Posix platforms - ActiveState Code http://code.activestate.com/recipes/440554/ [8] subprocess.rst - subprocdev - Project Hosting on Google Code http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=s… [9] subprocdev - Project Hosting on Google Code http://code.google.com/p/subprocdev [10] Python Subprocess Dev http://subdev.blogspot.com/ Copyright: This P.E.P. is licensed under the Open Publication License; http://www.opencontent.org/openpub/. On Tue, Sep 8, 2009 at 22:56, Benjamin Peterson <benjamin(a)python.org> wrote: > 2009/9/7 Eric Pruitt <eric.pruitt(a)gmail.com>: >> Hello all, >> >> I have been working on adding asynchronous I/O to the Python >> subprocess module as part of my Google Summer of Code project. Now >> that I have finished documenting and pruning the code, I present PEP >> 3145 for its inclusion into the Python core code. Any and all feedback >> on the PEP (http://www.python.org/dev/peps/pep-3145/) is appreciated. > > Hi Eric, > One of the reasons you're not getting many response is that you've not > pasted the contents of the PEP in this message. That makes it really > easy for people to comment on various sections. > > BTW, it seems like you were trying to use reST formatting with the > text PEP layout. Double backquotes only mean something in reST. > > > -- > Regards, > Benjamin >

7 years, 6 months

Problems with hex-conversion functions

by Ender Wiggin

Hello everyone. I see several problems with the two hex-conversion function pairs that Python offers: 1. binascii.hexlify and binascii.unhexlify 2. bytes.fromhex and bytes.hex Problem #1: bytes.hex is not implemented, although it was specified in PEP 358. This means there is no symmetrical function to accompany bytes.fromhex. Problem #2: Both pairs perform the same function, although The Zen Of Python suggests that "There should be one-- and preferably only one --obvious way to do it." I do not understand why PEP 358 specified the bytes function pair although it mentioned the binascii pair... Problem #3: bytes.fromhex may receive spaces in the input string, although binascii.unhexlify may not. I see no good reason for these two functions to have different features. Problem #4: binascii.unhexlify may receive both input types: strings or bytes, whereas bytes.fromhex raises an exception when given a bytes parameter. Again there is no reason for these functions to be different. Problem #5: binascii.hexlify returns a bytes type - although ideally, converting to hex should always return string types and converting from hex should always return bytes. IMO there is no meaning of bytes as an output of hexlify, since the output is a representation of other bytes. This is also the suggested behavior of bytes.hex in PEP 358 Problems #4 and #5 call for a decision about the input and output of the functions being discussed: Option A : Strict input and output unhexlify (and bytes.fromhex) may only receives string and may only return bytes hexlify (and bytes.hex) may only receives bytes and may only return strings Option B : Robust input and strict output unhexlify (and bytes.fromhex) may receive bytes and strings and may only return bytes hexlify (and bytes.hex) may receive bytes or strings and may only return strings Of course we may also consider a third option, which will allow the return type of all functions to be robust (perhaps specified in a keyword argument), but as I wrote in the description of problem #5, I see no sense in that. Note that PEP 3137 describes: "... the more strict definitions of encoding and decoding in Python 3000: encoding always takes a Unicode string and returns a bytes sequence, and decoding always takes a bytes sequence and returns a Unicode string." - suggesting option A. To repeat problems #4 and #5, the current behavior does not match any option: * The return type of binascii.hexlify should be string, and this is not the current behavior. As for the input: * Option A is not the current behavior because binascii.unhexlify may receive both input types. * Option B is not the current behavior because bytes.fromhex does not allow bytes as input. To fix these issues, three changes should be applied: 1. Deprecate bytes.fromhex. This fixes the following problems: #4 (go with option B and remove the function that does not allow bytes input) #2 (the binascii functions will be the only way to "do it") #1 (bytes.hex should not be implemented) 2. In order to keep the functionality that bytes.fromhex has over unhexlify, the latter function should be able to handle spaces in its input (fix #3) 3. binascii.hexlify should return string as its return type (fix #5)

9 years, 9 months

PEP 382: Namespace Packages

by "Martin v. Löwis"

I propose the following PEP for inclusion to Python 3.1. Please comment. Regards, Martin Abstract ======== Namespace packages are a mechanism for splitting a single Python package across multiple directories on disk. In current Python versions, an algorithm to compute the packages __path__ must be formulated. With the enhancement proposed here, the import machinery itself will construct the list of directories that make up the package. Terminology =========== Within this PEP, the term package refers to Python packages as defined by Python's import statement. The term distribution refers to separately installable sets of Python modules as stored in the Python package index, and installed by distutils or setuptools. The term vendor package refers to groups of files installed by an operating system's packaging mechanism (e.g. Debian or Redhat packages install on Linux systems). The term portion refers to a set of files in a single directory (possibly stored in a zip file) that contribute to a namespace package. Namespace packages today ======================== Python currently provides the pkgutil.extend_path to denote a package as a namespace package. The recommended way of using it is to put:: from pkgutil import extend_path __path__ = extend_path(__path__, __name__) int the package's ``__init__.py``. Every distribution needs to provide the same contents in its ``__init__.py``, so that extend_path is invoked independent of which portion of the package gets imported first. As a consequence, the package's ``__init__.py`` cannot practically define any names as it depends on the order of the package fragments on sys.path which portion is imported first. As a special feature, extend_path reads files named ``*.pkg`` which allow to declare additional portions. setuptools provides a similar function pkg_resources.declare_namespace that is used in the form:: import pkg_resources pkg_resources.declare_namespace(__name__) In the portion's __init__.py, no assignment to __path__ is necessary, as declare_namespace modifies the package __path__ through sys.modules. As a special feature, declare_namespace also supports zip files, and registers the package name internally so that future additions to sys.path by setuptools can properly add additional portions to each package. setuptools allows declaring namespace packages in a distribution's setup.py, so that distribution developers don't need to put the magic __path__ modification into __init__.py themselves. Rationale ========= The current imperative approach to namespace packages has lead to multiple slightly-incompatible mechanisms for providing namespace packages. For example, pkgutil supports ``*.pkg`` files; setuptools doesn't. Likewise, setuptools supports inspecting zip files, and supports adding portions to its _namespace_packages variable, whereas pkgutil doesn't. In addition, the current approach causes problems for system vendors. Vendor packages typically must not provide overlapping files, and an attempt to install a vendor package that has a file already on disk will fail or cause unpredictable behavior. As vendors might chose to package distributions such that they will end up all in a single directory for the namespace package, all portions would contribute conflicting __init__.py files. Specification ============= Rather than using an imperative mechanism for importing packages, a declarative approach is proposed here, as an extension to the existing ``*.pkg`` mechanism. The import statement is extended so that it directly considers ``*.pkg`` files during import; a directory is considered a package if it either contains a file named __init__.py, or a file whose name ends with ".pkg". In addition, the format of the ``*.pkg`` file is extended: a line with the single character ``*`` indicates that the entire sys.path will be searched for portions of the namespace package at the time the namespace packages is imported. Importing a package will immediately compute the package's __path__; the ``*.pkg`` files are not considered anymore after the initial import. If a ``*.pkg`` package contains an asterisk, this asterisk is prepended to the package's __path__ to indicate that the package is a namespace package (and that thus further extensions to sys.path might also want to extend __path__). At most one such asterisk gets prepended to the path. extend_path will be extended to recognize namespace packages according to this PEP, and avoid adding directories twice to __path__. No other change to the importing mechanism is made; searching modules (including __init__.py) will continue to stop at the first module encountered. Discussion ========== With the addition of ``*.pkg`` files to the import mechanism, namespace packages can stop filling out the namespace package's __init__.py. As a consequence, extend_path and declare_namespace become obsolete. It is recommended that distributions put a file <distribution>.pkg into their namespace packages, with a single asterisk. This allows vendor packages to install multiple portions of namespace package into a single directory, with no risk of overlapping files. Namespace packages can start providing non-trivial __init__.py implementations; to do so, it is recommended that a single distribution provides a portion with just the namespace package's __init__.py (and potentially other modules that belong to the namespace package proper). The mechanism is mostly compatible with the existing namespace mechanisms. extend_path will be adjusted to this specification; any other mechanism might cause portions to get added twice to __path__. Copyright ========= This document has been placed in the public domain.

10 years, 8 months

transitioning from % to {} formatting

by Steven Bethard

There's a lot of code already out there (in the standard library and other places) that uses %-style formatting, when in Python 3.0 we should be encouraging {}-style formatting. We should really provide some sort of transition plan. Consider an example from the logging docs: logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s") We'd like to support both this style as well as the following style: logging.Formatter("{asctime} - {name} - {levelname} - {message}") Perhaps we'd eventually deprecate the %-style formatting, but at least in the intervening time, we'd have to support both. I see a few possibilities: * Add a new class, NewFormatter, which uses the {}-style formatting. This is ugly because it makes the formatting we're trying to encourage look like the alternative implementation instead of the standard one. It also means we have to come up with new names for every API that uses format strings. * Have Formatter try to guess whether it got %-style formatting or {}-style formatting, and then delegate to the appropriate one. I don't know how accurately we can guess. We also end up still relying on both formatting styles under the hood. * Have Formatter convert all %-style formatting strings to {}-style formatting strings (automatically). This still involves some guessing, and involves some serious hacking to translate from one to the other (maybe it wouldn't even always be possible?). But at least we'd only be using {}-style formatting under the hood. Thoughts? Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

10 years, 8 months

PEP 389: argparse - new command line parsing module

by Steven Bethard

On Sun, Sep 27, 2009 at 8:41 PM, Benjamin Peterson <benjamin(a)python.org> wrote: > 2009/9/27 Steven Bethard <steven.bethard(a)gmail.com>: >> The first release where any real deprecation message would show up is >> Python 3.4, more than 3 years away. If you think 3 years isn't long >> enough for people to be over the Python 3 transition, let's stick in >> another version in there and make it 4.5 years. > > So, why even bother deprecating it if nobody is going to see the warnings? I feel like I'm repeating the PEP, but here it is again anyway. There will be messages in the docs and pending deprecation warnings (which don't show up by default but can be requested) starting in Python 2.7 and 3.2. Regular deprecation warnings wouldn't show up until Python 3.4, 3 years away. This compromise was intended exactly to address the issue you brought up about people getting over the Python 3 transition. Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

10 years, 9 months

please consider changing --enable-unicode default to ucs4

by Zooko O'Whielacronx

Dear Pythonistas: This issue causes serious problems. Users occasionally get binaries built for a compatible Linux and Python version but with a different UCS2-vs-UCS4 setting, and those users get mysterious memory corruption errors which are hard to diagnose. It is possible that these situations also open up security vulnerabilities. A couple such instances are documented on http://bugs.python.org/setuptools/issue78, but you can find more by googling. I would like to get this problem fixed! In order to help address this issue I sampled what UCS size is used by python executables in the wild. I instrumented a few buildslaves that are contributed by various people to the Tahoe-LAFS project to print out their platform, python version, and sys.maxunicode. The full results are appended below. maxunicode: 1114111 means that python executable was configured with --enable-unicode=ucs4, and maxunicode: 65535 means that python executable was configured with --enable-unicode=ucs2 or just with --enable-unicode . The only incompatibilities that I found are because some packagers have deliberately set UCS4 configuration and other packagers have left the default setting. In the three cases where someone configured python with UCS2, one of the three is certainly an accident (a custom-built python executable on an Ubuntu server) and the other two just use the default instead of specifically configuring ucs2 in their configurations of Python and I suspect that they don't know the difference and that it was an accident that they built a Python incompatible with other distributions of their operating system. In sum, while it would be good to add the unicode setting to the platform's ABI (as discussed in setuptools ticket #78), it would also be good to make the default value be UCS4 instead of UCS2. This would fix all three of the potential incompatibilities that I found (listed below), and once we have proper inclusion of the unicode setting in the ABI in order to prevent the memory corruption, defaulting to UCS4 would increase the likelihood that a binary built on one distribution would be usable on another. I'm sure that someone can come up with a reason why UCS2 is better than UCS4, but I'm also sure that the benefits of compatibility outweigh any benefits of UCS2 encoding, and that the widespread use of UCS4 demonstrates that there is nothing fatally wrong with it, and that people who really value UCS2 encoding more than compatibility can choose that for themselves by explicitly setting UCS2. Let me restate that I am not suggesting taking away anyone's options, only making the setting for people who don't specify default to the compatible option. Hm, I guess that means that it should default to UCS2 on Windows and Mac and to UCS4 on Linux and Solaris. Regards, Zooko Ubuntu 6.10 "edgy" i386: python: 2.4.4c1 (#2, Mar 7 2008, 03:03:38) [GCC 4.1.2 20060928 (prerelease) (Ubuntu 4.1.1-13ubuntu5)], maxunicode: 1114111 Ubuntu 7.04 "feisty": python: 2.5.1 (r251:54863, Jul 31 2008, 22:53:39) [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)], maxunicode: 1114111 Ubuntu 7.10 "gutsy" i386: python: 2.5.1 (r251:54863, Jul 31 2008, 23:17:40) [GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)], maxunicode: 1114111 Ubuntu 8.04 "hardy" amd64: python: 2.5.2 (r252:60911, Jul 22 2009, 15:33:10) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)], maxunicode: 1114111 Ubuntu 8.04 "hardy" i386: *custom* python: 2.6 (r26:66714, Oct 2 2008, 13:40:28) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)], maxunicode: 65535 Ubuntu 8.04 "hardy" i386: python: 2.5.2 (r252:60911, Jul 22 2009, 15:35:03) [GCC 4.2.4 (Ubuntu 4.2.4-1ubuntu3)], maxunicode: 1114111 Ubuntu 9.04 "jaunty" amd64: *custom* python: 2.6.2 (release26-maint, Apr 19 2009, 01:58:18) [GCC 4.3.3], maxunicode: 1114111 Debian 4.0 "etch" i386: python: 2.4.4 (#2, Oct 22 2008, 19:52:44) [GCC 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)], maxunicode: 1114111 Debian 5.0 "lenny" i386: python: 2.5.2 (r252:60911, Jan 4 2009, 17:40:26) [GCC 4.3.2], maxunicode: 1114111 Debian 5.0 "lenny" amd64: python: 2.5.2 (r252:60911, Jan 4 2009, 21:59:32) [GCC 4.3.2], maxunicode: 1114111 Debian 5.0 "lenny" armv5tel: python: 2.5.2 (r252:60911, Jan 5 2009, 02:00:00) [GCC 4.3.2], maxunicode: 1114111 Debian unstable "squeeze/sid" i386: python: 2.5.4 (r254:67916, Feb 17 2009, 20:16:45) [GCC 4.3.3], maxunicode: 1114111 Fedora 11 "leonidas" amd64: python: 2.6 (r26:66714, Jul 4 2009, 17:37:13) [GCC 4.4.0 20090506 (Red Hat 4.4.0-4)], maxunicode: 1114111 ArchLinux: python: 2.6.2 (r262:71600, Jul 20 2009, 02:23:30) [GCC 4.4.0 20090630 (prerelease)], maxunicode: 65535 NetBSD 4: python: 2.5.2 (r252:60911, Mar 20 2009, 14:00:07) [GCC 4.1.2 20060628 prerelease (NetBSD nb2 20060711)], maxunicode: 65535 OpenSolaris SunOS-5.11-i86pc-i386-32bit: python: 2.4.4 (#1, Mar 10 2009, 09:35:36) [C], maxunicode: 65535 Nexenta NCP1 SunOS-5.11-i86pc-i386-32bit: python: 2.4.3 (#2, May 3 2006, 19:12:42) [GCC 4.0.3 (GNU_OpenSolaris 4.0.3-1nexenta4)], maxunicode: 1114111 Mac OS 10.6 "snow leopard" i386: python: 2.6.1 (r261:67515, Jul 7 2009, 23:51:51) [GCC 4.2.1 (Apple Inc. build 5646)], maxunicode: 65535 Mac OS 10.5 "leopard" i386: python: 2.5.1 (r251:54863, Feb 6 2009, 19:02:12) [GCC 4.0.1 (Apple Inc. build 5465)], maxunicode: 65535 Mac OS 10.4 "tiger" *custom* python: 2.5.4 (release25-maint:72153M, Apr 30 2009, 12:28:20) [GCC 4.0.1 (Apple Computer, Inc. build 5367)], maxunicode: 65535 Cygwin CYGWIN_NT-5.1-1.5.25-0.156-4-2-i686-32bit-WindowsPE: python: 2.5.2 (r252:60911, Dec 2 2008, 09:26:14) [GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)], maxunicode: 65535 Windows: python: 2.6.2 (r262:71600, Apr 21 2009, 15:05:37) [MSC v.1500 32 bit (Intel)], maxunicode: 65535 Windows: python: 2.5.1 (r251:54863, Apr 18 2007, 08:51:08) [MSC v.1310 32 bit (Intel)], maxunicode: 65535

10 years, 9 months

Re: [Python-Dev] please consider changing --enable-unicode default to ucs4

by M.-A. Lemburg

Zooko O'Whielacronx wrote: > Folks: > > I'm sorry, I think I didn't make my concern clear. My users, and lots > of other users, are having a problem with incompatibility between > Python binary extension modules. One way to improve the situation > would be if the Python devs would use their "bully pulpit" -- their > unique position as a source respected by all Linux distributions -- > and say "We recommend that Linux distributions use UCS4 for > compatibility with one another". This would not abrogate anyone's > ability to choose their preferred setting nor, as far as I can tell, > would it interfere with the ongoing development of Python. -1 Please note that we did not choose to ship Python as UCS4 binary on Linux - the Linux distributions did. The Python default is UCS2 for a good reason: it's a good trade-off between memory consumption, functionality and performance. As already mentioned, I also don't understand how the changing the Python default on Linux would help your users in any way - if you let distutils compile your extensions, it's automatically going to use the right Unicode setting for you (as well as your users). Unfortunately, this automatic support doesn't help you when shipping e.g. setuptools eggs, but this is a tool problem, not one of Python: setuptools completely ignores the fact that there are two ways to build Python. I'd suggest you ask the tool maintainers to adjust their tools to support the Python Unicode option. > Here are the details: > > I'm the maintainer of several Python packages. I work hard to make it > easy for users, even users who don't know anything about Python, to > use my software. There have been many pain points in this process and > I've spent a lot of time on it for about three years now working on > packaging, including the tools such as setuptools and distutils and > the new "distribute" tool. Python packaging has been improving during > these years -- things are looking up. > > One of the remaining pain points is that I can distribute binaries of > my Python extension modules for Windows or Mac, but if I distribute a > binary Python extension module on Linux, then if the user has a > different UCS2/UCS4 setting then they won't be able to use the > extension module. The current de facto standard for Linux is UCS4 -- > it is used by Debian, Ubuntu, Fedora, RHEL, OpenSuSE, etc. etc.. The > vast majority of Linux users in practice have UCS4, and most binary > Python modules are compiled for UCS4. > > That means that a few folks will get left out. Those folks, from my > experience, are people who built their python executable themselves > without specifying an override for the default, and the smaller Linux > distributions who insist on doing whatever upstream Python devs > recommend instead of doing whatever the other Linux distros are doing. > One of the data points that I reported was a Python interpreter that > was built locally on an Ubuntu server. Since the person building it > didn't know to override the default setting of --enable-unicode, he > ended up with a Python interpreter built for UCS2, even though all the > Python extension modules shipped by Ubuntu were built with UCS4. People building their own Python version will usually also build their own extensions, so I don't really believe that the above scenario is very common. Also note that Python will complain loudly when you try to load a UCS2 extension in a UCS4 build and vice-versa. We've made sure that any extension using the Python Unicode C API has to be built for the same UCS version of Python. This is done by using different names for the C APIs at the C level. > These are not isolated incidents. The following google searches > suggest that a number of people spend time trying to figure out why > Python extension modules fail on their linux systems: > > http://www.google.com/search?q=PyUnicodeUCS4_FromUnicode+undefined+symbol > http://www.google.com/search?q=+PyUnicodeUCS2_FromUnicode+undefined+symbol > http://www.google.com/search?q=_PyUnicodeUCS2_AsDefaultEncodedString+undefi… Perhaps we should add a FAQ entry for these linker errors (which are caused by the mentioned C API changes to prevent mixing UCS version) ?! Here's a quick way to determine you Python Unicode build type: python -c "import sys;print((sys.maxunicode<66000)and'UCS2'or'UCS4')" Perhaps we should include this info as well as an 32/64-bit indicator and the processor type in the Python startup line: # python Python 2.6 (r26:66714, Feb 3 2009, 20:49:49, UCS4, 64-bit, x86_64) [GCC 4.3.2 [gcc-4_3-branch revision 141291]] on linux2 Type "help", "copyright", "credits" or "license" for more information. This would help users find the right binaries to install as extension. > Another data point is the Mandriva Linux distribution. It is probably > much smaller than Debian, Ubuntu, or RedHat, but it is still one of > the major, well-known distributions. I requested of the Python > maintainer for Mandriva, Michael Scherer, that they switch from UCS2 > to UCS4 in order to reduce compatibility problems like these. His > answer as I understood it was that it is best to follow the > recommendations of the upstream Python devs by using the default > setting instead of choosing a setting for himself. Which is IMHO what all Linux distributions should have done. Distributions should really not be put in charge of upstream coding design decisions. Regards, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Sep 28 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

10 years, 9 months

Announcing PEP 3136

by Hatem Nassrat

Tue Jul 3 10:14:17 CEST 2007, Guido van Rossum wrote: > On 6/30/07, Matt Chisholm <matt-python at theory.org> wrote: > > I've created and submitted a new PEP proposing support for labels in > > Python's break and continue statements. Georg Brandl has graciously > > added it to the PEP list as PEP 3136: > > > > http://www.python.org/dev/peps/pep-3136/ > > I think this is a good summary of various proposals that have been > floated in the past, plus some new ones. As a PEP, it falls short > because it doesn't pick a solution but merely offers a large menu of > possible options. Also, there is nothing about implementation yet. > > However, I'm rejecting it on the basis that code so complicated to > require this feature is very rare. In most cases there are existing > work-arounds that produce clean code, for example using 'return'. I agree that this feature will only serve as a quick hack and in many cases it would be misused and ugly code will be the result. However, it seems that there are some shortcuts that have sneaked into python (I am a fairly new Python programmer, only since late 2.4, so don't shoot me). The specific one of which I speak about is: while_stmt ::= "while" expression ":" suite ["else" ":" suite] for_stmt ::= "for" target_list "in" expression_list ":" suite ["else" ":" suite] try1_stmt ::= "try" ":" suite ("except" [expression ["as" target]] ":" suite)+ ["else" ":" suite] ["finally" ":" suite] All these else's seem like shortcuts to me. I did find a use for them, once I found out they existed, but I get butterflies whenever I do. -- Hatem Nassrat

10 years, 9 months

Re: [Python-Dev] Distutils ML wrap-up: setup.cfg new format

by P.J. Eby

At 07:00 PM 9/23/2009 +0200, Tarek Ziadé wrote: >While it's great to have Philipp being part of our distutils design >discussions, >for his experience, I am not concerned of not having him in this "internal >consensus" since Setuptools is not maintained anymore. > >He said some months ago, he would work on a brand new setuptools >version with zero >dependency to distutils. But it's still vaporware (from his own >words), and the previous version is unmaintained for a year, so it was >forked. Here's what actually happened, if anyone cares. Tarek and friends announced a fork of setuptools. I reviewed the work and saw that -- for the most part -- I was happy with it, and opined as how I might be willing to bless the the "package inquisition" team as official maintainers of the 0.6 branch of setuptools, so that I could work on the fun bits I've long planned for 0.7, but never felt free to start on while there was so much still needing to be done on 0.6. However, just as I mentioned this, and suggested an option for what I could do that would be helpful to his Distribute 0.7 project as well as various other tools (e.g. implementing some of Jim Fulton's long-requested features for better modularization of setuptools), Tarek accused me of somehow trying to undermine his plans. In addition, it appears Tarek was also offended by my earlier statement that there were only a few people in the Python community who had *already* earned my implicit trust to not only hack on setuptools unsupervised, but also to take over its *future* direction and BDFL-ship. (For example, Jim Fulton and Ian Bicking.) Tarek, however, appears to have taken this to mean that I personally thought he was an incompetent programmer or something (when I actually had no opinion one way or the other), and ever since he has taken to levelling potshots like the above at me on a semi-regular basis. I've tried to ignore this and play nice, because he is actually working on this stuff and I am not. But it's hard for me to actually give any help in practice, if Tarek is too busy projecting hidden plots onto everything I say and do. If you read Tarek's distutils-sig posts, it appears my already-existing trust in Ian and Jim was not only a personal insult to Tarek, but also a plot to ensure that nobody with any time to do so would ever work on setuptools, just as my excitement about working on setuptools again was a plot to steal thunder from his fork. All I want is for good stuff to happen for setuptools users and Python users in general, so I don't think all the suspicion and backbiting is merited. I certainly don't appreciate it, and I would like it to stop. It also isn't even relevant to the thread, since my lack of work on setuptools says exactly zero about the merits or lack thereof of Tarek's proposals for the distutils! Hell, I *support* the bulk of Tarek's setup.cfg proposal, and don't even object to him Pronouncing it or cutting off the discussion! My only issue on Python-Dev was his inaccurate implication that it was a SIG consensus rather than a pronouncement on it. There is and was no need for any of this to get personal, and I have continually strived to keep my posts here and distutils-sig civil, even when I didn't feel like being civil in response to Tarek's jabs. I have in fact bent over backwards to be *nice* to Tarek, because he seemed so damn sensitive about everything. Apparently, however, this does not actually help things. :-(

10 years, 9 months

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

1999

Python-Dev September 2009