Mailman 3 September 2009 - Python-Dev

Re: [Python-Dev] Distutils ML wrap-up: setup.cfg new format
by P.J. Eby 07 Oct '09

07 Oct '09

At 07:00 PM 9/23/2009 +0200, Tarek Ziadé wrote: >While it's great to have Philipp being part of our distutils design >discussions, >for his experience, I am not concerned of not having him in this "internal >consensus" since Setuptools is not maintained anymore. > >He said some months ago, he would work on a brand new setuptools >version with zero >dependency to distutils. But it's still vaporware (from his own >words), and the previous version is unmaintained for a year, so it was >forked. Here's what actually happened, if anyone cares. Tarek and friends announced a fork of setuptools. I reviewed the work and saw that -- for the most part -- I was happy with it, and opined as how I might be willing to bless the the "package inquisition" team as official maintainers of the 0.6 branch of setuptools, so that I could work on the fun bits I've long planned for 0.7, but never felt free to start on while there was so much still needing to be done on 0.6. However, just as I mentioned this, and suggested an option for what I could do that would be helpful to his Distribute 0.7 project as well as various other tools (e.g. implementing some of Jim Fulton's long-requested features for better modularization of setuptools), Tarek accused me of somehow trying to undermine his plans. In addition, it appears Tarek was also offended by my earlier statement that there were only a few people in the Python community who had *already* earned my implicit trust to not only hack on setuptools unsupervised, but also to take over its *future* direction and BDFL-ship. (For example, Jim Fulton and Ian Bicking.) Tarek, however, appears to have taken this to mean that I personally thought he was an incompetent programmer or something (when I actually had no opinion one way or the other), and ever since he has taken to levelling potshots like the above at me on a semi-regular basis. I've tried to ignore this and play nice, because he is actually working on this stuff and I am not. But it's hard for me to actually give any help in practice, if Tarek is too busy projecting hidden plots onto everything I say and do. If you read Tarek's distutils-sig posts, it appears my already-existing trust in Ian and Jim was not only a personal insult to Tarek, but also a plot to ensure that nobody with any time to do so would ever work on setuptools, just as my excitement about working on setuptools again was a plot to steal thunder from his fork. All I want is for good stuff to happen for setuptools users and Python users in general, so I don't think all the suspicion and backbiting is merited. I certainly don't appreciate it, and I would like it to stop. It also isn't even relevant to the thread, since my lack of work on setuptools says exactly zero about the merits or lack thereof of Tarek's proposals for the distutils! Hell, I *support* the bulk of Tarek's setup.cfg proposal, and don't even object to him Pronouncing it or cutting off the discussion! My only issue on Python-Dev was his inaccurate implication that it was a SIG consensus rather than a pronouncement on it. There is and was no need for any of this to get personal, and I have continually strived to keep my posts here and distutils-sig civil, even when I didn't feel like being civil in response to Tarek's jabs. I have in fact bent over backwards to be *nice* to Tarek, because he seemed so damn sensitive about everything. Apparently, however, this does not actually help things. :-(

7 8

Python 2.6.3rc1 available
by Barry Warsaw 04 Oct '09

04 Oct '09

The first (and hopefully last) release candidate for Python 2.6.3 is now available via http://www.python.org/download/releases/2.6.3/ Source releases and Windows binaries are currently available, and Mac OS X binaries should be forthcoming. Nearly 100 bugs have been fixed since 2.6.2. Barring any unforeseen problems, we will make the final 2.6.3 release this Friday, October 2nd. Please give this release candidate a spin and let us know if you encounter any show stopping problems. Enjoy, -Barry

4 3

PEP 3144 review.
by Peter Moody 03 Oct '09

03 Oct '09

Folks, Guido, I believe PEP 3144 is ready for your review. When you get a chance, can you take a look/make a pronouncement? Cheers, /peter

35 266

Python 2.6.3
by Barry Warsaw 02 Oct '09

02 Oct '09

I had previously wanted to release Python 2.6.3 over the summer, but for various personal reasons, the summer was just too insane. I'd like to reschedule a 2.6.3 release, shooting for final release on 25- September. We should probably do a release candidate, so I'd like to make that on 23-September. Does anybody have objections to that schedule? If not, I'll try to spend some time over the next few days looking at outstanding bugs, and marking release blockers, etc. -Barry

14 35

Python logging and 1.5.2 compatibility
by Vinay Sajip 01 Oct '09

01 Oct '09

I'm planning to "officially" drop support for Python 1.5.2 in the logging package. When the logging package was introduced in Python 2.3, many Linux distros were shipping 1.5.2 as the system's Python, even though 2.2 had been out for a while. So it seemed important to support 1.5.2 for those sysadmins who wanted to use logging with their system Python. The Linux landscape has changed a bit since then. Most Linux distros ship with much more recent versions of Python, and so I no longer see 1.5.2 support as important. Dropping support for 1.5.2 means that future changes to logging will not be concerned with 1.5.2 compatibility. For example, boolean values which were 0/1 in the logging package will at some point be replaced by False/True, and newer language features will start to be used when changes are made. There are no plans for a specific "cleanup" exercise at the moment. In fact some changes made a while ago inadvertently broke 1.5.2 compatibility, but no-one's complained. So I'm assuming the whole thing is really a non-issue, and this post is just to keep everyone in the picture. A 1.5.2-compatible version of the package is still available via http://www.red-dove.com/python_logging.html if anyone needs it. This version is not actively maintained, but that shouldn't be an issue. Regards, Vinay Sajip

4 4

operator precedence of __eq__, __ne__, etc, if both object have implementations
by Chris Withers 30 Sep '09

30 Sep '09

Hi All, I didn't see any docs on this: http://docs.python.org/reference/datamodel.html?highlight=__eq__#object.__e… Where are the specifications on what happens if two objects are compared and both have implementations of __eq__? Which __eq__ is called? What happens if the first one called returns False? Is the second one called? What is one implements __eq__ and the other __ne__? If I've missed something, please point me in the right direction. To all those about to tell me to go read the source: that's not good enough here. I'm hoping there *are* "official" rules for how these interact and they just need better linking in, otherwise, I worry that IronPython could do one thing, Jython another and CPython a third... cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

16 37

Distutils ML wrap-up: setup.cfg new format
by Tarek Ziadé 30 Sep '09

30 Sep '09

Hello Here's a wrapup of the Distutils-SIG discussion we had on the "static metadata" topic. I realize that it's a good thing to send in. python-dev such wrapup on distutils design decisions, to keep everyone informed and get more feedback when required. I will try to do it for every upcoming changes that are not going in a PEP process (when it's not a 'big' change). The rate of such mails should not be very high. (around 1 mail in python-dev for +150 mails in distutils-SIG) If you feel that what we are about to change in distutils is wrong, you can go ahead and help us by participating in Distutils-ML, so we keep one and only one media for these discussions. The four sentences summary for people in a hurry: Getting metadata of a distribution that is not. installed means running its setup.py. This means. downloading the whole archive, and running. third party code on your system. To avoid it, we are adding a [setup] section in. setup.cfg where you can express the package metadata statically. Conditional sections, specific to some system. can be added to add some specific fields in [setup]. At the end, you will be able to get metadata fields without running any third-party code, and possibly get only the distribution setup.cfg for this. Now the long version. Rational ======== Today, if you want to list all the metadata (PEP 314) of a. distribution that is not installed, you need to use it's. setup.py CLI. So, basically, you download it, and run:: $ python setup.py --name Foo $ python setup.py --version 1.2 Where `name` and `version` are metadata fields. That's OK but as. soon as the developers add more code in setup.py, this feature might break or (worse) might do unwanted things on the target. system. Why should we run third-party code just to get its metadata ? So we worked on a way to express those metadata statically, by pushing them in `setup.cfg`. That's not hard, since all. the metadata fields are static values. Adding a setup section in setup.cfg =================================== So the first thing we want to introduce is a [setup] section, that may contain any field from the metadata:: [setup] name: Foo version: 1.2 Distutils will look for them, and will use them. Of course setup.py is still required, and in case an option that is. a metadata field is passed to setup(), it will override one. located in setup.cfg. PEP 341 is coming up ==================== Remember the last Pycon Summit ? We said that we would introduce a new metadata field to describe requirements.. That's basically what PEP 341 is about, and we are still. working on it. Basically, we will be able to write things like:: requires: Twisted == 8.2.0 What takes us so long is that adding requirements like this in the metadata requires more things: - the ability to compare versions (this was described in. PEP 386 but not finished yet) - the ability to make these requirements vary depending on. the target system And the later makes our "setup.cfg" new [setup] section. obsolete as soon as this new metadata field is added in. Python. So we need more that what I've shown in the previous section Context-dependant sections ========================== The second change we are going to introduce is context-dependant sections in setup.cfg. A context dependant section is a section with a condition that is. used only if the execution environment meets its condition. Here's an example:: [setup] name: Foo version: 1.3 [setup:sys_platform == 'win32'] requires: pywin32 requires: bar > 1.0 [setup:os_machine == '64bits'] requires: some_package [setup:python_version == '2.4' or python_version == '2.5'] requires: some_package Every [setup:condition] section will be added in [setup] only if the condition is met. The micro-language behind this is the simplest possible: it compares only strings, with usual string operators, and with the ability to combine expressions. It makes it also easy to understand to non-pythoneers (os packagers). The pseudo-grammar is (I don't know how to write those but you'll get it hopefully):: comp: '<'|'>'|'=='|'>='|'<='|'<>'|'!='|'in'|'not' 'in' comparison: expr (comp_op expr)* expr: STRING test: or_test or_test: and_test ('or' and_test)* and_test: not_test ('and' not_test)* not_test: 'not' not_test | comparison where STRING belongs to any of those: - python_version = '%s.%s' % (sys.version_info[0], sys.version_info[1]) - os_name = os.name - sys_platform = sys.platform - os_sysname = os.uname()[0]. - os_nodename = os.uname()[1] - os_release = os.uname()[2]. - os_version = os.uname()[3].. - os_machine = os.uname()[4]. - a free string, like '2.4', or 'win32' Distutils will provide a function that is able to read the metadata of a distribution, given a setup.cfg file, for the target environment:: >>> from distutils.util import local_metadata >>> local_metadata('setup.cfg') <DistributionMetadata instance> Meaning that a Vanilla Python will be able to read the metadata of a package without running any code. That's what distutils will use when setup.py is run. That's what any package manager will be able to use to work. with distributions. Limitations =========== If a distribution is unable for obscure reasons (or does not wish) to set all metadata fields in setup.cfg, that's fine :. the fields will be set to UNKNOWN when `local_metadata` is called. When setup.py is run, options passed to setup() will complete those. When a package manager gets 'UNKOWN' values, it knows it might need to do further processing by running setup.py. Why this change is good ===================== Once the requires metadata is added in PEP 341, being able to query the metadata for a distribution can be done without doing anything else than reading a static file and interpreting its conditions in a restricted and secured manner. So there's no need to run third-party code anymore. Possible use cases: - setup.cfg is published on PyPI. Packages managers like easy_install or pip will be able to list all the distributions needed and their versions for installing a distribution without having to download anything ! - OS packagers will be able to list the dependencies of a distribution without having to browse Python code. And, again, this new [setup] section is backward-compatible, and will be a good complement to the work done in PEP 376. Thanks for reading so far. This code will be added in 2.7 and 3.2. Tarek -- Tarek Ziadé | http://ziade.org | オープンソースはすごい!

15 40

test_thread tracebacks
by skip＠pobox.com 30 Sep '09

30 Sep '09

It's been awhile since I rebuilt Python and ran the test suite. This evening I noticed this on my Mac (OS X 10.5): test_thread Unhandled exception in thread started by <bound method ThreadRunningTests.task of <test.test_thread.ThreadRunningTests testMethod=test_stack_size>> Traceback (most recent call last): File "/Users/skip/src/python/trunk/Lib/test/test_thread.py", line 51, in task self.done_mutex.release() thread.error: release unlocked lock Unhandled exception in thread started by <bound method ThreadRunningTests.task of <test.test_thread.ThreadRunningTests testMethod=test_stack_size>> Traceback (most recent call last): File "/Users/skip/src/python/trunk/Lib/test/test_thread.py", line 51, in task self.done_mutex.release() thread.error: release unlocked lock Unhandled exception in thread started by <bound method ThreadRunningTests.task of <test.test_thread.ThreadRunningTests testMethod=test_stack_size>> Traceback (most recent call last): File "/Users/skip/src/python/trunk/Lib/test/test_thread.py", line 51, in task self.done_mutex.release() thread.error: release unlocked lock Unhandled exception in thread started by <bound method ThreadRunningTests.task of <test.test_thread.ThreadRunningTests testMethod=test_starting_threads>> Traceback (most recent call last): File "/Users/skip/src/python/trunk/Lib/test/test_thread.py", line 51, in task self.done_mutex.release() thread.error: release unlocked lock Oddly enough, this didn't cause the test to fail. Is this a known problem? Should I open a ticket? Skip

3 4

Re: [Python-Dev] transitioning from % to {} formatting
by Steven Bethard 30 Sep '09

30 Sep '09

On Tue, Sep 29, 2009 at 5:54 PM, Benjamin Peterson <benjamin(a)python.org> wrote: > * Provide a flag to Formatter which controls whether new or old > formatting is used. Emit a warning when it's not true. So then the transition strategy is something like: version N: Add formatting flag which uses {}-style formatting on True and %-style formatting on False, which defaults to False version N + 1: Deprecate False value for formatting flag (all code should start specifying flag=True) version N + 2: Raise error on False value for formatting flag (all code must specify flag=True) version N + 3: Deprecate formatting flag version N + 4: Remove formatting flag ? Steve -- Where did you get that preposterous hypothesis? Did Steve tell you that? --- The Hiphopopotamus

1 0

PEP 389: argparse - new command line parsing module
by Steven Bethard 30 Sep '09

30 Sep '09

Below is a new PEP based on discussions from the stdlib-sig, which proposes to add the argparse module to the standard library in Python 2.7 and 3.2. Looking forward to your feedback! Steve http://www.python.org/dev/peps/pep-0389/ ---------------------------------------------------------------------- PEP: 389 Title: argparse - new command line parsing module Version: $Revision: 75097 $ Last-Modified: $Date: 2009-09-27 12:42:40 -0700 (Sun, 27 Sep 2009) $ Author: Steven Bethard <steven.bethard(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 25-Sep-2009 Python-Version: 2.7 and 3.2 Post-History: Abstract ======== This PEP proposes inclusion of the argparse [1]_ module in the Python standard library in Python 2.7 and 3.2. Motivation ========== The argparse module is a command line parsing library which provides more functionality than the existing command line parsing modules in the standard library, getopt [2]_ and optparse [3]_. It includes support for positional arguments (not just options), subcommands, required options, options syntaxes like "/f" and "+rgb", zero-or-more and one-or-more style arguments, and many other features the other two lack. The argparse module is also already a popular third-party replacement for these modules. It is used in projects like IPython (the Scipy Python shell) [4]_, is included in Debian testing and unstable [5]_, and since 2007 has had various requests for its inclusion in the standard library [6]_ [7]_ [8]_. This popularity suggests it may be a valuable addition to the Python libraries. Why aren't getopt and optparse enough? ====================================== One argument against adding argparse is that thare are "already two different option parsing modules in the standard library" [9]_. The following is a list of features provided by argparse but not present in getopt or optparse: * While it is true there are two *option* parsing libraries, there are no full command line parsing libraries -- both getopt and optparse support only options and have no support for positional arguments. The argparse module handles both, and as a result, is able to generate better help messages, avoiding redundancies like the ``usage=`` string usually required by optparse. * The argparse module values practicality over purity. Thus, argparse allows required options and customization of which characters are used to identify options, while optparse explicitly states "the phrase 'required option' is self-contradictory" and that the option syntaxes ``-pf``, ``-file``, ``+f``, ``+rgb``, ``/f`` and ``/file`` "are not supported by optparse, and they never will be". * The argparse module allows options to accept a variable number of arguments using ``nargs='?'``, ``nargs='*'`` or ``nargs='+'``. The optparse module provides an untested recipe for some part of this functionality [10]_ but admits that "things get hairy when you want an option to take a variable number of arguments." * The argparse module supports subcommands, where a main command line parser dispatches to other command line parsers depending on the command line arguments. This is a common pattern in command line interfaces, e.g. ``svn co`` and ``svn up``. Why isn't the functionality just being added to optparse? ========================================================= Clearly all the above features offer improvements over what is available through optparse. A reasonable question then is why these features are not simply provided as patches to optparse, instead of introducing an entirely new module. In fact, the original development of argparse intended to do just that, but because of various fairly constraining design decisions of optparse, this wasn't really possible. Some of the problems included: * The optparse module exposes the internals of its parsing algorithm. In particular, ``parser.largs`` and ``parser.rargs`` are guaranteed to be available to callbacks [11]_. This makes it extremely difficult to improve the parsing algorithm as was necessary in argparse for proper handling of positional arguments and variable length arguments. For example, ``nargs='+'`` in argparse is matched using regular expressions and thus has no notion of things like ``parser.largs``. * The optparse extension APIs are extremely complex. For example, just to use a simple custom string-to-object conversion function, you have to subclass ``Option``, hack class attributes, and then specify your custom option type to the parser, like this:: class MyOption(Option): TYPES = Option.TYPES + ("mytype",) TYPE_CHECKER = copy(Option.TYPE_CHECKER) TYPE_CHECKER["mytype"] = check_mytype parser = optparse.OptionParser(option_class=MyOption) parser.add_option("-m", type="mytype") For comparison, argparse simply allows conversion functions to be used as ``type=`` arguments directly, e.g.:: parser = argparse.ArgumentParser() parser.add_option("-m", type=check_mytype) But given the baroque customization APIs of optparse, it is unclear how such a feature should interact with those APIs, and it is quite possible that introducing the simple argparse API would break existing custom Option code. * Both optparse and argparse parse command line arguments and assign them as attributes to an object returned by ``parse_args``. However, the optparse module guarantees that the ``take_action`` method of custom actions will always be passed a ``values`` object which provides an ``ensure_value`` method [12]_, while the argparse module allows attributes to be assigned to any object, e.g.:: foo_object = ... parser.parse_args(namespace=foo_object) foo_object.some_attribute_parsed_from_command_line Modifying optparse to allow any object to be passed in would be difficult because simply passing the ``foo_object`` around instead of a ``Values`` instance will break existing custom actions that depend on the ``ensure_value`` method. Because of issues like these, which made it unreasonably difficult for argparse to stay compatible with the optparse APIs, argparse was developed as an independent module. Given these issues, merging all the argparse features into optparse with no backwards incompatibilities seems unlikely. Deprecation of getopt and optparse ================================== There is still some debate over the best way (if at all) to encourage users to move from getopt and optparse to their replacement, argparse. The current recommendation of this PEP is the following conservative deprecation strategy: * Python 3.2, Python 2.7 and any later Python 2.X releases -- PendingDeprecation warnings, which by default are not displayed, and documentation notes directing users of getopt and optparse to argparse. * Python 3.3 -- Same as above * Python 3.4 -- Deprecation warnings for getopt and optparse, which by default *are* displayed. Though this is slower than the usual deprecation process, it seems prudent to avoid producing any casually visible Deprecation warnings until Python 3.X has had some additional time to attract developers. Open Issues =========== The argparse module supports Python from 2.3 up through 3.2 and as a result relies on traditional ``%(foo)s`` style string formatting. It has been suggested that it might be better to use the new style ``{foo}`` string formatting [13]_. This seems like a good idea, but would break backwards compatibility for existing argparse-based scripts unless we can come up with a way to reasonably support both syntaxes. References ========== .. [1] argparse (http://code.google.com/p/argparse/) .. [2] getopt (http://docs.python.org/library/getopt.html) .. [3] optparse (http://docs.python.org/library/optparse.html) .. [4] argparse in IPython (http://mail.scipy.org/pipermail/ipython-dev/2009-April/005102.html) .. [5] argparse in Debian (http://packages.debian.org/search?keywords=python-argparse) .. [6] 2007-01-03 request for argparse in the standard library (http://mail.python.org/pipermail/python-list/2007-January/592646.html) .. [7] 2009-06-09 request for argparse in the standard library (http://bugs.python.org/issue6247) .. [8] 2009-09-10 request for argparse in the standard library (http://mail.python.org/pipermail/stdlib-sig/2009-September/000342.html) .. [9] Fredrik Lundh response to [6]_ (http://mail.python.org/pipermail/python-list/2007-January/592675.html) .. [10] optparse variable args (http://docs.python.org/library/optparse.html#callback-example-6-variable-ar…) .. [11] parser.largs and parser.rargs (http://docs.python.org/library/optparse.html#how-callbacks-are-called) .. [12] take_action values argument (http://docs.python.org/library/optparse.html#adding-new-actions) .. [13] use {}-formatting instead of %-formatting (http://bugs.python.org/msg89279) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

20 32