proto-pep: plugin proposal (for unittest)

Hello all, My apologies in advance if email mangles whitespace in the code examples. I can reformulate as a PEP if that is deemed useful and this document can be found online at: http://hg.python.org/unittest2/file/tip/description.txt (Please excuse errors and omissions - but do feel free to point them out.) This is a description, and request for feedback, of the unittest plugin system that I am currently prototyping in the plugins branch of unittest2_. My goal is to merge the plugin system back into unittest itself in Python 3.2. .. _unittest2: http://hg.python.org/unittest2 As part of the prototype I have been implementing some example plugins (in unittest2/plugins/) so I can develop the mechanism against real rather than imagined use cases. Jason Pellerin, creator of nose, has been providing me with feedback and has been trying it out by porting some of the nose plugins to unittest [#]_. He comments on the system "it looks very flexible and clean". ;-) Example plugins available and included: * a pep8 and pyflakes checker * a debugger plugin that drops you into pdb on test fail / error * a doctest loader (looks for doctests in all text files in the project) * use a regex for matching files in test discovery instead of a glob * growl notifications on test run start and stop * filter individual *test methods* using a regex * load test functions from modules as well as TestCases * integration with the coverage module for coverage reporting In addition I intend to create a plugin that outputs junit compatible xml from a test run (for integration with tools like the hudson continuous integration server) and a test runner that runs tests in parallel using multiprocessing. Not all of these will be included in the merge to unittest. Which ones will is still an open question. I'd like feedback on the proposal, and hopefully approval to port it into unittest after discussion / amendment / completion. In particular I'd like feedback on the basic system, plus which events should be available and what information should be available in them. Note that the system is *not* complete in the prototype. Enough is implemented to get "the general idea" and to formalise the full system. It still needs extensive tests and the extra work in TestProgram makes it abundantly clear that refactoring there is well overdue... In the details below open questions and todos are noted. I *really* value feedback (but will ignore bikeshedding ;-) .. note:: Throughout this document I refer to the prototype implementation using names like ``unittest2.events.hooks``. Should this proposal be accepted then the names will live in the unittest package instead of unittest2. The core classes for the event system live in the current implementation in the ``unittest2.events`` namespace. Abstract ======== unittest lacks a standard way of extending it to provide commonly requested functionality, other than subclassing and overriding (and reimplementing) parts of its behaviour. This document describes a plugin system already partially prototyped in unittest2. Aspects of the plugin system include: * an events mechanism where handlers can be registered and called during a test run * a Plugin class built over the top of this for easy creation of plugins * a configuration system for specifying which plugins should be loaded and for configuring individual plugins * command line integration * the specific set of events and the information / actions available to them As the plugin system essentially just adds event calls to key places it has few backwards compatibility issues. Unfortunately existing extensions that override the parts of unittest that call these events will not be compatible with plugins that use them. Framework authors who re-implement parts of unittest, for example custom test runners, may want to add calling these events in appropriate places. Rationale ========= Why a plugin system for unittest? unittest is the standard library test framework for Python but in recent years has been eclipsed in functionality by frameworks like nose and py.test. Among the reasons for this is that these frameworks are easier to extend with plugins than unittest. unittest makes itself particularly difficult to extend by using subclassing as its basic extension mechanism. You subclass and override behaviour in its core classes like the loader, runner and result classes. This means that where you have more than one "extension" working in the same area it is very hard for them to work together. Whilst various extensions to unittest do exist (e.g. testtools, zope.testrunner etc) they don't tend to work well together. In contrast the plugin system makes creating extensions to unittest much simpler and less likely that extensions will clash with each other. nose itself exists as a large system built over the top of unittest. Extending unittest in this way was very painful for the creators of nose, and every release of Python breaks nose in some way due to changes in unittest. One of the goals of the extension mechanism is to allow nose2 to be a much thinner set of plugins over unittest(2) that is much simpler to maintain [#]_. The early indications are that the proposed system is a good fit for this goal. Low Level Mechanism ==================== The basic mechanism is having events fired at various points during a test run. Plugins can register event handler functions that will be called with an event object. Multiple functions may be registered to handle an event and event handlers can also be removed. Over the top of this is a ``Plugin`` class that simplifies building plugins on top of this mechanism. This is described in a separate section. The events live on the ``unittest2.events.hooks`` class. Handlers are added using ``+=`` and removed using ``-=``, a syntax borrowed from the .NET system. For example adding a handler for the ``startTestRun`` event:: from unittest2.events import hooks def startTestRun(event): print 'test run started at %s' % event.startTime hooks.startTestRun += startTestRun Handlers are called with an Event object specific to the event. Each event provides different information on its event objects as attributes. For example the attributes available on ``StartTestRunEvent`` objects are: * ``suite`` - the test suite for the full test run * ``runner`` - the test runner * ``result`` - the test result * ``startTime`` The name of events, whether any should be added or removed, and what information is available on the event objects are all valid topics for discussion. Specific events and the information available to them is covered in a section below. An example plugin using events directly is the ``doctestloader`` plugin. Framework authors who re-implement parts of unittest, for example custom test runners, may want to add calling these events in appropriate places. This is very simple. For example the ``pluginsLoaded`` event is fired with a ``PluginsLoadedEvent`` object that is instantiated without parameters:: from unittest2.events import hooks, PluginsLoadedEvent hooks.pluginsLoaded(PluginsLoadedEvent()) Why use event objects and not function parameters? -------------------------------------------------- There are several reasons to use event objects instead of function parameters. The *disadvantage* of this is that the information available to an event is not obvious from the signature of a handler. There are several compelling advantages however: * the signature of all handler functions is identical and therefore easy to remember * backwards compatibility - new attributes can be added to event objects (and parameters deprecated) without breaking existing plugins. Changing the way a function is called (unless all handlers have a ``**kw`` signature) is much harder. * several of the events have a lot of information available. This would make the signature of handlers huge. With an event object handlers only need to be aware of attributes they are interested in and ignore information they aren't interested in ("only pay for what you eat"). * some of the attributes are mutable - the event object is shared between all handlers, this would be less obvious if function parameters were used * calling multiple handlers and still returning a value (see the handled pattern below) The handled pattern -------------------- Several events can be used to *override* the default behaviour. For example the 'matchregexp' plugin uses the ``matchPath`` event to replace the default way of matching files for loading as tests during test discovery. The handler signals that it is handling this event, and the default implementation should not be run, by setting ``event.handled = True``:: def matchRegexp(event): pattern = event.pattern name = event.name event.handled = True path = event.path if matchFullPath: return re.match(pattern, path) return re.match(pattern, name) Where the default implementation returns a value, for example creating a test suite, or in the case of ``matchPath`` deciding if a path matches a file that should be loaded as a test, the handler can return a result. If an event sets handled on an event then no more handlers will be called for that event. Which events can be handled, and which not, is discussed in the events section. The Plugin Class ================ A sometimes-more-convenient way of creating plugins is to subclass the ``unittest2.events.Plugin`` class. By default subclassing ``Plugin`` will auto-instantiate the plugin and store the instance in a list of loaded plugins. Each plugin has a ``register()`` method that auto-hooks up all methods whose names correspond to events. Plugin classes may also provide ``configSection`` and ``commandLineSwitch`` class attributes which simplifies enabling the plugin through the command line and making available a section from the configuration file(s). A simple plugin using this is the 'debugger' plugin that starts ``pdb`` when the ``onTestFail`` event fires:: from unittest2.events import Plugin import pdb import sys class Debugger(Plugin): configSection = 'debugger' commandLineSwitch = ('D', 'debugger', 'Enter pdb on test fail or error') def __init__(self): self.errorsOnly = self.config.as_bool('errors-only', default=False) def onTestFail(self, event): value, tb = event.exc_info[1:] test = event.test if self.errorsOnly and isinstance(value, test.failureException): return original = sys.stdout sys.stdout = sys.__stdout__ try: pdb.post_mortem(tb) finally: sys.stdout = original A plugin that doesn't want to be auto-instantiated (for example a base class used for several plugins) can set ``autoCreate = False`` as a class attribute. (This attribute is only looked for on the class directly and so isn't inherited by subclasses.) If a plugin is auto-instantiated then the instance will be set as the ``instance`` attribute on the plugin class. ``configSection`` and ``commandLineSwitch`` are described in the `configuration system`_ and `command line integration`_ sections. Plugin instances also have an ``unregister`` method that unhooks all events. It doesn't exactly correspond to the ``register`` method (it undoes some of the work done when a plugin is instantiated) and so can only be called once. Plugins to be loaded are specified in configuration files. For frameworks not using the unittest test runner and configuration system APIs for loading plugins are available in the form of the ``loadPlugins`` function (which uses the configuration system to load plugins) and ``loadPlugin`` which loads an individual plugin by module name. Loading plugins just means importing the module containing the plugin. Configuration system ==================== By default the unittest2 test runner (triggered by the unit2 script or for unittest ``python -m unittest``) loads two configuration files to determine which plugins to load. A user configuration file, ~/unittest.cfg (alternative name and location would be possible), can specify plugins that will always be loaded. A per-project configuration file, unittest.cfg which should be located in the current directory when unit2 is launched, can specify plugins for individual projects. To support this system several command line options have been added to the test runner:: --config=CONFIGLOCATIONS Specify local config file location --no-user-config Don't use user config file --no-plugins Disable all plugins Several config files can be specified using ``--config``. If the user config is being loaded then it will be loaded first (if it exists), followed by the project config (if it exists) *or* any config files specified by ``--config``. ``--config`` can point to specific files, or to a directory containing a ``unittest.cfg``. Config files loaded later are merged into already loaded ones. Where a *key* is in both the later key overrides the earlier one. Where a section is in both but with different keys they are merged. (The exception to keys overriding is the 'plugins' key in the unittest section - these are combined to create a full list of plugins. Perhaps multiline values in config files could also be merged?) plugins to be loaded are specified in the ``plugins`` key of the ``unittest`` section:: [unittest] plugins = unittest2.plugins.checker unittest2.plugins.doctestloader unittest2.plugins.matchregexp unittest2.plugins.moduleloading unittest2.plugins.debugger unittest2.plugins.testcoverage unittest2.plugins.growl unittest2.plugins.filtertests The plugins are simply module names. They either hook themselves up manually on import or are created by virtue of subclassing ``Plugin``. A list of all loaded plugins is available as ``unittest2.events.loadedPlugins`` (a list of strings). For accessing config values there is a ``getConfig(sectionName=None)`` function. By default it returns the whole config data-structure but it an also return individual sections by name. If the section doesn't exist an empty section will be returned. The config data-structure is not read-only but there is no mechanism for persisting changes. The config is a dictionary of ``Section`` objects, where a section is a dictionary subclass with some convenience methods for accessing values:: section = getConfig(sectionName) integer = section.as_int('foo', default=3) number = section.as_float('bar', default=0.0) # as_list returns a list with empty lines and comment lines removed items = section.as_list('items', default=[]) # as_bool allows 'true', '1', 'on', 'yes' for True (matched case-insensitively) and # 'false', 'off', '0', 'no', '' for False value = section.as_bool('value', default=True) If a plugin specifies a ``configSection`` as a class attribute then that section will be fetched and set as the ``config`` attribute on instances. By convention plugins should use the 'always-on' key in their config section to specify that the plugin should be switched on by default. If 'always-on' exists and is set to 'True' then the ``register()`` method will be called on the plugin to hook up all events. If you don't want a plugin to be auto-registered you should fetch the config section yourself rather than using ``configSection``. If the plugin is configured to be 'always-on', and is auto-registered, then it doesn't need a command line switch to turn it on (although it may add other command line switches or options) and ``commandLineSwitch`` will be ignored. Command Line Interface ====================== Plugins may add command line options, either switches with a callback function or options that take values and will be added to a list. There are two functions that do this: ``unittest2.events.addOption`` and ``unittest2.events.addDiscoveryOption``. Some of the events are only applicable to test discovery (``matchPath`` is the only one currently I think), options that use these events should use ``addDiscoveryOption`` which will only be used if test discovery is invoked. Both functions have the same signature:: addDiscoveryOption(callback, opt=None, longOpt=None, help=None) addOption(plugin.method, 'X', '--extreme', 'Run tests in extreme mode') * ``callback`` is a callback function (taking no arguments) to be invoked if the option is on *or* a list indicating that this is an option that takes arguments, values passed in at the command line will be added to the list * ``opt`` is a short option for the command (or None) not including the leading '-' * ``longopt`` a long option for the command (or None) not including the leading '--' * ``help`` is optional help text for the option, to be displayed by ``unit2 -h`` Lowercase short options are reserved for use by unittest2 internally. Plugins may only add uppercase short options. If a plugin needs a simple command line switch (on/off) then it can set the ``commandLineSwitch`` class attribute to a tuple of ``(opt, longOpt, help)``. The ``register()`` method will be used as the callback function, automatically hooking the plugin up to events if it is switched on. The Events ========== This section details the events implemented so far, the order they are called in, what attributes are available on the event objects, whether the event is 'handleable' (and what that means for the event), plus the intended use case for the event. Events in rough order are: * ``pluginsLoaded`` * ``handleFile`` * ``matchPath`` * ``loadTestsFromNames`` * ``loadTestsFromName`` * ``loadTestsFromModule`` * ``loadTestsFromTestCase`` * ``getTestCaseNames`` * ``runnerCreated`` * ``startTestRun`` * ``startTest`` * ``onTestFail`` * ``stopTest`` * ``stopTestRun`` pluginsLoaded ------------- This event is useful for plugin initialisation. It is fired after all plugins have been loaded, the config file has been read and command line options processed. The ``PluginsLoadedEvent`` has one attribute: ``loadedPlugins`` which is a list of strings referring to all plugin modules that have been loaded. handleFile ---------- This event is fired when a file is looked at in test discovery or a *filename* is passed at the command line. It can be used for loading tests from non-Python files, like doctests from text files, or adding tests for a file like pep8 and pyflakes checks. A ``HandleFileEvent`` object has the following attributes: * ``extraTests`` - a list, extend this with tests to *add* tests that will be loaded from this file without preventing the default test loading * ``name`` - the name of the file * ``path`` - the full path of the file being looked at * ``loader`` - the ``TestLoader`` in use * ``pattern`` - the pattern being used to match files, or None if not called during test discovery * ``top_level_directory`` - the top level directory of the project tests are being loaded from, or the current working directory if not called during test discovery This event *can* be handled. If it is handled then the handler should return a test suite or None. Returning None means no tests will be loaded from this file. If any plugin has created any ``extraTests`` then these will be used even if a handler handles the event and returns None. If this event is not handled then it will be matched against the pattern (test discovery only) and either be rejected or go through for standard test loading. matchPath --------- ``matchPath`` is called to determine if a file should be loaded as a test module. This event only fires during test discovery. ``matchPath`` is only fired if the filename can be converted to a valid python module name, this is because tests are loaded by importing. If you want to load tests from files whose paths don't translate to valid python identifiers then you should use ``handleFile`` instead. A ``MatchPathEvent`` has the following attributes: * ``path`` - full path to the file * ``name`` - filename only * ``pattern`` - pattern being used for discovery This event *can* be handled. If it is handled then the handler should return True or False to indicate whether or not test loading should be attempted from this file. If this event is not handled then the pattern supplied to test discovery will be used as a glob pattern to match the filename. loadTestsFromNames ------------------ This event is fired when ``TestLoader.loadTestsFromNames`` is called. Attributes on the ``LoadFromNamesEvent`` object are: * ``loader`` - the test loader * ``names`` - a list of the names being loaded * ``module`` - the module passed to ``loader.loadTestsFromNames(...)`` * ``extraTests`` - a list of extra tests to be added to the suites loaded from the names This event can be handled. If it is handled then the handler should return a list of suites or None. Returning None means no tests will be loaded from these names. If any plugin has created any ``extraTests`` then these will be used even if a handler handles the event and returns None. If this event is not handled then ``loader.loadTestFromName`` will be called for each name to build up the list of suites. loadTestsFromName ----------------- This event is fired when ``TestLoader.loadTestsFromName`` is called. Attributes on the ``LoadFromNameEvent`` object are: * ``loader`` - the test loader * ``name`` - the name being loaded * ``module`` - the module passed to ``loader.loadTestsFromName(...)`` * ``extraTests`` - a suite of extra tests to be added to the suite loaded from the name This event can be handled. If it is handled then the handler should return a TestSuite or None. Returning None means no tests will be loaded from this name. If any plugin has created any ``extraTests`` then these will be used even if a handler handles the event and returns None. If the event is not handled then each name will be resolved and tests loaded from it, which may mean calling ``loader.loadTestsFromModule`` or ``loader.loadTestsFromTestCase``. loadTestsFromModule ------------------- This event is fired when ``TestLoader.loadTestsFromModule`` is called. It can be used to customise the loading of tests from a module, for example loading tests from functions as well as from TestCase classes. Attributes on the ``LoadFromModuleEvent`` object are: * ``loader`` - the test loader * ``module`` - the module object tests * ``extraTests`` - a suite of extra tests to be added to the suite loaded from the module This event can be handled. If it is handled then the handler should return a TestSuite or None. Returning None means no tests will be loaded from this module. If any plugin has created any ``extraTests`` then these will be used even if a handler handles the event and returns None. If the event is not handled then ``loader.loadTestsFromTestCase`` will be called for every TestCase in the module. Event if the event is handled, if the module defines a ``load_tests`` function then it *will* be called for the module. This removes the responsibility for implementing the ``load_tests`` protocol from plugin authors. loadTestsFromTestCase --------------------- This event is fired when ``TestLoader.loadTestsFromTestCase`` is called. It could be used to customise the loading of tests from a TestCase, for example loading tests with an alternative prefix or created generative / parameterized tests. Attributes on the ``LoadFromTestCaseEvent`` object are: * ``loader`` - the test loader * ``testCase`` - the test case class being loaded * ``extraTests`` - a suite of extra tests to be added to the suite loaded from the TestCase This event can be handled. If it is handled then the handler should return a TestSuite or None. Returning None means no tests will be loaded from this module. If any plugin has created any ``extraTests`` then these will be used even if a handler handles the event and returns None If the event is not handled then ``loader.getTestCaseNames`` will be called to get method names from the test case and a suite will be created by instantiating the TestCase class with each name it returns. getTestCaseNames ---------------- This event is fired when ``TestLoader.getTestCaseNames`` is called. It could be used to customise the method names used to load tests from a TestCase, for example loading tests with an alternative prefix from the default or filtering for specific names. Attributes on the ``GetTestCaseNamesEvent`` object are: * ``loader`` - the test loader * ``testCase`` - the test case class that tests are being loaded from * ``testMethodPrefix`` - set to None, modify this attribute to *change* the prefix being used for this class * ``extraNames`` - a list of extra names to use for this test case as well as the default ones * ``excludedNames`` - a list of names to exclude from loading from this class This event can be handled. If it is handled it should return a list of strings. Note that if this event returns an empty list (or None which will be replaced with an empty list then ``loadTestsFromTestCase`` will still check to see if the TestCase has a ``runTest`` method. Even if the event is handled ``extraNames`` will still be added to the list, however *excludedNames`` won't be removed as they are filtered out by the default implementation which looks for all attributes that are methods (or callable) whose name begins with ``loader.testMethodPrefix`` (or ``event.testMethodPrefix`` if that is set) and aren't in the list of excluded names (converted to a set first for efficient lookup). The list of names will also be sorted using ``loader.sortTestMethodsUsing``. runnerCreated ------------- This event is fired when the ``TextTestRunner`` is instantiated. It can be used to customize the test runner, for example replace the stream and result class, without needing to write a custom test harness. This should allow the default test runner script (``unit2`` or ``python -m untitest``) to be suitable for a greater range of projects. Projects that want to use custom test reporting should be able to do it through a plugin rather than having to rebuild the runner and result machinery, which also requires writing custom test collection too. The ``RunnerCreatedEvent`` object only has one attribute; ``runner`` which is the runner instance. startTestRun ------------ This event is fired when the test run is started. This is used, for example, by the growl notifier that displays a growl notification when a test run begins. It can also be used for filtering tests after they have all been loaded or for taking over the test run machinery altogether, for distributed testing for example. The ``StartTestRunEvent`` object has the following attributes: * ``test`` - the full suite of all tests to be run (may be modified in place) * ``result`` - the result object * ``startTime`` - the time the test run started Currently this event can be handled. This prevents the normal test run from executing, allowing an alternative implementation, but the return value is unused. Handling this event (as with handling any event) prevents other plugins from executing. This means that the it wouldn't be possible to safely combine a distributed test runner with a plugin that filters the suite. Fixing this issue is one of the open issues with the plugin system. startTest --------- This event is fired immediately before a test is executed (inside ``TestCase.run(...)``). The ``StartTestEvent`` object has the following attributes: * ``test`` - the test to be run * ``result`` - the result object * ``startTime`` - the time the test starts execution This event cannot be handled. onTestFail ---------- This event is fired when a test setUp, a test, a tearDown or a cleanUp fails or errors. It is currently used by the debugger plugin. It is *not* currently called for 'internal' unittest exceptions like ``SkipTest`` or expected failures and unexpected successes. Attributes on the ``TestFailEvent`` are: * ``test`` - the test * ``result`` - the result * ``exc_info`` - the result of ``sys.exc_info()`` after the error / fail * ``when`` - one of 'setUp', 'call', 'tearDown', or 'cleanUp' This event cannot be handled. Should this event be able to suppress raised exceptions? It should also be able to modify the traceback so that bare asserts could be used but still provide useful diagnostic information. Should this event be fired for test skips? stopTest -------- This event is fired when a test execution is completed. It includes a great deal of information about the test and could be used to completely replace test reporting, making the test result potentially obsolete. It will be used by the junit-xml plugin to generate the xml reports describing the test run. If there are errors during a tearDown or clean up functions then this event may be fired several times for a test. For each call the ``stage`` will be different, although there could be several errors during clean up functions. Attributes on the ``StopTestEvent`` are: * ``test`` - the test * ``result`` - the result * ``exc_info`` - the result of ``sys.exc_info()`` after an error / fail or None for success * ``stopTime``- time the test stopped, including tear down and clean up functions * ``timeTaken`` - total time for test execution from setUp to clean up functions * ``stage`` - one of setUp, call, tearDown, cleanUp, or None for success * ``outcome`` - one of passed, failed, error, skipped, unexpectedSuccess, expectedFailure The outcomes all correspond to an attribute that will be set to True or False depending on outcome: * ``passed`` * ``failed`` * ``error`` * ``skipped`` * ``unexpectedSuccess`` * ``expectedFailure`` In addition there is a ``skipReason`` that will be None unless the test was skipped, in which case it will be a string containing the reason. This event cannot be handled. stopTestRun ----------- This event is fired when the test run completes. It is useful for reporting tools. The ``StopTestRunEvent`` event objects have the following attributes: * ``runner`` - the test runner * ``result`` - the test result * ``stopTime`` - the time the test run completes * ``timeTaken`` - total time taken by the test run

Damn, the email was truncated. Probably my fault. The part missed off is: Not Yet Implemented =================== Except where noted, everything in this document is already working in the prototype. There are a few open issues and things still to be implemented. Certain event attributes should be read only (like extraTests, extraNames etc) because they should only be extended or modified in place instead of being replaced. Add an epilogue to the optparse help messages. A messaging API for plugins that respects verbosity. The ``startTestRun`` event needs an alternative way of replacing the default test run machinery without stopping other plugins from executing. It should only be possible for *one* plugin to use this. Should the ``onTestFailEvent`` be able to supress raised exceptions? It should also be able to modify the traceback so that bare asserts could be used but still provide useful diagnostic information. Should this event be fired for test skips? Command line options to unittest, like verbosity, should be put into the config data structure so that they are accessible to plugins. TestFailEvent needs to be fired on failures in setUpClass and setUpModule etc Plugin.register and Plugin.unregister aren't precisely complementary. This means that unregister can only be called once. This may not be an issue. Plugin.unregister will fail if any events have already been unhooked. This can easily be fixed. If the merge to unittest in Python 3.2 happens we need to decide which of the example plugins will be included in unittest. (They will probably all remain in unittest2.) Custom test outcomes are *not* supported; making interoperating tools or plugins in the presence of custom outcomes can be very hard. Should unittest2 have a different config file from unittest, so they can be configured side-by-side? (``unittest2.cfg`` perhaps.) Alternatively the same config file could be used with a '[unittest2]' section instead of '[unittest]'. Should ``StopTestEvent.timeTaken`` (plus startTime / stopTime etc) include the time for setUp, tearDown and cleanUps? Should this information be available separately perhaps? The discovery command line interface has changed a lot and the tests need reviewing to ensure they still provide full coverage. Should there be an "excluded-plugins" section in the config files, so projects can prevent incompatible plugins from being loaded? Should multiline values in config files be merged instead of overriding each other. (This is effectively what happens for the plugins list.) Should ``handleFile`` be fired when a test name is specified at the command line? (This would be 'tricky' as ``handleFile`` will have to be called for the containing module and then the correct name pulled out of the module.) Additional Changes ================== Alongside, and in support of, the plugin system a few changes have been made to unittest2. These either have no, or very minor, backwards compatibility issues. Changes so far are: TestLoader has a new attribute ``DEFAULT_PATTERN``. This is so that the regex matching plugin can change the default pattern used for test discovery when no pattern is explicitly provided. Command line parsing is all done by optparse, removing the use of getopt. This makes the help messages more consistent but makes the usage messages less useful in some situations. This can be fixed with the use of the optparse epilogue. unit2 (the default test runner) runs test discovery if invoked without any arguments. unit2 can execute tests in filenames as well as module names - so long as the module pointed to by the filename is importable from the current directory. FunctionTestCase.id() returns 'module.funcname' instead of just funcname. Added util.formatTraceback, the default way of formatting tracebacks. TestCase has a new formatTraceback method (delegating to util.formatTraceback). TestCase instances can implement formatTraceback to control how the traceback for errors and failures are represented. Useful for test items that don't represent Python tests, for example the pep8 / pyflakes checker and theoretical javascript test runners such as exist for py.test and nose. If you specify test names (modules, classes etc) at the command line they will be loaded individually using ``loader.loadTestsFromName`` instead of collectively with ``loader.loadTestsFromNames``. This enables individual names to be checked to see if they refer to filenames. References ========== .. [#] See http://bitbucket.org/jpellerin/unittest2/src/tip/unittest2/plugins/attrib.py and http://bitbucket.org/jpellerin/unittest2/src/tip/unittest2/plugins/errormast... .. [#] http://lists.idyll.org/pipermail/testing-in-python/2010-March/002799.html On 29/07/2010 23:55, Michael Foord wrote:
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Fri, Jul 30, 2010 at 12:55 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote: ...
What about using ABCs here instead ? The big advantage is that you don't force people to subclass your Plugin base class, and you get a registration mechanism for free. They would be able to explicitly register any class using : from unittest2.events import Plugin Plugin.register(MySuperPlugin) Regards Tarek -- Tarek Ziadé | http://ziade.org

On 30/07/2010 11:09, Tarek Ziadé wrote:
Is that an advantage? Subclassing Plugin provides functionality rather than particularly an interface.
and you get a registration mechanism for free.
Actually it looks like it would be *extra* code. (The registration mechanism is already "free" to the programmer when they subclass.) I will investigate this, but if the only benefit it provides is "you don't have to subclass" then I'm not sure there is any point. Note that the Plugin class is only "sugar" over hooking up to the events directly *anyway*, so there is no *need* to use Plugin to write extensions. I'm going to read your blog entry on the topic to evaluate it properly though: https://tarekziade.wordpress.com/2009/05/01/basic-plugin-system-using-abcs-a... Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Jul 30, 2010, at 11:38 AM, Michael Foord wrote:
Very interesting. For Mailman 3, I have YAPS (yet another plugin system) based on zope.interface and setuptools. Bazaar has its own plugin system which is different still. I bet there are as many plugin APIs as there are Python frameworks. (And isn't everything a framework these days? :) I'm glad to see discussion begin to focus on providing consolidation in the world of plugins for Python frameworks, and to begin to refactor basic functionality into common tools. I'd love to see a blessed plugin API promoted to the stdlib for Python 3.2. I think it has to address a number of issues: * Registration - How do third party plugins declare themselves to exist, and be enabled? Part of this seems to me to include interface declarations too. Is installation of the plugin enough to register it? How do end users enable and disable plugins that me be registered on their system? How do plugins describe themselves (provide short and log descriptions, declare options, hook into command line interfaces, etc.)? * Installation - How are plugins installed on the system? Do they have to appear in a special directory on the file system? Do they need special setup.py magic to write extra files? Do they need to live in a pre-defined namespace? * Discoverability - How do frameworks discover all the plugins that are available? Which available plugins claim to support a particular plugin-point? How to do strict type checking on plugins? Which plugins are enabled? I'm sure there are more. As always, I'd like to see simple APIs on both sides that cover the common 80%. Both Tarek's and Michael's posts and proto-peps are great starts. You guys should definitely write up a plugin PEP! -Barry

On 30/07/2010 15:04, Barry Warsaw wrote:
Whilst in principle I agree with you... the plugin requirements for unittest(2) and disutils2 are very different. The biggest advantage of using ABCs in Tarek's post is around interfaces - you can ensure that registered plugins have the required interface. unittest doesn't *have* a required interface for plugins, which may optionally implement handlers for *any* of the unittest events and the rest of the functionality (configuration and command line interface integration) is provided by virtue of the subclassing. Explicit registration over implicit registration by subclassing is an interesting discussion, but I like the simplicity provided by just subclassing. unittest allows any namespace to provide a plugin but has no discoverability built in to it. Users specify which plugins they want to use (per project and per user), plugins are then activated by importing. Framework authors can load whichever plugins they want - it is probable that discoverability would be useful here. Automatic discoverability, a-la setuptools entry points, is not without its problems though. Tarek outlines some of these in a more recent blog post: https://tarekziade.wordpress.com/2010/07/25/plugins-system-thoughts-for-an-e... Again I think the *needs* of unittest and distutils are different, so I wonder if a single system can usefully suit both our needs (let alone universally support other systems). Definitely an area worth exploring though. Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Fri, Jul 30, 2010 at 4:34 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote: ...
Yes, even if they do if for different needs, both projects want to do the same thing: to be easily extensible by third party project. If we can provide a generic tool for both unittest and distutils extensions use cases, that's great and that will probably be useful for other projects. If we fail, I doubt we will ever have a generic plugin system in the stdlib. But that worth trying I think, and having different use cases is good for this imho

At 03:34 PM 7/30/2010 +0100, Michael Foord wrote:
FWIW, it's not discovery that's the problem, but configuring *which* plugins you wish to have active. Entry points support access by name, and it's up to the application using them to decide *which* ones to load. The underlying idea is that entry points expose a hook; it's up to the app to decide which ones it should actually import and use. An application also can list the available plugins and ask the user, etc. (For example, setuptools only loads "setup() argument" entry points for specified arguments, and command entry points only for the commands a user explicitly invokes.) IOW, entry points provide access to plugins, not policy or configuration for *which* plugins you wish to use. This was an intentional decision since applications vary widely in what sort of configuration mechanism they use. In the simplest cases (e.g. single-app environments like Chandler), simply making the plugin available on sys.path (e.g. via a special plugins directory) is configuration enough. In more complex use cases, an app might have to import plugins in order to get more information about them.

On 30/07/2010 21:56, P.J. Eby wrote:
Right (and thanks), and in the unittest plugin system the user decides which plugins should be active by listing them explicitly in the configuration file. Discovery could be useful for a tool that tells the user which plugins are available and modify the config file *for them* to switch plugins on and off. Useful metadata would then be which config options a plugin supports (and their defaults) so they can be added to the config file too when a plugin is activated. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Sat, Jul 31, 2010 at 12:34 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Note that ABCs are deliberately designed to let *users* choose to do either. Subclassing gets you better implementation support (since you pick up all the concrete method implementations "for free"), but you can still use explicit registration if you have an existing class that provides the API but isn't a subclass of the ABC. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Fri, Jul 30, 2010 at 4:04 PM, Barry Warsaw <barry@python.org> wrote: ..
FWIW We are thinking about adding in distutils2 a system quite similar to the entry points setuptools has, but with extra abilities for the end user : - activate / deactivate plugins without having to remove the project that added them - configure globally if plugins are implicitely activated or not -- and maybe allow the distutils2 installer to ask the user when a plugin is detected if he wants it activate or not - provide a tool to browse them This will be done through files added in the dist-info/ dirs, with the new PEP 376 api we are adding to pkgutil The idea is that the end user should be able to have a full control on what's activated in his system, without relying on the developer choices Cheers Tarek -- Tarek Ziadé | http://ziade.org

On 30/07/2010 15:37, Tarek Ziadé wrote:
This system sounds great. unittest could certainly use it for discovering plugins provided by other packages. The question then is still how to decide which ones should be active for any individual project (just because a plugin is available doesn't mean you want it used for every project). A configuration system is still good for this, but that kind of negates the advantage of discovery if the user still has to configure the plugin *anyway*. For framework authors not using the default test runner ("python -m unittest" or "unit2") this would be very useful. Michael
Cheers Tarek
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Fri, Jul 30, 2010 at 4:04 PM, Barry Warsaw <barry@python.org> wrote:
You guys should definitely write up a plugin PEP!
I am all for it, I am pretty sure we can come up with a generic tool that can be useful for many packages in the stdlib Starting this... -- Tarek Ziadé | http://ziade.org

This is my first post to python-dev, so for those who might not know me, I'm the author of Pro Django and more recently, Pro Python. I haven't looked at the plugin landscape in a while, but I was very disappointed the last time I looked at how complex typical systems were in this regard. There seems to be entirely too much time spend worrying about type checking, interface requirements and the like. Python is based on simple ideas providing simple solutions, and I would argue that duck typing is a perfectly acceptable route to take. I'd suggest that any plugin system proposal start off simple and work up as needs arise, rather than trying to cover as many bases as possible at the outset. I know it doesn't cover all the bases yet (particularly with regard to discoverability), so I'm not throwing it in as a proper suggestion, but I'll point out that I wrote up a dead-simple plugin system a while back[1] that may provide a useful starting point for discussion. I haven't read the existing proposals yet, but I just wanted to breathe a word of caution into this before it gets too far into interfaces and whatnot. I'll be glad to help with any discussion that takes place on this, though. It'll be a good way to ease into the mailing list, since it's an area where I've already spent a good deal of thought. -Marty [1] http://martyalchin.com/2008/jan/10/simple-plugin-framework/

On 30/07/2010 15:41, Marty Alchin wrote:
There is no type checking or interface requirements in my plugin proposal for unittest. It is essentially an event based system.
Tarek and I will look at what common ground there is between our plugin needs and see if *that* can be usefully abstracted out. FWIW I think PEP 376 (as amended - needs updating I *believe*) for plugin installation and discovery probably covers most of the common ground anyway. For the actual plugins our two systems are very different.
I haven't read the existing proposals yet,
If you want to help us this may be a good place to start... ;-) Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

Le 30/07/2010 17:04, Michael Foord a écrit :
There is no type checking or interface requirements in my plugin proposal for unittest. It is essentially an event based system.
Event-based sounds good. unittest2 does have an interface IMO: configuration loading, plugin registration/activation, signal names. As you said, the common ground may be little more than an API to discover or register plugins, activate and configure them, and the design decision of using events and callbacks, but even this small subset is very much worth a PEP. Regards

On 30/07/2010 16:28, Éric Araujo wrote:
It has an API, but the plugins are not interface based (so interface requirements don't need to be part of the plugin system).
PEP 376 is *very* promising for this part - well at least if the PLUGINS stuff becomes part of PEP 376, it doesn't seem to be mentioned at all at the moment. Based on what you and tarek are saying it *sounds* good though... (Metadata for packages to declare that they provide plugins for another package and an API for frameworks to discover installed plugins.) Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

It has an API, but the plugins are not interface based (so interface requirements don't need to be part of the plugin system).
Oh, I see. With duck-typing and ABCs, I don’t make a difference between protocols and interfaces in my head :) Also, the I in API does mean interface, but not imply bondage-and-discipline interface systems.
The dist-info directory introduced by PEP 376 will certainly be reused. Not sure if Tarek will want to edit PEP 376 (already accepted) to add plugins in metadata (provides-dist is in metadata, so having plugins here can make sense), or start a new PEP about a new concept in a new file. Regards

On Fri, Jul 30, 2010 at 5:54 PM, Éric Araujo <merwok@netwok.org> wrote:
What we want to do I think is to make dist-info a place when any file can be added, besides the ones described in PEP 376, then build the plugins proposal on this
-- Tarek Ziadé | http://ziade.org

For those of you who found this document perhaps just a little bit too long, I've written up a *much* shorter intro to the plugin system (including how to get the prototype) on my blog: http://www.voidspace.org.uk/python/weblog/arch_d7_2010_07_24.shtml#e1186 Michael On 29 July 2010 23:55, Michael Foord <fuzzyman@voidspace.org.uk> wrote:

On Jul 30, 2010, at 6:23 AM, Michael Foord wrote:
For those of you who found this document perhaps just a little bit too long, I've written up a *much* shorter intro to the plugin system (including how to get the prototype) on my blog:
http://www.voidspace.org.uk/python/weblog/arch_d7_2010_07_24.shtml#e1186
Nice work. Raymond

On Fri, Jul 30, 2010 at 10:23 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
This looks nice and simple, but I am a bit worried about the configuration file for registration. My experience is that end users don't like editing files much. I understand that may be considered as bikesheding, but have you considered a system analog to bzr instead ? A plugin is a directory somewhere, which means that disabling it is just removing a directory. In my experience, it is more reliable from a user POV than e.g. the hg way of doing things. The plugin system of bzr is one of the thing that I still consider the best in its category, even though I stopped using bzr for quite some time. The registration was incredibly robust and easy to use from a user and developer POV, David

2010/7/31 David Cournapeau <cournape@gmail.com>:
Hi David, I think the point Michael tries to make is to be able to activate (not register) some plugins for some projects. In this case, even if bzr system is used (and I agree, it's a really good system), how will you activate only some of them for your project? I don't think it is geared towards end users, much more towards developers (at least for the unittest plugins) in which case adding a .cfg file to your vcs is not much work to do ;) Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher

On 31/07/2010 01:51, David Cournapeau wrote:
Definitely not bikeshedding, a useful suggestion David. As Matthieu says in his reply, individual projects need to be able to enable (and configure) individual plugins that their tests depend on - potentially even shipping the plugin with the project. The other side of this is generally useful plugins that developers may want to have permanently active (like the debugger plugin), so that it is always available to them (via a command line switch). The proposed system allows this with a user configuration file plus a per-project configuration file. I take your point about users not liking configuration files though. I've looked a little bit at the bzr plugin system and I like the plugins subcommand. If PEP 376 goes ahead then we could keep the user plugin and use the PEP 376 metadata, in concert with a user config file, to discover all plugins *available*. A plugins subcommand could then activate / deactivate individual plugins by editing (or creating) the config file for the user. This could be bolted *on top* of the config file solution once PEP 376 is in place. It *doesn't* handle the problem of configuring plugins. So long as metadata is available about what configuration options plugins have (through a plugins API) then the plugins subcommand could also handle configuration. Installation of plugins would still be done through the standard distutils(2) machinery. (Using PEP 376 would depend on distutils2. I would be fine with this.) Another possibility would be to have a zero-config plugin installation solution *as well* as the config files. Create a plugins directory (in the user home directory?) and automatically activate plugins in this directory. This violates TOOWTDI though. As it happens adding a plugin directory would be easy to implement as a plugin... All the best, Michael
David
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 31/07/2010 12:46, Michael Foord wrote:
[snip...] If PEP 376 goes ahead then we could keep the user plugin
I meant "keep the user config file". Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

Putting .pydistutils.cfg .pypirc .unittest2.cfg .idlerc and possibly other in the user home directory (or %APPDATA% on win32 and what-have-you on Mac) is unnecessary clutter. However, $PYTHONUSERBASE is not the right directory for configuration files, as pointed in http://bugs.python.org/issue7175 It would be nice to agree on a ~/.python (resp. %APPADATA%/Python) or $XDG_CONFIG_HOME/python directory and put config files there. Regards

On Sun, 01 Aug 2010 17:22:55 +0200, <merwok@netwok.org> wrote:
+1 Certainly ~/unittest.cfg is the wrong name for unix (it doesn't start with a '.'), and certainly the location is OS specific. Anyone who cares about config file locations should read issue 7175. -- R. David Murray www.bitdance.com

On 01/08/2010 18:38, R. David Murray wrote:
I'm happy to choose a location / name based on consensus. There doesn't seem to be any consensus yet. As a (mainly ex) windows user I would hate to have user editable data in APPDATA as it is not a location the user ever expects to visit. The home directory, or a subdirectory thereof, for user editable app specific data is more usual and more friendly. All the best, Michael Foord
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Aug 1, 2010, at 3:52 PM, Ronald Oussoren wrote:
"100% formally" speaking, MacOS behaves like UNIX in many ways. <http://en.wikipedia.org/wiki/Single_UNIX_Specification#Mac_OS_X_and_Mac_OS_X...> It's fine to have a mac-pathname-convention-following place for such data, but please _also_ respect the UNIX-y version on the Mac. The only possible outcome of python on the Mac respect only Mac pathnames is to have automation scripts that work fine on BSD and Linux, but then break when you try to run them on a Mac. There is really no benefit to intentionally avoiding honoring the UNIX conventions. (For another example, note that although Python resides in /System/Library, on the mac, the thing that's in your $PATH when you're using a terminal is the symlink in /usr/bin/python.) Also, no, "~/Preferences" isn't the right place for it either; there's no such thing. You probably meant "~/Library/Preferences". I'd say that since ~/Library/Python is already used, there's no particular reason to add a new ~/Library/Preferences/Python location. After all, if you really care a lot about platform conventions, you should put it in ~/Library/Preferences/org.python.distutils.plist, but I don't see what benefit that extra complexity would have for anyone.

On 2 Aug, 2010, at 7:18, Glyph Lefkowitz wrote:
Storing files in unix location will be confusing to many Mac users, Apple has an explicitly documented convention for where to store files and dot-files in the user's home directory aren't part of that convention. An important reason for storing files in ~/Library/Python of ~/Library/Preferences/Python is that these locations are both logical for mac users and can be navigated to from the Finder without resorting to typing the folder name in "Go -> Go to Folder".
It's fine to have a mac-pathname-convention-following place for such data, but please _also_ respect the UNIX-y version on the Mac. The only possible outcome of python on the Mac respect only Mac pathnames is to have automation scripts that work fine on BSD and Linux, but then break when you try to run them on a Mac.
The stdlib should have APIs for locating common directories, although I must admit that I haven't checked if it actually does have them. That way you can write automation scripts that work anyware, not just on systems that look a lot like Linux. I've written a lot of scripts that had to follow platform conventions on a lot of unix platforms, and those tended to require small changes for every new unix flavor we encountered. Non-linux platforms also have filesystem hierarchy standards, even though they are often not as detailed as the Linux ones and most unix platforms don't have a user-base that is as vocal as the linux packagers when software doesn't follow the conventions.
There is really no benefit to intentionally avoiding honoring the UNIX conventions. (For another example, note that although Python resides in /System/Library, on the mac, the thing that's in your $PATH when you're using a terminal is the symlink in /usr/bin/python.)
Also, no, "~/Preferences" isn't the right place for it either; there's no such thing. You probably meant "~/Library/Preferences". I'd say that since ~/Library/Python is already used, there's no particular reason to add a new ~/Library/Preferences/Python location. After all, if you really care a lot about platform conventions, you should put it in ~/Library/Preferences/org.python.distutils.plist, but I don't see what benefit that extra complexity would have for anyone.
Your right, I meant ~/Library/Preferences, and I'd prefer ~/Library/Python for the reason you meant. The actual format of the configuration files is not prescribed in any way, and I'd by -1 on storing the preferences in a plist on OSX because that is making live for programmers actively harder. Ronald

On 02/08/2010 07:18, Ronald Oussoren wrote:
Really? As a Mac user I have never edited (or even looked at) files in ~/Library. I would never think of going there for finding config files to edit. However in my home directory I have: .Xauthority .Xcode . CFUserTextEncoding - (an Apple encoding configuration for Core Foundation) .bash_profile .cups .dropbox .dvdcss .filezilla .fontconfig .hgrc .idlerc .ipython .mono .netbeans .parallels_settings .pypirc .wingide3 Actually that is just a small selection of the .config files/directories in my home directory. It is certainly *a* standard location for config files on the Mac, including some Apple software (XCode) and Python applications. My preference would be to follow this established and well used convention. Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 02 Aug, 2010,at 11:48 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote: On 02/08/2010 07:18, Ronald Oussoren wrote: On 2 Aug, 2010, at 7:18, Glyph Lefkowitz wrote: On Aug 1, 2010, at 3:52 PM, Ronald Oussoren wrote: On 1 Aug, 2010, at 17:22, Éric Araujo wrote: Speaking of which... Your documentation says it's named ~/unittest.cfg, could you make this a file in the user base (that is, the prefix where 'setup.py install --user' will install files)? Putting .pydistutils.cfg .pypirc .unittest2.cfg .idlerc and possibly other in the user home directory (or %APPDATA% on win32 and what-have-you on Mac) is unnecessary clutter. However, $PYTHONUSERBASE is not the right directory for configuration files, as pointed in http://bugs.python.org/issue7175 It would be nice to agree on a ~/.python (resp. %APPADATA%/Python) or $XDG_CONFIG_HOME/python directory and put config files there. ~/Library/Python would be a good location on OSX, even if the 100% formally correct location would be ~/Preferences/Python (at least of framework builds, unix-style builds may want to follow the unix convention). "100% formally" speaking, MacOS behaves like UNIX in many ways. <http://en.wikipedia.org/wiki/Single_UNIX_Specification#Mac_OS_X_and_Mac_OS_X...> Storing files in unix location will be confusing to many Mac users, Apple has an explicitly documented convention for where to store files and dot-files in the user's home directory aren't part of that convention. An important reason for storing files in ~/Library/Python of ~/Library/Preferences/Python is that these locations are both logical for mac users and can be navigated to from the Finder without resorting to typing the folder name in "Go -> Go to Folder". Really? As a Mac user I have never edited (or even looked at) files in ~/Library. I would never think of going there for finding config files to edit. However in my home directory I have: .Xauthority .Xcode . CFUserTextEncoding - (an Apple encoding configuration for Core Foundation) .bash_profile .cups .dropbox .dvdcss .filezilla .fontconfig .hgrc .idlerc .ipython .mono .netbeans .parallels_settings .pypirc .wingide3 Actually that is just a small selection of the .config files/directories in my home directory. It is certainly *a* standard location for config files on the Mac, including some Apple software (XCode) and Python applications. The only apple one that is actually used is the .CFUserTextEncoding file, I have an .Xcode in my home as well but that is empty and last updated in 2007. AFAIK current versions of Xcode store preferences in ~/Library/Preferences. Most of the other ones are ports of unix tools and store junk in the standard unix location for storing configuration. Try edit one without resorting to the command-line, with a default configuration of the Finder you cannot even see these files (and that includes the File open dialog of tools like Text Edit). The reason you don't normally look in ~/Library/Preferences is that GUI tools on OSX have configuration screens for editing preferences and you don't have to edit them manually. My preference would be to follow this established and well used convention. My preference is still to use ~/Library/Python (or a subdirectory thereof) and filenames that don't start with a dot. Ronald

On 02/08/2010 11:48, Ronald Oussoren wrote:
Right, so what you are saying is that for user editable text config files ~/Library/Preferences is *not* a convention - and in fact the unix convention of dot files in the home directory is commonly used on the Mac. :-) Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 02/08/2010 11:48, Ronald Oussoren wrote:
The configuration files we are discussing are for command line tools - so I don't think that having to resort to the command line is a disadvantage at all. If users don't / can't use the command line then they *won't* want to edit these files anyway. If they are used to the command line then ~/.python32/distutils.cfg is going to be a very natural place for them. If we provide GUI tools that use these config files then we will also provide GUI tools that use these config files then we will also provide GUI tools to configure them - so I can't see a downside to having them in the unix location and no upside to putting them elsewhere. Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 02 Aug, 2010,at 01:00 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
The configuration files we are discussing are for command line tools - so I don't think that having to resort to the command line is a disadvantage at all. If users don't / can't use the command line then they *won't* want to edit these files anyway. Not being comfortable at the command-line is not the same as not wanting to edit the global configuration of unittest or distutils. Anyway, does that mean that the configuration should move if I create a patch for IDLE that allows you the manage the unittest configuration through a GUI? If they are used to the command line then ~/.python32/distutils.cfg is going to be a very natural place for them. That location isn't natural for me. If we invent a new location for python-related configuration file we might as well do it properly and follow platform conventions. The MacOSX conventions are described here <http://developer.apple.com/mac/library/documentation/MacOSX/Conceptual/BPFil...>. Macosx-is-not-a-crappy-linux-ly yours, Ronald

On 02/08/2010 14:34, Ronald Oussoren wrote:
Nope, as I doubt for *most* users it will be the primary way of editing the file.
But you yourself said that it *isn't* normal to have files we expect the user to edit in the location you suggested - and a glance on my system is that use ~/.app files is a *very* common convention on the Mac. Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 02/08/2010 14:34, Ronald Oussoren wrote:
But both of those are primarily command line tools. A basic ability to use the command line is a prerequisite of wanting to configure them. I would be interested in hearing from other Mac users as to where they would look for configuration files for command line tools - in ~ or in ~/Library/Preferences? Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

Michael Foord <fuzzyman@voidspace.org.uk> writes:
My primary personal machine has been OSX for years now, and as someone who lives in the command line (well, or Emacs shells), I think either approach is workable for command line only tools. I will say that if I need to find preferences or application-specific files my first inclination is absolutely to look under ~/Library. That's the "platform" location (just as looking for dot files under Linux is my first choice, or "Documents and Settings/<user>" for Windows). I don't think I shift mental gears automatically for command line tools versus not, unless I have some prior knowledge about the tool. With that said, given all the third party and Unix-oriented stuff I install under OSX, it's hardly rare (as has been pointed out elsewhere) to be working with ~/.<something> either, so it's not like I'd consider it that unusual to find that's where I need to go. In glancing at my current system, it does appear command line only tools are more commonly using ~/.<something> files rather than under ~/Library (which tends to be stuff packaged up as an application in /Applications, even if they can also be run from the command line). Though it might be a biased sample set since I'm more likely to have brought in command line tools to OSX from the Unix side of things, I suspect that's true of other users of command line tools as well. I will say that it's rarer to find a native (Cocoa/Carbon) GUI application that doesn't store preferences or application settings beneath ~/Library, and in such a case I'd feel they were more "wrong" and non-conforming if they didn't do that. So it depends on how "native" an application wishes to be perceived. I guess in thinking about it while writing this, having something installed in /Applications is more strongly linked with ~/Library in my mind than other tools. Of course, even with /Applications, non-native GUI apps are more of a mixed bag. For example, the X versions of Gimp and Inkscape - Gimp properly uses "~/Library/Application Support" while Inkscape still uses ~/.inkscape. Of course, as X apps, neither truly feels native or conforming anyway. So that probably helps make things as clear as mud :-) -- David

(Ronald, the text version of your message was very difficult to sort through and read, because all of the quoting information was lost. Just thought you'd want to know.) What I hear Glyph saying is that we should support looking in *both* locations for configuration info on OSX, and I don't see a downside to that. Most unix applications look in multiple places for configuration info. Michael seems to be arguing for not using the standard OSX locations because the Finder can't edit them anyway. Is that true? -- R. David Murray www.bitdance.com

On 02/08/2010 15:24, R. David Murray wrote:
That adds extra complexity to the implementation of the configuration system. If it uses the first one it finds which order does it look in, if we have tools that create the file which location does it create it in and so on.
Michael seems to be arguing for not using the standard OSX locations because the Finder can't edit them anyway. Is that true?
I am saying that Ronald's suggestion is *not* a natural location for user editable configuration files - as far as I can tell I have no user editable files there and plenty in ~/.something. Ronald himself said that the location he is specifying is the standard location for configuration / preference files created and used by *gui* tools, and files in that location are not usually intended to be user editable. I don't believe a Mac user basically competent at the command line is *likely* to go looking in the gui preferences location for these config files and I think they would easily find them in ~. This is backed up the number of existing programs that use this convention on Mac OS X. It *is* a widely used convention on Mac OS X to use ~/.appname for configuration files. Applications like mercurial use this location on the Mac for example. Ronald was wrong when he said that the only configuration file in ~ used by the Mac itself is .CFUserTextEncoding. Terminal (the Mac OS X command line) has a user editable config file which it stores in ~. The issue with the finder is that by default . files and directories aren't shown and so they wouldn't be editable from the finder. As basic willingness to use the command line is a prerequisite for *wanting* to edit these files I don't see this as an issue. A user unfamiliar with the command line is not likely to guess the correct location for these files if we put them elsewhere, so they are going to have to refer to some documentation anyway. Michael
-- R. David Murray www.bitdance.com
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 2 Aug, 2010, at 16:24, R. David Murray wrote:
I'll stop using the mobile-me webmail client for lists, it seems to mess things up.
A lot of tools seem to look both in a system location and a per user location (such as /etc/profile and ~/.profile). OSX ads a 3th level to that, although I have never used that myself (technically there are 4 levels, but that isn't important unless you are Apple).
Michael seems to be arguing for not using the standard OSX locations because the Finder can't edit them anyway. Is that true?
The Finder can open OSX locations just fine, but you cannot see dot-files (or -directories) in the Finder. I won't argue this anymore, at least not this week. I'll work around the issue in my private tree if that's what's needed. Ronald

Glyph Lefkowitz wrote:
I'd say that since ~/Library/Python is already used, there's no particular reason to add a new ~/Library/Preferences/Python location.
I think the reason for separating out Preferences is so that you can install a new version of a library or application without losing the user's preferences from the previous version. -- Greg

On 31/07/2010 17:22, Tarek Ziadé wrote:
Ok. It would be helpful for unittest2 (the backport) if it was *still* available in distutils2 even after the merge into pkgutil (for use by earlier versions of Python). I guess you will do this anyway for the distutils2 backport itself anyway... (?) All the best, Michael Foord
Regards Tarek
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

Yes. Even if the goal is to have distutils2 in the stdlib for 3.2 or 3.3, there will still be a standalone release on PyPI for Python 2.4-3.1. You’ll just have to write such compat code: try: from pkgutil import shiny_new_function except ImportError: from distutils2._backport.pkgutil import shiny_new_function Regards

Damn, the email was truncated. Probably my fault. The part missed off is: Not Yet Implemented =================== Except where noted, everything in this document is already working in the prototype. There are a few open issues and things still to be implemented. Certain event attributes should be read only (like extraTests, extraNames etc) because they should only be extended or modified in place instead of being replaced. Add an epilogue to the optparse help messages. A messaging API for plugins that respects verbosity. The ``startTestRun`` event needs an alternative way of replacing the default test run machinery without stopping other plugins from executing. It should only be possible for *one* plugin to use this. Should the ``onTestFailEvent`` be able to supress raised exceptions? It should also be able to modify the traceback so that bare asserts could be used but still provide useful diagnostic information. Should this event be fired for test skips? Command line options to unittest, like verbosity, should be put into the config data structure so that they are accessible to plugins. TestFailEvent needs to be fired on failures in setUpClass and setUpModule etc Plugin.register and Plugin.unregister aren't precisely complementary. This means that unregister can only be called once. This may not be an issue. Plugin.unregister will fail if any events have already been unhooked. This can easily be fixed. If the merge to unittest in Python 3.2 happens we need to decide which of the example plugins will be included in unittest. (They will probably all remain in unittest2.) Custom test outcomes are *not* supported; making interoperating tools or plugins in the presence of custom outcomes can be very hard. Should unittest2 have a different config file from unittest, so they can be configured side-by-side? (``unittest2.cfg`` perhaps.) Alternatively the same config file could be used with a '[unittest2]' section instead of '[unittest]'. Should ``StopTestEvent.timeTaken`` (plus startTime / stopTime etc) include the time for setUp, tearDown and cleanUps? Should this information be available separately perhaps? The discovery command line interface has changed a lot and the tests need reviewing to ensure they still provide full coverage. Should there be an "excluded-plugins" section in the config files, so projects can prevent incompatible plugins from being loaded? Should multiline values in config files be merged instead of overriding each other. (This is effectively what happens for the plugins list.) Should ``handleFile`` be fired when a test name is specified at the command line? (This would be 'tricky' as ``handleFile`` will have to be called for the containing module and then the correct name pulled out of the module.) Additional Changes ================== Alongside, and in support of, the plugin system a few changes have been made to unittest2. These either have no, or very minor, backwards compatibility issues. Changes so far are: TestLoader has a new attribute ``DEFAULT_PATTERN``. This is so that the regex matching plugin can change the default pattern used for test discovery when no pattern is explicitly provided. Command line parsing is all done by optparse, removing the use of getopt. This makes the help messages more consistent but makes the usage messages less useful in some situations. This can be fixed with the use of the optparse epilogue. unit2 (the default test runner) runs test discovery if invoked without any arguments. unit2 can execute tests in filenames as well as module names - so long as the module pointed to by the filename is importable from the current directory. FunctionTestCase.id() returns 'module.funcname' instead of just funcname. Added util.formatTraceback, the default way of formatting tracebacks. TestCase has a new formatTraceback method (delegating to util.formatTraceback). TestCase instances can implement formatTraceback to control how the traceback for errors and failures are represented. Useful for test items that don't represent Python tests, for example the pep8 / pyflakes checker and theoretical javascript test runners such as exist for py.test and nose. If you specify test names (modules, classes etc) at the command line they will be loaded individually using ``loader.loadTestsFromName`` instead of collectively with ``loader.loadTestsFromNames``. This enables individual names to be checked to see if they refer to filenames. References ========== .. [#] See http://bitbucket.org/jpellerin/unittest2/src/tip/unittest2/plugins/attrib.py and http://bitbucket.org/jpellerin/unittest2/src/tip/unittest2/plugins/errormast... .. [#] http://lists.idyll.org/pipermail/testing-in-python/2010-March/002799.html On 29/07/2010 23:55, Michael Foord wrote:
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Fri, Jul 30, 2010 at 12:55 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote: ...
What about using ABCs here instead ? The big advantage is that you don't force people to subclass your Plugin base class, and you get a registration mechanism for free. They would be able to explicitly register any class using : from unittest2.events import Plugin Plugin.register(MySuperPlugin) Regards Tarek -- Tarek Ziadé | http://ziade.org

On 30/07/2010 11:09, Tarek Ziadé wrote:
Is that an advantage? Subclassing Plugin provides functionality rather than particularly an interface.
and you get a registration mechanism for free.
Actually it looks like it would be *extra* code. (The registration mechanism is already "free" to the programmer when they subclass.) I will investigate this, but if the only benefit it provides is "you don't have to subclass" then I'm not sure there is any point. Note that the Plugin class is only "sugar" over hooking up to the events directly *anyway*, so there is no *need* to use Plugin to write extensions. I'm going to read your blog entry on the topic to evaluate it properly though: https://tarekziade.wordpress.com/2009/05/01/basic-plugin-system-using-abcs-a... Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Jul 30, 2010, at 11:38 AM, Michael Foord wrote:
Very interesting. For Mailman 3, I have YAPS (yet another plugin system) based on zope.interface and setuptools. Bazaar has its own plugin system which is different still. I bet there are as many plugin APIs as there are Python frameworks. (And isn't everything a framework these days? :) I'm glad to see discussion begin to focus on providing consolidation in the world of plugins for Python frameworks, and to begin to refactor basic functionality into common tools. I'd love to see a blessed plugin API promoted to the stdlib for Python 3.2. I think it has to address a number of issues: * Registration - How do third party plugins declare themselves to exist, and be enabled? Part of this seems to me to include interface declarations too. Is installation of the plugin enough to register it? How do end users enable and disable plugins that me be registered on their system? How do plugins describe themselves (provide short and log descriptions, declare options, hook into command line interfaces, etc.)? * Installation - How are plugins installed on the system? Do they have to appear in a special directory on the file system? Do they need special setup.py magic to write extra files? Do they need to live in a pre-defined namespace? * Discoverability - How do frameworks discover all the plugins that are available? Which available plugins claim to support a particular plugin-point? How to do strict type checking on plugins? Which plugins are enabled? I'm sure there are more. As always, I'd like to see simple APIs on both sides that cover the common 80%. Both Tarek's and Michael's posts and proto-peps are great starts. You guys should definitely write up a plugin PEP! -Barry

On 30/07/2010 15:04, Barry Warsaw wrote:
Whilst in principle I agree with you... the plugin requirements for unittest(2) and disutils2 are very different. The biggest advantage of using ABCs in Tarek's post is around interfaces - you can ensure that registered plugins have the required interface. unittest doesn't *have* a required interface for plugins, which may optionally implement handlers for *any* of the unittest events and the rest of the functionality (configuration and command line interface integration) is provided by virtue of the subclassing. Explicit registration over implicit registration by subclassing is an interesting discussion, but I like the simplicity provided by just subclassing. unittest allows any namespace to provide a plugin but has no discoverability built in to it. Users specify which plugins they want to use (per project and per user), plugins are then activated by importing. Framework authors can load whichever plugins they want - it is probable that discoverability would be useful here. Automatic discoverability, a-la setuptools entry points, is not without its problems though. Tarek outlines some of these in a more recent blog post: https://tarekziade.wordpress.com/2010/07/25/plugins-system-thoughts-for-an-e... Again I think the *needs* of unittest and distutils are different, so I wonder if a single system can usefully suit both our needs (let alone universally support other systems). Definitely an area worth exploring though. Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Fri, Jul 30, 2010 at 4:34 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote: ...
Yes, even if they do if for different needs, both projects want to do the same thing: to be easily extensible by third party project. If we can provide a generic tool for both unittest and distutils extensions use cases, that's great and that will probably be useful for other projects. If we fail, I doubt we will ever have a generic plugin system in the stdlib. But that worth trying I think, and having different use cases is good for this imho

At 03:34 PM 7/30/2010 +0100, Michael Foord wrote:
FWIW, it's not discovery that's the problem, but configuring *which* plugins you wish to have active. Entry points support access by name, and it's up to the application using them to decide *which* ones to load. The underlying idea is that entry points expose a hook; it's up to the app to decide which ones it should actually import and use. An application also can list the available plugins and ask the user, etc. (For example, setuptools only loads "setup() argument" entry points for specified arguments, and command entry points only for the commands a user explicitly invokes.) IOW, entry points provide access to plugins, not policy or configuration for *which* plugins you wish to use. This was an intentional decision since applications vary widely in what sort of configuration mechanism they use. In the simplest cases (e.g. single-app environments like Chandler), simply making the plugin available on sys.path (e.g. via a special plugins directory) is configuration enough. In more complex use cases, an app might have to import plugins in order to get more information about them.

On 30/07/2010 21:56, P.J. Eby wrote:
Right (and thanks), and in the unittest plugin system the user decides which plugins should be active by listing them explicitly in the configuration file. Discovery could be useful for a tool that tells the user which plugins are available and modify the config file *for them* to switch plugins on and off. Useful metadata would then be which config options a plugin supports (and their defaults) so they can be added to the config file too when a plugin is activated. Michael -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Sat, Jul 31, 2010 at 12:34 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
Note that ABCs are deliberately designed to let *users* choose to do either. Subclassing gets you better implementation support (since you pick up all the concrete method implementations "for free"), but you can still use explicit registration if you have an existing class that provides the API but isn't a subclass of the ABC. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Fri, Jul 30, 2010 at 4:04 PM, Barry Warsaw <barry@python.org> wrote: ..
FWIW We are thinking about adding in distutils2 a system quite similar to the entry points setuptools has, but with extra abilities for the end user : - activate / deactivate plugins without having to remove the project that added them - configure globally if plugins are implicitely activated or not -- and maybe allow the distutils2 installer to ask the user when a plugin is detected if he wants it activate or not - provide a tool to browse them This will be done through files added in the dist-info/ dirs, with the new PEP 376 api we are adding to pkgutil The idea is that the end user should be able to have a full control on what's activated in his system, without relying on the developer choices Cheers Tarek -- Tarek Ziadé | http://ziade.org

On 30/07/2010 15:37, Tarek Ziadé wrote:
This system sounds great. unittest could certainly use it for discovering plugins provided by other packages. The question then is still how to decide which ones should be active for any individual project (just because a plugin is available doesn't mean you want it used for every project). A configuration system is still good for this, but that kind of negates the advantage of discovery if the user still has to configure the plugin *anyway*. For framework authors not using the default test runner ("python -m unittest" or "unit2") this would be very useful. Michael
Cheers Tarek
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Fri, Jul 30, 2010 at 4:04 PM, Barry Warsaw <barry@python.org> wrote:
You guys should definitely write up a plugin PEP!
I am all for it, I am pretty sure we can come up with a generic tool that can be useful for many packages in the stdlib Starting this... -- Tarek Ziadé | http://ziade.org

This is my first post to python-dev, so for those who might not know me, I'm the author of Pro Django and more recently, Pro Python. I haven't looked at the plugin landscape in a while, but I was very disappointed the last time I looked at how complex typical systems were in this regard. There seems to be entirely too much time spend worrying about type checking, interface requirements and the like. Python is based on simple ideas providing simple solutions, and I would argue that duck typing is a perfectly acceptable route to take. I'd suggest that any plugin system proposal start off simple and work up as needs arise, rather than trying to cover as many bases as possible at the outset. I know it doesn't cover all the bases yet (particularly with regard to discoverability), so I'm not throwing it in as a proper suggestion, but I'll point out that I wrote up a dead-simple plugin system a while back[1] that may provide a useful starting point for discussion. I haven't read the existing proposals yet, but I just wanted to breathe a word of caution into this before it gets too far into interfaces and whatnot. I'll be glad to help with any discussion that takes place on this, though. It'll be a good way to ease into the mailing list, since it's an area where I've already spent a good deal of thought. -Marty [1] http://martyalchin.com/2008/jan/10/simple-plugin-framework/

On 30/07/2010 15:41, Marty Alchin wrote:
There is no type checking or interface requirements in my plugin proposal for unittest. It is essentially an event based system.
Tarek and I will look at what common ground there is between our plugin needs and see if *that* can be usefully abstracted out. FWIW I think PEP 376 (as amended - needs updating I *believe*) for plugin installation and discovery probably covers most of the common ground anyway. For the actual plugins our two systems are very different.
I haven't read the existing proposals yet,
If you want to help us this may be a good place to start... ;-) Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

Le 30/07/2010 17:04, Michael Foord a écrit :
There is no type checking or interface requirements in my plugin proposal for unittest. It is essentially an event based system.
Event-based sounds good. unittest2 does have an interface IMO: configuration loading, plugin registration/activation, signal names. As you said, the common ground may be little more than an API to discover or register plugins, activate and configure them, and the design decision of using events and callbacks, but even this small subset is very much worth a PEP. Regards

On 30/07/2010 16:28, Éric Araujo wrote:
It has an API, but the plugins are not interface based (so interface requirements don't need to be part of the plugin system).
PEP 376 is *very* promising for this part - well at least if the PLUGINS stuff becomes part of PEP 376, it doesn't seem to be mentioned at all at the moment. Based on what you and tarek are saying it *sounds* good though... (Metadata for packages to declare that they provide plugins for another package and an API for frameworks to discover installed plugins.) Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

It has an API, but the plugins are not interface based (so interface requirements don't need to be part of the plugin system).
Oh, I see. With duck-typing and ABCs, I don’t make a difference between protocols and interfaces in my head :) Also, the I in API does mean interface, but not imply bondage-and-discipline interface systems.
The dist-info directory introduced by PEP 376 will certainly be reused. Not sure if Tarek will want to edit PEP 376 (already accepted) to add plugins in metadata (provides-dist is in metadata, so having plugins here can make sense), or start a new PEP about a new concept in a new file. Regards

On Fri, Jul 30, 2010 at 5:54 PM, Éric Araujo <merwok@netwok.org> wrote:
What we want to do I think is to make dist-info a place when any file can be added, besides the ones described in PEP 376, then build the plugins proposal on this
-- Tarek Ziadé | http://ziade.org

For those of you who found this document perhaps just a little bit too long, I've written up a *much* shorter intro to the plugin system (including how to get the prototype) on my blog: http://www.voidspace.org.uk/python/weblog/arch_d7_2010_07_24.shtml#e1186 Michael On 29 July 2010 23:55, Michael Foord <fuzzyman@voidspace.org.uk> wrote:

On Jul 30, 2010, at 6:23 AM, Michael Foord wrote:
For those of you who found this document perhaps just a little bit too long, I've written up a *much* shorter intro to the plugin system (including how to get the prototype) on my blog:
http://www.voidspace.org.uk/python/weblog/arch_d7_2010_07_24.shtml#e1186
Nice work. Raymond

On Fri, Jul 30, 2010 at 10:23 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
This looks nice and simple, but I am a bit worried about the configuration file for registration. My experience is that end users don't like editing files much. I understand that may be considered as bikesheding, but have you considered a system analog to bzr instead ? A plugin is a directory somewhere, which means that disabling it is just removing a directory. In my experience, it is more reliable from a user POV than e.g. the hg way of doing things. The plugin system of bzr is one of the thing that I still consider the best in its category, even though I stopped using bzr for quite some time. The registration was incredibly robust and easy to use from a user and developer POV, David

2010/7/31 David Cournapeau <cournape@gmail.com>:
Hi David, I think the point Michael tries to make is to be able to activate (not register) some plugins for some projects. In this case, even if bzr system is used (and I agree, it's a really good system), how will you activate only some of them for your project? I don't think it is geared towards end users, much more towards developers (at least for the unittest plugins) in which case adding a .cfg file to your vcs is not much work to do ;) Matthieu -- Information System Engineer, Ph.D. Blog: http://matt.eifelle.com LinkedIn: http://www.linkedin.com/in/matthieubrucher

On 31/07/2010 01:51, David Cournapeau wrote:
Definitely not bikeshedding, a useful suggestion David. As Matthieu says in his reply, individual projects need to be able to enable (and configure) individual plugins that their tests depend on - potentially even shipping the plugin with the project. The other side of this is generally useful plugins that developers may want to have permanently active (like the debugger plugin), so that it is always available to them (via a command line switch). The proposed system allows this with a user configuration file plus a per-project configuration file. I take your point about users not liking configuration files though. I've looked a little bit at the bzr plugin system and I like the plugins subcommand. If PEP 376 goes ahead then we could keep the user plugin and use the PEP 376 metadata, in concert with a user config file, to discover all plugins *available*. A plugins subcommand could then activate / deactivate individual plugins by editing (or creating) the config file for the user. This could be bolted *on top* of the config file solution once PEP 376 is in place. It *doesn't* handle the problem of configuring plugins. So long as metadata is available about what configuration options plugins have (through a plugins API) then the plugins subcommand could also handle configuration. Installation of plugins would still be done through the standard distutils(2) machinery. (Using PEP 376 would depend on distutils2. I would be fine with this.) Another possibility would be to have a zero-config plugin installation solution *as well* as the config files. Create a plugins directory (in the user home directory?) and automatically activate plugins in this directory. This violates TOOWTDI though. As it happens adding a plugin directory would be easy to implement as a plugin... All the best, Michael
David
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 31/07/2010 12:46, Michael Foord wrote:
[snip...] If PEP 376 goes ahead then we could keep the user plugin
I meant "keep the user config file". Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

Putting .pydistutils.cfg .pypirc .unittest2.cfg .idlerc and possibly other in the user home directory (or %APPDATA% on win32 and what-have-you on Mac) is unnecessary clutter. However, $PYTHONUSERBASE is not the right directory for configuration files, as pointed in http://bugs.python.org/issue7175 It would be nice to agree on a ~/.python (resp. %APPADATA%/Python) or $XDG_CONFIG_HOME/python directory and put config files there. Regards

On Sun, 01 Aug 2010 17:22:55 +0200, <merwok@netwok.org> wrote:
+1 Certainly ~/unittest.cfg is the wrong name for unix (it doesn't start with a '.'), and certainly the location is OS specific. Anyone who cares about config file locations should read issue 7175. -- R. David Murray www.bitdance.com

On 01/08/2010 18:38, R. David Murray wrote:
I'm happy to choose a location / name based on consensus. There doesn't seem to be any consensus yet. As a (mainly ex) windows user I would hate to have user editable data in APPDATA as it is not a location the user ever expects to visit. The home directory, or a subdirectory thereof, for user editable app specific data is more usual and more friendly. All the best, Michael Foord
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On Aug 1, 2010, at 3:52 PM, Ronald Oussoren wrote:
"100% formally" speaking, MacOS behaves like UNIX in many ways. <http://en.wikipedia.org/wiki/Single_UNIX_Specification#Mac_OS_X_and_Mac_OS_X...> It's fine to have a mac-pathname-convention-following place for such data, but please _also_ respect the UNIX-y version on the Mac. The only possible outcome of python on the Mac respect only Mac pathnames is to have automation scripts that work fine on BSD and Linux, but then break when you try to run them on a Mac. There is really no benefit to intentionally avoiding honoring the UNIX conventions. (For another example, note that although Python resides in /System/Library, on the mac, the thing that's in your $PATH when you're using a terminal is the symlink in /usr/bin/python.) Also, no, "~/Preferences" isn't the right place for it either; there's no such thing. You probably meant "~/Library/Preferences". I'd say that since ~/Library/Python is already used, there's no particular reason to add a new ~/Library/Preferences/Python location. After all, if you really care a lot about platform conventions, you should put it in ~/Library/Preferences/org.python.distutils.plist, but I don't see what benefit that extra complexity would have for anyone.

On 2 Aug, 2010, at 7:18, Glyph Lefkowitz wrote:
Storing files in unix location will be confusing to many Mac users, Apple has an explicitly documented convention for where to store files and dot-files in the user's home directory aren't part of that convention. An important reason for storing files in ~/Library/Python of ~/Library/Preferences/Python is that these locations are both logical for mac users and can be navigated to from the Finder without resorting to typing the folder name in "Go -> Go to Folder".
It's fine to have a mac-pathname-convention-following place for such data, but please _also_ respect the UNIX-y version on the Mac. The only possible outcome of python on the Mac respect only Mac pathnames is to have automation scripts that work fine on BSD and Linux, but then break when you try to run them on a Mac.
The stdlib should have APIs for locating common directories, although I must admit that I haven't checked if it actually does have them. That way you can write automation scripts that work anyware, not just on systems that look a lot like Linux. I've written a lot of scripts that had to follow platform conventions on a lot of unix platforms, and those tended to require small changes for every new unix flavor we encountered. Non-linux platforms also have filesystem hierarchy standards, even though they are often not as detailed as the Linux ones and most unix platforms don't have a user-base that is as vocal as the linux packagers when software doesn't follow the conventions.
There is really no benefit to intentionally avoiding honoring the UNIX conventions. (For another example, note that although Python resides in /System/Library, on the mac, the thing that's in your $PATH when you're using a terminal is the symlink in /usr/bin/python.)
Also, no, "~/Preferences" isn't the right place for it either; there's no such thing. You probably meant "~/Library/Preferences". I'd say that since ~/Library/Python is already used, there's no particular reason to add a new ~/Library/Preferences/Python location. After all, if you really care a lot about platform conventions, you should put it in ~/Library/Preferences/org.python.distutils.plist, but I don't see what benefit that extra complexity would have for anyone.
Your right, I meant ~/Library/Preferences, and I'd prefer ~/Library/Python for the reason you meant. The actual format of the configuration files is not prescribed in any way, and I'd by -1 on storing the preferences in a plist on OSX because that is making live for programmers actively harder. Ronald

On 02/08/2010 07:18, Ronald Oussoren wrote:
Really? As a Mac user I have never edited (or even looked at) files in ~/Library. I would never think of going there for finding config files to edit. However in my home directory I have: .Xauthority .Xcode . CFUserTextEncoding - (an Apple encoding configuration for Core Foundation) .bash_profile .cups .dropbox .dvdcss .filezilla .fontconfig .hgrc .idlerc .ipython .mono .netbeans .parallels_settings .pypirc .wingide3 Actually that is just a small selection of the .config files/directories in my home directory. It is certainly *a* standard location for config files on the Mac, including some Apple software (XCode) and Python applications. My preference would be to follow this established and well used convention. Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 02 Aug, 2010,at 11:48 AM, Michael Foord <fuzzyman@voidspace.org.uk> wrote: On 02/08/2010 07:18, Ronald Oussoren wrote: On 2 Aug, 2010, at 7:18, Glyph Lefkowitz wrote: On Aug 1, 2010, at 3:52 PM, Ronald Oussoren wrote: On 1 Aug, 2010, at 17:22, Éric Araujo wrote: Speaking of which... Your documentation says it's named ~/unittest.cfg, could you make this a file in the user base (that is, the prefix where 'setup.py install --user' will install files)? Putting .pydistutils.cfg .pypirc .unittest2.cfg .idlerc and possibly other in the user home directory (or %APPDATA% on win32 and what-have-you on Mac) is unnecessary clutter. However, $PYTHONUSERBASE is not the right directory for configuration files, as pointed in http://bugs.python.org/issue7175 It would be nice to agree on a ~/.python (resp. %APPADATA%/Python) or $XDG_CONFIG_HOME/python directory and put config files there. ~/Library/Python would be a good location on OSX, even if the 100% formally correct location would be ~/Preferences/Python (at least of framework builds, unix-style builds may want to follow the unix convention). "100% formally" speaking, MacOS behaves like UNIX in many ways. <http://en.wikipedia.org/wiki/Single_UNIX_Specification#Mac_OS_X_and_Mac_OS_X...> Storing files in unix location will be confusing to many Mac users, Apple has an explicitly documented convention for where to store files and dot-files in the user's home directory aren't part of that convention. An important reason for storing files in ~/Library/Python of ~/Library/Preferences/Python is that these locations are both logical for mac users and can be navigated to from the Finder without resorting to typing the folder name in "Go -> Go to Folder". Really? As a Mac user I have never edited (or even looked at) files in ~/Library. I would never think of going there for finding config files to edit. However in my home directory I have: .Xauthority .Xcode . CFUserTextEncoding - (an Apple encoding configuration for Core Foundation) .bash_profile .cups .dropbox .dvdcss .filezilla .fontconfig .hgrc .idlerc .ipython .mono .netbeans .parallels_settings .pypirc .wingide3 Actually that is just a small selection of the .config files/directories in my home directory. It is certainly *a* standard location for config files on the Mac, including some Apple software (XCode) and Python applications. The only apple one that is actually used is the .CFUserTextEncoding file, I have an .Xcode in my home as well but that is empty and last updated in 2007. AFAIK current versions of Xcode store preferences in ~/Library/Preferences. Most of the other ones are ports of unix tools and store junk in the standard unix location for storing configuration. Try edit one without resorting to the command-line, with a default configuration of the Finder you cannot even see these files (and that includes the File open dialog of tools like Text Edit). The reason you don't normally look in ~/Library/Preferences is that GUI tools on OSX have configuration screens for editing preferences and you don't have to edit them manually. My preference would be to follow this established and well used convention. My preference is still to use ~/Library/Python (or a subdirectory thereof) and filenames that don't start with a dot. Ronald

On 02/08/2010 11:48, Ronald Oussoren wrote:
Right, so what you are saying is that for user editable text config files ~/Library/Preferences is *not* a convention - and in fact the unix convention of dot files in the home directory is commonly used on the Mac. :-) Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 02/08/2010 11:48, Ronald Oussoren wrote:
The configuration files we are discussing are for command line tools - so I don't think that having to resort to the command line is a disadvantage at all. If users don't / can't use the command line then they *won't* want to edit these files anyway. If they are used to the command line then ~/.python32/distutils.cfg is going to be a very natural place for them. If we provide GUI tools that use these config files then we will also provide GUI tools that use these config files then we will also provide GUI tools to configure them - so I can't see a downside to having them in the unix location and no upside to putting them elsewhere. Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 02 Aug, 2010,at 01:00 PM, Michael Foord <fuzzyman@voidspace.org.uk> wrote:
The configuration files we are discussing are for command line tools - so I don't think that having to resort to the command line is a disadvantage at all. If users don't / can't use the command line then they *won't* want to edit these files anyway. Not being comfortable at the command-line is not the same as not wanting to edit the global configuration of unittest or distutils. Anyway, does that mean that the configuration should move if I create a patch for IDLE that allows you the manage the unittest configuration through a GUI? If they are used to the command line then ~/.python32/distutils.cfg is going to be a very natural place for them. That location isn't natural for me. If we invent a new location for python-related configuration file we might as well do it properly and follow platform conventions. The MacOSX conventions are described here <http://developer.apple.com/mac/library/documentation/MacOSX/Conceptual/BPFil...>. Macosx-is-not-a-crappy-linux-ly yours, Ronald

On 02/08/2010 14:34, Ronald Oussoren wrote:
Nope, as I doubt for *most* users it will be the primary way of editing the file.
But you yourself said that it *isn't* normal to have files we expect the user to edit in the location you suggested - and a glance on my system is that use ~/.app files is a *very* common convention on the Mac. Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 02/08/2010 14:34, Ronald Oussoren wrote:
But both of those are primarily command line tools. A basic ability to use the command line is a prerequisite of wanting to configure them. I would be interested in hearing from other Mac users as to where they would look for configuration files for command line tools - in ~ or in ~/Library/Preferences? Michael
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

Michael Foord <fuzzyman@voidspace.org.uk> writes:
My primary personal machine has been OSX for years now, and as someone who lives in the command line (well, or Emacs shells), I think either approach is workable for command line only tools. I will say that if I need to find preferences or application-specific files my first inclination is absolutely to look under ~/Library. That's the "platform" location (just as looking for dot files under Linux is my first choice, or "Documents and Settings/<user>" for Windows). I don't think I shift mental gears automatically for command line tools versus not, unless I have some prior knowledge about the tool. With that said, given all the third party and Unix-oriented stuff I install under OSX, it's hardly rare (as has been pointed out elsewhere) to be working with ~/.<something> either, so it's not like I'd consider it that unusual to find that's where I need to go. In glancing at my current system, it does appear command line only tools are more commonly using ~/.<something> files rather than under ~/Library (which tends to be stuff packaged up as an application in /Applications, even if they can also be run from the command line). Though it might be a biased sample set since I'm more likely to have brought in command line tools to OSX from the Unix side of things, I suspect that's true of other users of command line tools as well. I will say that it's rarer to find a native (Cocoa/Carbon) GUI application that doesn't store preferences or application settings beneath ~/Library, and in such a case I'd feel they were more "wrong" and non-conforming if they didn't do that. So it depends on how "native" an application wishes to be perceived. I guess in thinking about it while writing this, having something installed in /Applications is more strongly linked with ~/Library in my mind than other tools. Of course, even with /Applications, non-native GUI apps are more of a mixed bag. For example, the X versions of Gimp and Inkscape - Gimp properly uses "~/Library/Application Support" while Inkscape still uses ~/.inkscape. Of course, as X apps, neither truly feels native or conforming anyway. So that probably helps make things as clear as mud :-) -- David

(Ronald, the text version of your message was very difficult to sort through and read, because all of the quoting information was lost. Just thought you'd want to know.) What I hear Glyph saying is that we should support looking in *both* locations for configuration info on OSX, and I don't see a downside to that. Most unix applications look in multiple places for configuration info. Michael seems to be arguing for not using the standard OSX locations because the Finder can't edit them anyway. Is that true? -- R. David Murray www.bitdance.com

On 02/08/2010 15:24, R. David Murray wrote:
That adds extra complexity to the implementation of the configuration system. If it uses the first one it finds which order does it look in, if we have tools that create the file which location does it create it in and so on.
Michael seems to be arguing for not using the standard OSX locations because the Finder can't edit them anyway. Is that true?
I am saying that Ronald's suggestion is *not* a natural location for user editable configuration files - as far as I can tell I have no user editable files there and plenty in ~/.something. Ronald himself said that the location he is specifying is the standard location for configuration / preference files created and used by *gui* tools, and files in that location are not usually intended to be user editable. I don't believe a Mac user basically competent at the command line is *likely* to go looking in the gui preferences location for these config files and I think they would easily find them in ~. This is backed up the number of existing programs that use this convention on Mac OS X. It *is* a widely used convention on Mac OS X to use ~/.appname for configuration files. Applications like mercurial use this location on the Mac for example. Ronald was wrong when he said that the only configuration file in ~ used by the Mac itself is .CFUserTextEncoding. Terminal (the Mac OS X command line) has a user editable config file which it stores in ~. The issue with the finder is that by default . files and directories aren't shown and so they wouldn't be editable from the finder. As basic willingness to use the command line is a prerequisite for *wanting* to edit these files I don't see this as an issue. A user unfamiliar with the command line is not likely to guess the correct location for these files if we put them elsewhere, so they are going to have to refer to some documentation anyway. Michael
-- R. David Murray www.bitdance.com
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

On 2 Aug, 2010, at 16:24, R. David Murray wrote:
I'll stop using the mobile-me webmail client for lists, it seems to mess things up.
A lot of tools seem to look both in a system location and a per user location (such as /etc/profile and ~/.profile). OSX ads a 3th level to that, although I have never used that myself (technically there are 4 levels, but that isn't important unless you are Apple).
Michael seems to be arguing for not using the standard OSX locations because the Finder can't edit them anyway. Is that true?
The Finder can open OSX locations just fine, but you cannot see dot-files (or -directories) in the Finder. I won't argue this anymore, at least not this week. I'll work around the issue in my private tree if that's what's needed. Ronald

Glyph Lefkowitz wrote:
I'd say that since ~/Library/Python is already used, there's no particular reason to add a new ~/Library/Preferences/Python location.
I think the reason for separating out Preferences is so that you can install a new version of a library or application without losing the user's preferences from the previous version. -- Greg

On 31/07/2010 17:22, Tarek Ziadé wrote:
Ok. It would be helpful for unittest2 (the backport) if it was *still* available in distutils2 even after the merge into pkgutil (for use by earlier versions of Python). I guess you will do this anyway for the distutils2 backport itself anyway... (?) All the best, Michael Foord
Regards Tarek
-- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (”BOGUS AGREEMENTS”) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer.

Yes. Even if the goal is to have distutils2 in the stdlib for 3.2 or 3.3, there will still be a standalone release on PyPI for Python 2.4-3.1. You’ll just have to write such compat code: try: from pkgutil import shiny_new_function except ImportError: from distutils2._backport.pkgutil import shiny_new_function Regards
participants (17)
-
Antoine Pitrou
-
Barry Warsaw
-
David Bolen
-
David Cournapeau
-
Glyph Lefkowitz
-
Greg Ewing
-
Marty Alchin
-
Matthieu Brucher
-
Michael Foord
-
Ned Deily
-
Nick Coghlan
-
P.J. Eby
-
R. David Murray
-
Raymond Hettinger
-
Ronald Oussoren
-
Tarek Ziadé
-
Éric Araujo