Mailman 3 December 2012 - Python-ideas

Dynamic code NOPing
by anatoly techtonik 31 Dec '12

31 Dec '12

For the logging module it will be extremely useful if Python included a way to disactivate processing certain blocks to avoid making sacrifices between extensive logging harness and performance. For example, instead of writing: if log.DEBUG==True: log(factorial(2**15)) It should be possible to just write: log(factorial(2**15)) if if log() is an instance of some Nopable class, the statement in log's braces is not executed. -- anatoly t.

15 20

Re: [Python-ideas] Order in the documentation search results
by Hernan Grecco 31 Dec '12

31 Dec '12

Hi Ned, On Sun, Dec 30, 2012 at 7:11 PM, Ned Batchelder <ned(a)nedbatchelder.com> wrote: > > While we're on the topic, why in this day and age do we have a custom > search? Using google site search would be faster for the user, and more > accurate. > > --Ned. In general I agree with you, but I find downloadable documentation very useful (one of the many reasons that I like sphinx). Keeping the search engine in that case is very convenient. Hernan

1 0

Re: [Python-ideas] [Python-Dev] question about packaging
by Michele Lacchia 29 Dec '12

29 Dec '12

Sorry if I interfere, but now what should be supported, distlib or packaging? It seems to me that the former was born to solve some problems packaging and distribute still had. In addition to that, packaging has not been included in Python 3.3, as it was first planned.

2 1

PEP 432: Simplifying the CPython startup sequence
by Nick Coghlan 29 Dec '12

29 Dec '12

After helping Brett with the migration to importlib in 3.3, and looking at some of the ideas kicking around for additional CPython features that would affect the startup sequence, I've come to the conclusion that what we have now simply isn't sustainable long term. It's already the case that if you use certain options (specifically -W or -X), the interpreter will start accessing the C API before it has called Py_Initialize(). It's not cool when other people do that (we'd never accept code that behaved that way as a valid reproducer for a bug report), and it's *definitely* not cool that we're doing it (even though we seem to be getting away with it for the moment, and have been for a long time). The attached PEP is a first attempt at a plan for doing something about it. (My notes at http://wiki.python.org/moin/CPythonInterpreterInitialization provide additional context - let me know if you think there's more material on that page that should be in the PEP itself) The PEP is also available online at http://www.python.org/dev/peps/pep-0432/ Cheers, Nick. PEP: 432 Title: Simplifying the CPython startup sequence Version: $Revision$ Last-Modified: $Date$ Author: Nick Coghlan <ncoghlan(a)gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Dec-2012 Python-Version: 3.4 Post-History: 28-Dec-2012 Abstract ======== This PEP proposes a mechanism for simplifying the startup sequence for CPython, making it easier to modify the initialisation behaviour of the reference interpreter executable, as well as making it easier to control CPython's startup behaviour when creating an alternate executable or embedding it as a Python execution engine inside a larger application. Proposal Summary ================ This PEP proposes that CPython move to an explicit 2-phase initialisation process, where a preliminary interpreter is put in place with limited OS interaction capabilities early in the startup sequence. This essential core remains in place while all of the configuration settings are determined, until a final configuration call takes those settings and finishes bootstrapping the interpreter immediately before executing the main module. As a concrete use case to help guide any design changes, and to solve a known problem where the appropriate defaults for system utilities differ from those for running user scripts, this PEP also proposes the creation and distribution of a separate system Python (``spython``) executable which, by default, ignores user site directories and environment variables, and does not implicitly set ``sys.path[0]`` based on the current directory or the script being executed. Background ========== Over time, CPython's initialisation sequence has become progressively more complicated, offering more options, as well as performing more complex tasks (such as configuring the Unicode settings for OS interfaces in Python 3 as well as bootstrapping a pure Python implementation of the import system). Much of this complexity is accessible only through the ``Py_Main`` and ``Py_Initialize`` APIs, offering embedding applications little opportunity for customisation. This creeping complexity also makes life difficult for maintainers, as much of the configuration needs to take place prior to the ``Py_Initialize`` call, meaning much of the Python C API cannot be used safely. A number of proposals are on the table for even *more* sophisticated startup behaviour, such as better control over ``sys.path`` initialisation (easily adding additional directories on the command line in a cross-platform fashion, as well as controlling the configuration of ``sys.path[0]``), easier configuration of utilities like coverage tracing when launching Python subprocesses, and easier control of the encoding used for the standard IO streams when embedding CPython in a larger application. Rather than attempting to bolt such behaviour onto an already complicated system, this PEP proposes to instead simplify the status quo *first*, with the aim of making these further feature requests easier to implement. Key Concerns ============ There are a couple of key concerns that any change to the startup sequence needs to take into account. Maintainability --------------- The current CPython startup sequence is difficult to understand, and even more difficult to modify. It is not clear what state the interpreter is in while much of the initialisation code executes, leading to behaviour such as lists, dictionaries and Unicode values being created prior to the call to ``Py_Initialize`` when the ``-X`` or ``-W`` options are used [1_]. By moving to a 2-phase startup sequence, developers should only need to understand which features are not available in the core bootstrapping state, as the vast majority of the configuration process will now take place in that state. By basing the new design on a combination of C structures and Python dictionaries, it should also be easier to modify the system in the future to add new configuration options. Performance ----------- CPython is used heavily to run short scripts where the runtime is dominated by the interpreter initialisation time. Any changes to the startup sequence should minimise their impact on the startup overhead. (Given that the overhead is dominated by IO operations, this is not currently expected to cause any significant problems). The Status Quo ============== Much of the configuration of CPython is currently handled through C level global variables:: Py_IgnoreEnvironmentFlag Py_HashRandomizationFlag _Py_HashSecretInitialized _Py_HashSecret Py_BytesWarningFlag Py_DebugFlag Py_InspectFlag Py_InteractiveFlag Py_OptimizeFlag Py_DontWriteBytecodeFlag Py_NoUserSiteDirectory Py_NoSiteFlag Py_UnbufferedStdioFlag Py_VerboseFlag For the above variables, the conversion of command line options and environment variables to C global variables is handled by ``Py_Main``, so each embedding application must set those appropriately in order to change them from their defaults. Some configuration can only be provided as OS level environment variables:: PYTHONHASHSEED PYTHONSTARTUP PYTHONPATH PYTHONHOME PYTHONCASEOK PYTHONIOENCODING Additional configuration is handled via separate API calls:: Py_SetProgramName() (call before Py_Initialize()) Py_SetPath() (optional, call before Py_Initialize()) Py_SetPythonHome() (optional, call before Py_Initialize()???) Py_SetArgv[Ex]() (call after Py_Initialize()) The ``Py_InitializeEx()`` API also accepts a boolean flag to indicate whether or not CPython's signal handlers should be installed. Finally, some interactive behaviour (such as printing the introductory banner) is triggered only when standard input is reported as a terminal connection by the operating system. Also see more detailed notes at [1_] Proposal ======== (Note: details here are still very much in flux, but preliminary feedback is appreciated anyway) Core Interpreter Initialisation ------------------------------- The only configuration that currently absolutely needs to be in place before even the interpreter core can be initialised is the seed for the randomised hash algorithm. However, there are a couple of settings needed there: whether or not hash randomisation is enabled at all, and if it's enabled, whether or not to use a specific seed value. The proposed API for this step in the startup sequence is:: void Py_BeginInitialization(Py_CoreConfig *config); Like Py_Initialize, this part of the new API treats initialisation failures as fatal errors. While that's still not particularly embedding friendly, the operations in this step *really* shouldn't be failing, and changing them to return error codes instead of aborting would be an even larger task than the one already being proposed. The new Py_CoreConfig struct holds the settings required for preliminary configuration:: typedef struct { int use_hash_seed; size_t hash_seed; } Py_CoreConfig; To "disable" hash randomisation, set "use_hash_seed" and pass a hash seed of zero. (This seems reasonable to me, but there may be security implications I'm overlooking. If so, adding a separate flag or switching to a 3-valued "no randomisation", "fixed hash seed" and "randomised hash" option is easy) The core configuration settings pointer may be NULL, in which case the default behaviour of randomised hashes with a random seed will be used. A new query API will allow code to determine if the interpreter is in the bootstrapping state between core initialisation and the completion of the initialisation process:: int Py_IsInitializing(); While in the initialising state, the interpreter should be fully functional except that: * compilation is not allowed (as the parser is not yet configured properly) * The following attributes in the ``sys`` module are all either missing or ``None``: * ``sys.path`` * ``sys.argv`` * ``sys.executable`` * ``sys.base_exec_prefix`` * ``sys.base_prefix`` * ``sys.exec_prefix`` * ``sys.prefix`` * ``sys.warnoptions`` * ``sys.flags`` * ``sys.dont_write_bytecode`` * ``sys.stdin`` * ``sys.stdout`` * The filesystem encoding is not yet defined * The IO encoding is not yet defined * CPython signal handlers are not yet installed * only builtin and frozen modules may be imported (due to above limitations) * ``sys.stderr`` is set to a temporary IO object using unbuffered binary mode * The ``warnings`` module is not yet initialised * The ``__main__`` module does not yet exist <TBD: identify any other notable missing functionality> The main things made available by this step will be the core Python datatypes, in particular dictionaries, lists and strings. This allows them to be used safely for all of the remaining configuration steps (unlike the status quo). In addition, the current thread will possess a valid Python thread state, allow any further configuration data to be stored on the interpreter object rather than in C process globals. Any call to Py_BeginInitialization() must have a matching call to Py_Finalize(). It is acceptable to skip calling Py_EndInitialization() in between (e.g. if attempting to read the configuration settings fails) Determining the remaining configuration settings ------------------------------------------------ The next step in the initialisation sequence is to determine the full settings needed to complete the process. No changes are made to the interpreter state at this point. The core API for this step is:: int Py_ReadConfiguration(PyObject *config); The config argument should be a pointer to a Python dictionary. For any supported configuration setting already in the dictionary, CPython will sanity check the supplied value, but otherwise accept it as correct. Unlike Py_Initialize and Py_BeginInitialization, this call will raise an exception and report an error return rather than exhibiting fatal errors if a problem is found with the config data. Any supported configuration setting which is not already set will be populated appropriately. The default configuration can be overridden entirely by setting the value *before* calling Py_ReadConfiguration. The provided value will then also be used in calculating any settings derived from that value. Alternatively, settings may be overridden *after* the Py_ReadConfiguration call (this can be useful if an embedding application wants to adjust a setting rather than replace it completely, such as removing ``sys.path[0]``). Supported configuration settings -------------------------------- At least the following configuration settings will be supported:: raw_argv (list of str, default = retrieved from OS APIs) argv (list of str, default = derived from raw_argv) warnoptions (list of str, default = derived from raw_argv and environment) xoptions (list of str, default = derived from raw_argv and environment) program_name (str, default = retrieved from OS APIs) executable (str, default = derived from program_name) home (str, default = complicated!) prefix (str, default = complicated!) exec_prefix (str, default = complicated!) base_prefix (str, default = complicated!) base_exec_prefix (str, default = complicated!) path (list of str, default = complicated!) io_encoding (str, default = derived from environment or OS APIs) fs_encoding (str, default = derived from OS APIs) skip_signal_handlers (boolean, default = derived from environment or False) ignore_environment (boolean, default = derived from environment or False) dont_write_bytecode (boolean, default = derived from environment or False) no_site (boolean, default = derived from environment or False) no_user_site (boolean, default = derived from environment or False) <TBD: at least more from sys.flags need to go here> Completing the interpreter initialisation ----------------------------------------- The final step in the process is to actually put the configuration settings into effect and finish bootstrapping the interpreter up to full operation:: int Py_EndInitialization(PyObject *config); Like Py_ReadConfiguration, this call will raise an exception and report an error return rather than exhibiting fatal errors if a problem is found with the config data. After a successful call, Py_IsInitializing() will be false, while Py_IsInitialized() will become true. The caveats described above for the interpreter during the initialisation phase will no longer hold. Stable ABI ---------- All of the APIs proposed in this PEP are excluded from the stable ABI, as embedding a Python interpreter involves a much higher degree of coupling than merely writing an extension. Backwards Compatibility ----------------------- Backwards compatibility will be preserved primarily by ensuring that Py_ReadConfiguration() interrogates all the previously defined configuration settings stored in global variables and environment variables. One acknowledged incompatiblity is that some environment variables which are currently read lazily may instead be read once during interpreter initialisation. As the PEP matures, these will be discussed in more detail on a case by case basis. The Py_Initialize() style of initialisation will continue to be supported. It will use the new API internally, but will continue to exhibit the same behaviour as it does today, ensuring that sys.argv is not set until a subsequent PySys_SetArgv call. A System Python Executable ========================== When executing system utilities with administrative access to a system, many of the default behaviours of CPython are undesirable, as they may allow untrusted code to execute with elevated privileges. The most problematic aspects are the fact that user site directories are enabled, environment variables are trusted and that the directory containing the executed file is placed at the beginning of the import path. Currently, providing a separate executable with different default behaviour would be prohibitively hard to maintain. One of the goals of this PEP is to make it possible to replace much of the hard to maintain bootstrapping code with more normal CPython code, as well as making it easier for a separate application to make use of key components of ``Py_Main``. Including this change in the PEP is designed to help avoid acceptance of a design that sounds good in theory but proves to be problematic in practice. One final aspect not addressed by the general embedding changes above is the current inaccessibility of the core logic for deciding between the different execution modes supported by CPython: * script execution * directory/zipfile execution * command execution ("-c" switch) * module or package execution ("-m" switch) * execution from stdin (non-interactive) * interactive stdin <TBD: concrete proposal for better exposing the __main__ execution step> Implementation ============== None as yet. Once I have a reasonably solid plan of attack, I intend to work on a reference implementation as a feature branch in my BitBucket sandbox [2_] References ========== .. [1] CPython interpreter initialization notes (http://wiki.python.org/moin/CPythonInterpreterInitialization) .. [2] BitBucket Sandbox (https://bitbucket.org/ncoghlan/cpython_sandbox) Copyright =========== This document has been placed in the public domain. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

7 12

Allow deleting slice in an OrderedDict
by Ram Rachum 27 Dec '12

27 Dec '12

When I have an OrderedDict, I want to be able to delete a slice of it. I want to be able to do: del ordered_dict[:3] To delete the first 3 items, like I would do in a list. Is there any reason why this shouldn't be implemented? Thanks, Ram.

8 7

collections.sortedset proposal
by Paul Colomiets 27 Dec '12

27 Dec '12

Hi, I want to propose to include SortedSet data structure into collections module. SortedSet (name borrowed from Redis) is a basically a mapping of (unique) keys to scores, that allows fast slicing by ordinal number and by score. There are plenty of use cases for the sorted sets: * Leaderboard for a game * Priority queue (that supports task deletion) * Timer list (e.g. can be used for tulip, supports deletion too) * Caches with TTL-based, LFU or LRU eviction (including `functools.lru_cache`) * Search databases with relevance scores * Statistics (many use cases) * Replacement for `collections.Counter` with faster `most_common()` I have first draft of pure python implementation: https://github.com/tailhook/sortedsets http://pypi.python.org/pypi/sortedsets/1.0 The implementation is closely modeled on Redis. Internally it consists of a dict for mapping between keys and scores, and a skiplist for scores. So most operations are done with O(log n) time. The actual performance is probably very slow for pure-python implementation, but can be fixed by C code later. The asymptotic performance seems to be OK. So my questions are: 1. Do you think SortedSets are eligible for inclusion to stdlib? 2. Do I need a PEP? 3. Any comments on the implementation? P.S.: Sorted sets in redis are not the same thing as sorted sets in blist. So maybe a better name? -- Paul

5 8

Add support keyword arguments with suitable defaults for OSError and subclasses
by Serhiy Storchaka 26 Dec '12

26 Dec '12

Now OSError constructor does not support keyword arguments. It will be good add support for followed keyword arguments: "errno", "strerror", "filename". If "strerror" is not specified, a standard error message corresponding to errno is used. If "errno" is not specified for an OSError subclass, an errno associated with this subclass is used (if only one errno associated). For backward compatibility perhaps keyword arguments should be incompatible with any positional arguments (or at least suitable defaults should used only if any keyword argument specified). Examples: >>> OSError(errno=errno.ENOENT) FileNotFoundError(2, 'No such file or directory') >>> FileNotFoundError(filename='qwerty') FileNotFoundError(2, 'No such file or directory') >>> FileNotFoundError(strerr='Bad file') FileNotFoundError(2, 'Bad file')

1 0

Re: [Python-ideas] Allow accessing return value inside finally clause
by alex23 26 Dec '12

26 Dec '12

On Dec 26, 7:55 pm, Ram Rachum <r...(a)rachum.com> wrote: > Alex: I'm getting the feeling that you misunderstand what I'm proposing > here. I'm proposing that the return value will be accessible in the > `finally` clause. In a similar (if shorter) way that the exception info is > available by using `sys.exc_info()`. I get what you're saying. What you haven't shown is how introducing 'return value' semantics to try/finally blocks does anything other than make them more confusing to people. Functions have return values. Decorators wrap functions and can thus be used to pre- or post-process the in/outputs for the function. This is clearly defined and a well known approach to your problem. The onus is on you to show how turning try/finally into a kitchen-sink of behaviour will improve the language, preferably without recourse to what you "think" or "feel". A concrete use-case would help here, but I'm 100% convinced that whatever you come up with, there'll be a better solution using decorators that works right now.

1 0

Re: [Python-ideas] Allow accessing return value inside finally clause
by alex23 26 Dec '12

26 Dec '12

On 26 Dec, 10:44, Ram Rachum <r...(a)rachum.com> wrote: > I don't think that this makes Python less clear; How can you possibly say this? You've changed the `finally` clause from _guaranteed_ execution to something utterly inconsistent. In fact, finally blocks would need to have _more_ code to guard against all of the different execution models you're proposing here. I'm not sure why you think forcing me to write more & less obvious code in a finally block is a better trade off than you making clear, explicit use of decorators. Ambiguity does not equate to clarity. Less typing doesn't either. Creating small re-usable pieces of code that do the "hard work" for you, however, _is a lot more clear_. > I think it's just another > minor feature that might be useful for some people, and for people who > don't, it won't matter. How many people use the `for..else` feature, for > example? Very, very few people do. I've used it only several times. But > it's still part of Python because it helps in a few rare cases, so that > makes it worth it *despite* the fact that it might confuse a newbie. The behaviour of `for..else` doesn't change based on arbitrary conditions, whereas what you propose is that the finally blocks behaviour is _fundamentally_ different depending on whether the try block is fully executed or not, whether an exception is raised or not. This is absolutely not the same thing, and trying to pass this concern off as "confusing to newbies" is rather disingenuous. The behaviour would be _confusing to everybody_. This is not a valid cost to save you from having to type a few more keystrokes to decorate the return value.

2 1

Re: [Python-ideas] Allow accessing return value inside finally clause
by alex23 26 Dec '12

26 Dec '12

On 26 Dec, 09:11, Ram Rachum <r...(a)rachum.com> wrote: > That works, sure. I've mentioned this in my email above. But I think that > in some cases making the post-process in the `finally` clause will be more > elegant. What you deem "elegant", I see as "laziness".

1 0