Mailman 3 March 2012 - Python-ideas

Exact integral types in struct
by Serhiy Storchaka March 22, 2012

March 22, 2012

The struct module works only with natural platform-specific integers. As a result, a lot of code in the standard library and third-party applications are forced either to rely on unreliable assumptions (short is always 2-byte, long is always 4-byte), which is not always true, either explicitly construct the integer from bytes (b[0]+(b[1]<<8)+(b[2]<<16)+...). I propose to introduce a special notation formats for signed and unsigned integers of arbitrary exact size (which is given a number preceded by the prefix). After that, eliminate the use of platform-specific formats when working with platform-independent data (such as zip, for example). Or maybe I'm behind, and the corresponding functions already exist, and the use of the struct module is only remnants?

8 14

Allow function __globals__ to be arbitrary mapping not just dict
by Steven D'Aprano March 22, 2012

March 22, 2012

Currently, if you try to construct a function from parts, the mapping that becomes func.__globals__ must be an actual dict: py> class Mapping: ... def __getitem__(self, key): ... if key == 'y': ... return 42 ... raise KeyError(key) ... py> from types import FunctionType py> f = lambda x: x + y py> g = FunctionType(f.__code__, Mapping(), 'g') Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: function() argument 2 must be dict, not Mapping I propose to allow function.__globals__ to accept any mapping type. That, plus the new collections.ChainMap class in Python 3.3, would allow some interesting experiments with namespaces and scoping rules. E.g. if I want to write a function with a custom namespace, I have to do something like this: ns = ChainMap( ... ) # set up a namespace def func(a, ns=ns): x = a + ns['b'] y = ns['some_func'](ns['c']) z = ns['another_func'](x, y) ns['d'] = (x, y, z) return ns['one_last_thing'](d) which is not a very natural way of writing code. But if we could use non-dict mappings as __globals__, I could write that function like this: ns = ChainMap( ... ) # set up a namespace def func(a): global d x = a + b y = some_func(c) z = another_func(x, y) d = (x, y, z) return one_last_thing(d) # This could be a decorator. func = FunctionType(func.__code__, ns, func.__name__) (By the way, ChainMap is only one possible example namespace.) -- Steven

7 9

Logging2 with default NullHandler
by anatoly techtonik March 19, 2012

March 19, 2012

Badly need `logging2` module that has NullHandler assigned by default for all loggers. http://packages.python.org/Logbook/api/handlers.html#logbook.NullHandler Why? Because logging fails to play well with libraries: import logging log = logging.getLogger(__name__) log.warn("WARN") No handlers could be found for logger "spyderlib.utils.bsdsocket" What do I want from library logging as a Python application developer? Nothing until I explicitly setup default behaviour. logging can not be changed, and leaving everything as-is is a PITA, that's why I am proposing for logging2 as the only viable solution. -- anatoly t.

6 15

Combining stat/lstat/fstatat etc.
by Serhiy Storchaka March 16, 2012

March 16, 2012

In Python3 added a large number of "at"-functions from the latter Posix standard. Such a large number of names only litters namespace. In C it is necessary for historical reasons (note that fstatat combines the stat and lstat and includes the possibility of extension) and because of static typing. But in Python because of dynamic typing we can extend existing functions. So instead of stat, lstat, and fstatat could use the same function `stat(path, *, dirfd=None, followlinks=True)`. Then `lstat(path)` == `stat(path, followlinks=False)`, and `fstatat(dirfd, path, flags)` == `stat(path, dirfd=(AT_FDCWD if dirfd is None else AT_FDCWD), followlinks=not (flags & AT_SYMLINK_NOFOLLOW))`. As special value for dirfd I suggest using the None, not AT_FDCWD (it's more pythonish). fstat could be included too, by specifically treating the case where the path is integer. And same for other functions. Old lstat and lchown will remain for compatibility as light wrappers around the stat, with time they may become deprecated.

2 2

set.add could return True or False
by Matt Joiner March 16, 2012

March 16, 2012

set.add(x) could return True if x was added to the set, and False if x was already in the set. Adding an element that is already present often constitutes an error in my code. As I understand, set.add is an atomic operation. Having set.add return a boolean will also allow EAFP-style code with regard to handling duplicates, the long winded form of which is currently: if a not in b: b.add(a) <-- race condition do_c() Which can be improved to: if b.add(a): do_c() Advantages: * Very common code pattern. * More concise. * Allows interpreter atomicity to be exploited, often removing the need for additional locking. * Faster because it avoids double contain check, and can avoid locking.

8 13

Save memory when forking with *really* immutable objects
by Larry Hastings March 13, 2012

March 13, 2012

I'll admit in advance that this is in all likelyhood a terrible idea. What I'm curious about is why it wouldn't work, or if it wouldn't even help ;-) One problem for CPython is that it can't share data across processes very often. If you have an application server, and you fork a hundred processes to handle requests, your memory use will be "C * n * p" where C is a constant, n is the number of processes, and p is the average memory consumption of your app. I fear C is very nearly near 1.0. Most of Python's memory usage is on the heap, and Python uses its memory to store objects, and objects are reference counted, and reference counts change. So all the COW data pages get written to sooner or later. The obvious first step: add a magical reference count number that never changes, called Py_REF_ETERNAL. I added this to CPython trunk with a quick hack. It seemed to work; I threw in some asserts to test it, which passed, and it was passing the unit test suite. I discussed this with Martin who (as usual) made some excellent points. Martin suggests that this wouldn't help unless we could concentrate the Py_REF_ETERNAL objects in their own memory pools in the small block allocator. Otherwise we'd never get a page that didn't get written to sooner or later. Obviously all interned strings could get Py_REF_ETERNAL. A slightly more controversial idea: mark code objects (but only those that get unmarshaled, says Martin!) as Py_REF_ETERNAL too. Yes, you can unload code from sys.modules, but in practice if you ever import something you never throw it away for the life of the process. If we went this route we could probably mark most (all?) immutable objects that get unmarshaled with Py_REF_ETERNAL. Martin's statistics from writing the flexible string representation says that for a toy Django app, memory consumption is mostly strings, and most strings are short (< 16 or even < 8 bytes)... in other words, identifiers. So if you ran 100 toy Django instances it seems likely this would help! And no I haven't benchmarked it, /arry

1 0

Saving state or data from Python objects (Was: Serializable method)
by anatoly techtonik March 10, 2012

March 10, 2012

On Sat, Mar 10, 2012 at 2:59 PM, Masklinn <masklinn(a)masklinn.net> wrote: > but it still ends > up with the format's semantics not necessarily mapping 1:1 to Python > semantics This is the right problem. If we take user approach to see what is the hard time users have - there are two different user stories: 1. Is about saving current state of a program by magically taking all object space (or a portion of) and save it in into a file Goals: Don't make me think Security: Not important (well, you're saving your program - what did you expect if everybody can alter it?) Concerns: Independence mismatch - what if source code for the saved state changes? - this needs further experiments Status: pickle or marshal seems to be designed just for this purpose, but does they provide an easy (one shot) solution for this scenario? 2. Is about saving data stored in Python objects into an some format Goals: Transform data into the format suitable for external requirement Goals: Exchange/alter data outside of Python process Goals: Make it easy, transparent and automatic (note no 'g' here) (May I call this Data Transformation Theory?) Basically, what we need it is to transform data. This requires some assumptions: 1. Transformation can be symmetrical (lossless) and non-symmetrical (lossy) 2. Non-symmetrical transformation can be made symmetrical by providing additional data about what is missed Now about Python serialization: 1. The problem is complex. 2. Up to the point that it becomes complicated 3. So there should be a systematic approach Systematic approach: 1. Define the scope of serialization (data in Python objects are fields and their values) 2. Define output format (JSON - for example) 3. Create mapping 3.1 Make sure lossy transformations are marked 3.2 Make sure 'additional data' to make them symmetrical is analysed 4. Create transformation rules 5. Alter rules to warn users (or raise) when transformation fails with explanation why 6. Summarize fundamental problems when transformations are not possible in documentation So, with such Data Transformation Framework it doesn't matter which is the target format. You can choose any. You can create your rules and test your particular chain of objects against default rules to see if transformation is possible, and if not - know why exactly. Full control -> confidence -> better tools and problem descriptions. Of course, such framework doesn't exist yet. ;) -- anatoly t.

1 0

Serializable method
by Jakob Bowyer March 10, 2012

March 10, 2012

So originally I presented this idea for serialize methods something like to_json() but the idea was rightly shot down where the language would have to support most formats and this could lead to confusion or complication. So this is a newer version of that idea. I think that object should provide an __serializable__ method which in-turn allows the user to define in it how the object is to be serialized, the default operation should be something along the lines of return self.__dict__ but that is just semantics. The idea with an object that offers a serializable method means that the object can be passed directly to any formater in python that offers a .dump method and the object is immediately formatted how the end user wants the data to be, without needing to write a middle layer formatter for this object. Is this another terrible idea from me? Or is there some ground in this?

8 12

Loop labels
by Matt Joiner March 10, 2012

March 10, 2012

Several languages allow break and continue statements to take a count or loop label for greater flexibility. Dealing with nested loops and control flow around them is definitely something I and probably most programmers deal with everyday. Generally for very complex control flow one might employ functions, and use return statements to work around any shortcomings. This is not always ideal in CPython because of the performance cost of function calls, and lack of anonymous functions. The other work around is usually goto statements, which clearly aren't available or appropriate in Python. So what about the following extensions? Allow for and while statements to be labelled using "as". Allow break and continue to take the name of a containing loop, or an integer. The corresponding named loop, or the nth containing loop are treated as though they are the nearest enclosing loop. loop_label ::= identifier break_stmt ::= "break" [decimalinteger | loop_label] continue_stmt ::= "continue" [decimalinteger | loop_label] while_stmt ::= "while" expression ["as" loop_label] ":" suite ["else" ":" suite] for_stmt ::= "for" target_list "in" expression_list ["as" loop_label] ":" suite ["else" ":" suite] Here's a really naive example: for a in b as c: for d in e: break c # or break 2 for f in g: continue c # or continue 2

8 13

import fallback syntax
by Chris Withers March 8, 2012

March 8, 2012

Hi All, I see a lot of Python like this: try: from cDecimal import Decimal except ImportError: from decimal import Decimal ...and nest deeper if you're talking about ElementTree. How about some syntactical sugar to make this less obnoxious: from cDecimal or decimal import Decimal from x.y.z import X as X or a.b.z import Y as X or what do people think? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk

8 9