Mailman 3 - Python-ideas

simplification pep-like thing
by tomer filiba Jan. 29, 2007

Jan. 29, 2007

this text needs more thought and rephrasing, but i think that it covers all the details. -------------------------------------------------------------------------- Abstract ========= The simplification API provides a mechanism to get the state of an object, as well as reconstructing an object with a given state. This new API would affect pickle, copy_reg, and copy modules, as well as the writers of serializers. The idea is to separate the state-gathering from the encoding: * State-gathering … [View More]is getting the state of the object * Encoding is the converting the state (which is an object) into a sequence of bytes (textual or binary) This is somewhat similar to the ISerializable interface of .NET, which defines only GetObjectData(); the actual encoding is done by a Formatter class, which has different implementations for SOAP-formatting, binary-formatting, and other formats. Motivation ========== There are many generic and niche serializers out there, including pickle, banana/jelly, Cerealizer, brine, and lots of serializers targeting XML output. Currently, all serializers have their own methods of retrieving the contents, or state of the object. This API attempts to solve this issue by providing standard means for getting object state, and creating objects with their state restored. Another issue is making the serialization process "proxy-friendly". Many frameworks use object proxies to indirectly refer to another object (for instance, RPC proxies, FFI, etc.). In this case, it's desirable to simplify the referenced object rather than the proxy, and this API addresses this issue too. Simplification ============== Simplification is the process of converting a 'complex' object into its "atomic" components. You may think of these atomic components as the *contents* of the object. This proposal does not state what "atomic" means -- this is open to the decision of the class. The only restriction imposed is, collections of any kind must be simplified as tuples. Moreover, the simplification process may be recursive: object X may simplify itself in terms of object Y, which in turn may go further simplification. Simplification Protocol ======================= This proposal introduces two new special methods: def __simplify__(self): return type, state @classmethod def __rebuild__(cls, state): return new_instance_of_cls_with_given_state __simplify__ takes no arguments (bar 'self'), and returns a tuple of '(type, state)', representing the contents of 'self': * 'type' is expected to be a class or a builtin type, although it can be any object that has a '__rebuild__' method. This 'type' will be used later to reconstruct the object (using 'type.__rebuild__(state)'). * 'state' is any other that represents the inner state of 'self', in a simplified form. This can be an atomic value, or yet another complex object, that may go further simplification. __rebuild__ is a expected to be classmethod that takes the state returned by __simplify__, and returns a new instance of 'cls' with the given state. If a type does not wish to be simplified, it may throw a TypeError in its __simplify__ method; however, this is not recommended. Types that want to be treated as atomic elements, such as file, should just return themselves, and let the serializer handle them. Default Simplification ===================== All the built in types would grow a __simplify__ and __rebuild__ methods, which would follow these guidelines: Primitive types (int, str, float, ...) are considered atomic. Composite types (I think 'complex' is the only type), are broken down into their components. For the complex type, that would be a tuple of (real, imaginary). Container types (tuples, lists, sets, dicts) represent themselves as tuples of items. For example, dicts would be simplified according to this pseudocode: def PyDict_Simplifiy(PyObject * self): return PyDictType, tuple(self.items()) Built in types would be considered atomic. User-defined classes can be simplified into their metaclass, __bases__, and __dict__. The type 'object' would simplify instances by returning their __dict__ and any __slots__ the instance may have. This is the default behavior; classes that desire a different behavior would override __simplify__ and __rebuild__. Example of default behavior: >>> class Foo(object): ... def __init__(self): ... self.a = 5 ... self.b = "spam" ... >>> f = Foo() >>> cls, state = f.__simplify__() >>> cls <class '__main__.Foo'> >>> state {"a" : 5, "b" : "spam"} >>> shallow_copy = cls.__rebuild__(state) >>> state.__simplify__() (<type 'dict'>, (("a", 5), ("b", "spam"))) Example of customized behavior >>> class Bar(object): ... def __init__(self): ... self.a = 5 ... self.b = "spam" ... def __simplify__(self): ... return Bar, 17.5 ... @clasmethod ... def __rebuild__(cls, state) ... self = cls.__new__(cls) ... if state == 17.5: ... self.a = 5 ... self.b = "spam" ... return self ... >>> b = Bar() >>> b.__simplify__() (<class '__main__.Bar'>, 17.5) Code objects ============= I wish that modules, classes and functions would also be simplifiable, however, there are some issues with that: * How to serialize code objects? These can be simplified as tuple of their co_* attributes, but these attributes are very implementation- specific. * How to serialize cell variables, or other globals? It would be nice if .pyc files where generated like so: import foo pickle.dump(foo) It would also allow sending of code between machines, just like any other object. Copying ======== Shallow copy, as well as deep copy, can be implemented using the semantics of this new API. The copy module should be rewritten accordingly: def copy(obj): cls, state = obj.__simplify__() return cls.__rebuild__(state) deepcopy() can be implemented similarly. Deprecation =========== With the proposed API, copy_reg, __reduce__, __reduce_ex__, and possibly other modules become deprecated. Apart from that, the pickle and copy modules need to be updated accordingly. C API ====== The proposal introduces two new C-API functions: PyObject * PyObject_Simplify(PyObject * self); PyObject * PyObject_Rebuild(PyObject * type, PyObject * state); Although this is only a suggestion. I'd like to hear ideas from someone with more experience in the core. I do not see a need for a convenience routine such as simplify(obj) <--> obj.__simplify__(), since this API is not designed for everyday usage. This is the case with __reduce__ today. Object Proxying =============== Because __simplify__ returns '(type, state)', it may choose to "lie" about it's actual type. This means that when the state is reconstructed, another type is used. Object proxies will use this mechanism to serialize the referred object, rather than the proxy. class Proxy(object): [...] def __simplify__(self): # this returns a tuple with the *real* type and state return self.value.__simplify__() Serialization ============ Serialization is the process of converting fully-simplified objects into byte sequences (strings). Fully simplified objects are created by a recursive simplifier, that simplifies the entire object graph into atomic components. Then, the serializer would convert the atomic components into strings. Note that this proposal does not define how atomic objects are to be converted to strings, or how a 'recursive simplifier' should work. These issues are to be resolved by the implementation of the serializer. For instance, file objects are atomic; one serializer may be able to handle them, by storing them as (filename, file-mode, file-position), while another may not be, so it would raise an exception. Recursive Simplifier =================== This code demonstrates the general idea of how recursive simplifiers may be implemented: def recursive_simplifier(obj): cls, state = obj.__simplify__() # simplify all the elements inside tuples if type(state) is tuple: nested_state = [] for item in state: nested_state.append(recursive_simplifier(item)) return cls, nested_state # see if the object is atomic; if not, dig deeper if (cls, state) == state.__simplify__(): # 'state' is an atomic object, no need to go further return cls, state else: # this object is not atomic, so dig deeper return cls, recusrive_simplifier(state) -tomer [View Less]

3 3

my take on mutable default arguments
by tomer filiba Jan. 27, 2007

Jan. 27, 2007

i thought this code be solved nicely with a decorator... it needs some more work, but it would make a good cookbook recipe: .>>> from copy import deepcopy. .>>> .>>> def defaults(**kwdefs): ... def deco(func): ... def wrapper(*args, **kwargs): ... for k,v in kwdefs.iteritems(): ... if k not in kwargs: ... kwargs[k] = deepcopy(v) ... return func(*args, **kwargs) ... return wrapper ... … [View More]

3 4

new pickle semantics/API
by tomer filiba Jan. 26, 2007

Jan. 26, 2007

---------- Forwarded message ---------- From: tomer filiba <tomerfiliba(a)gmail.com> Date: Jan 24, 2007 10:45 PM Subject: new pickle semantics/API To: Python-3000(a)python.org i'm having great trouble in RPyC with pickling object proxies. several users have asked for this feature, but no matter how hard i try to "bend the truth", pickle always complains. it uses type(obj) for the dispatching, which "uncovers" the object is actually a proxy, rather than a real object. recap: RPyC uses … [View More]local proxies that refer to objects of a remote interpreter (another process/machine). if you'd noticed, every RPC framework has its own serializer. for example banna/jelly in twisted and bunch of other XML serializers, and what not. for RPyC i wrote yet another serializer, but for different purposes, so it's not relevant for the issue at hand. what i want is a standard serialization *API*. the idea is that any framework could make use of this API, and that it would be generic enough to eliminate copy_reg and other misfortunes. this also means the built in types should be familiarized with this API. - - - - - - - - for example, currently the builtin types don't support __reduce__, and require pickle to use it's own internal registry. moreover, __reduce__ is very pickle-specific (i.e., it takes the protocol number). what i'm after is an API for "simplifying" complex objects into simpler parts. here's the API i'm suggesting: def __getstate__(self): # return a tuple of (type(self), obj), where obj is a simplified # version of self @classmethod def __setstate__(cls, state): # return an instance of cls, with the given state well, you may already know these two, although their semantics are different. but wait, there's more! the idea is of having the following simple building blocks: * integers (int/long) * strings (str) * arrays (tuples) all picklable objects should be able to express themselves as a collection of these building blocks. of course this will be recursive, i.e., object X could simplify itself as object Y, where object Y might go further simplification, until we are left with building blocks only. for example: * int - return self * float - string in the format "[+-]X.YYYe[+-]EEE" * complex - two floats * tuple - tuple of its simplified elements * list - tuple of its simplified elements * dict - a tuple of (key, value) tuples * set - a tuple of its items * file - raises TypeError("can't be simplified") all in all, i choose to call that *simplification* rather than *serialization*, as serialization is more about converting the simplified objects into a sequence of bytes. my suggestion leaves that out for the implementers of specific serializers. so this is how a typical serializer (e.g., pickle) would be implemented: * define its version of a "recursive simplifier" * optionally use a "memo" to remember objects that were already visited (so they would be serialized by reference rather than by value) * define its flavor of converting ints, strings, and arrays to bytes (binary, textual, etc. etc.) - - - - - - - - the default implementation of __getstate__, in object.__getstate__, will simply return self.__dict__ and any self.__slots__ this removes the need for __reduce__, __reduce_ex__, and copy_reg, and simplifies pickle greatly. it does require, however, adding support for simplification for all builtin types... but this doesn't call for much code: def PyList_GetState(self): state = tuple(PyObject_GetState(item) for item in self) return PyListType, state also note that it makes the copy module much simpler: def copy(obj): state = obj.__getstate__() return type(obj).__setstate__(state) - - - - - - - - executive summary: simplifying object serialization and copying by revising __getstate__ and __setstate__, so that they return a "simplified" version of the object. this new mechanism should become an official API to getting or setting the "contents" of objects (either builtin or user-defined). having this unified mechanism, pickling proxy objects would work as expected. if there's interest, i'll write a pep-like document to explain all the semantics. -tomer [View Less]

4 14

oneof() and multi split and replace for stirngs
by Calvin Spealman Jan. 23, 2007

Jan. 23, 2007

I don't know if this would make sense to try to push to 2.6 or 3.0. I was talking with some people about how the ability to split or replace on multiple substrings would be added to python, without adding new methods or having ugly tuple passing requirents like s.split(('foo', 'bar'), 4). This idea came to mind, so I wanted to toss it out there for scrutination. It would be a builtin, but can be implemented in python like this, basically: class oneof(list): def __init__(self, *args): … [View More]

3 2

Re: [Python-ideas] oneof() and multi split and replace for stirngs
by Calvin Spealman Jan. 23, 2007

Jan. 23, 2007

On 1/23/07, Josiah Carlson <jcarlson(a)uci.edu> wrote: > Whether it is a tuple being passed, or a "magic" container, I don't > think it matters; though I would lean towards a tuple because it is 5 > less characters to type out, and one fewer data types to worry about. I had talked to others and the concensus was against the tuples for ugliness. s.split((a,b), c) wasn't a popular choice, but s.split(oneof(a, b), c) reads better. > This has been discussed before in python-dev, … [View More]

2 1

Fwd: multi-dispatch again
by tomer filiba Jan. 14, 2007

Jan. 14, 2007

Guido asked me to move it here. anyway, i might as well state my case better. in the formal definitions of object oriented programming, objects are said to encapsulate state and behavior. behavior is largely dependent on the type of the object, but the state is important nonetheless. for example, file objects are stateful: they can be opened for reading-only, writing-only, both, or be closed altogether. still, they are all instances of the file type. since generic functions/multi-dispatch … [View More]come to help the programmer by reducing boilerplate early type-checking code and providing a more capable dispatch mechanism -- we can't overlook the state of the object. this early type-checking goes against the spirit of pure duck typing (which i'm fond of), but is crucial when the code has side effects. in this case, you can't just start executing the code and "hope it works", as the resources (i.e., files) involved are modified. in this kind of code, you want to check everything is alright *instead* of suddenly having an AttributeError/TypeError somewhere. here's an example: @dispatch def copy(src: file, dst: file): while True: buf = src.read(1000) if not buf: break dst.write(buf) suppose now that dst is mistakenly opened for reading. this means src would have already been modified, while dst.write is bound to fail. if src is a socket, for instance, this would be destructive. so if we already go as far as having multiple dispatch, which imposes constraints on the arguments a function accepts, we might as well base it on the type and state of the object, rather than only on it's type. we could say, for example: @dispatch def copy(src: file_for_reading, dst: file_for_writing): file_for_reading is not a type -- it's a checker. it may be defined as def file_for_reading(obj: file): return file.mode == "r" and not file.closed types, by default, would check using isinstance(), but costume checkers could check for stateful requirements too. and a note about performance: this check is required whether it's done explicitly or by the dispatch mechanism, and since most functions don't require so many overloads, i don't think it's an issue. besides, we can have a different decorator for dispatching by type or by checkers, i.e., @dispatch vs @stateful_dispatch, or something. the simple @dispatch would use a dictionary, while the stateful version would use a loop. -tomer ---------- Forwarded message ---------- From: tomer filiba <tomerfiliba(a)gmail.com> Date: Jan 14, 2007 2:28 PM Subject: multi-dispatch again To: Python-3000(a)python.org i just thought of a so-to-speak counter-example for ABCs... it's not really a counter-example, but i believe it shows a deficiency in the concept. theoretically speaking, objects are a made of type and state. ABCs, isinstance() and interfaces at general only check the type part. for example: @dispatch def log_to_file(text: str, device: file): file.write(text) this will constrain the *type* of the device, but not its *state*. practically speaking, i can pass a closed file, or a file open for reading-only, and it would pass silently. basing multi-dispatch on types is of course a leap forward, but if we already plan to take this leap, why not make it general enough to support more complex use-cases? this way we could rewrite the snippet above as @dispatch def log_to_file(text: str, device: open_file): file.write(text) where open_file isn't a type, but rather a "checker" that may also examine the state. by default, type objects would check for inheritance (via a special method), but checkers could extend this behavior. for efficiency purposes, we can have two decorators: @type_dispatch - dispatches based on type only @full_dispatch - dispatches based on type and state bottom line -- we can't just look at the type of the object for dispatching, overlooking its state. the state is meaningful, and we'd want the function not to be called at all if the state of the object is wrong. -tomer [View Less]

1 0

Re: [Python-ideas] [Python-Dev] features i'd like [Python 3000] ... #3: fix super()
by Eduardo "EdCrypt" O. Padoan Jan. 6, 2007

Jan. 6, 2007

> I was bitten by the urge to play with this today, and modified my > previous "self" hack to handle "super" also, so that the following > code works: > > class D (C): > @method > def sum(n): > return super.sum(n * 2) - self.base > > Posted as "evil2.py" here: > > http://www.lag.net/robey/code/surf/ > > Because hacking "super" requires having the class object handy, this > one needs a metaclass to do its magic,… [View More]

1 1

Module local namespaces
by Matt Draisey Jan. 3, 2007

Jan. 3, 2007

A while ago, I developed a small PyGtk programme that could dynamically reload all the working callbacks and logic while the GUI was still running. I could get away with this because of the flexible way Python loads modules at runtime, but it ended up being a waste of time as implementing it took more time that actually using it. For sanity's sake it quickly becomes clear you almost never want to rely on being able to refer to a half initialized module. And wouldn't it be nice if Python … [View More]

4 7

PEP 3107 Function Annotations: interoperability (again)
by Tony Lownds Jan. 3, 2007

Jan. 3, 2007

In discussing the Function Annotations PEP on python-list, interoperability between schemes came up again: http://mail.python.org/pipermail/python-list/2006-December/420645.html John Roth wrote: > Third, it's half of a proposal. Type checking isn't the only use > for metadata about functions/methods, classes, properties > and other objects, and the notion that there are only going to > be a small number of non-intersecting libraries out there is > an abdication of … [View More]

4 6

Re: [Python-ideas] new operators via backquoting
by George Sakkis Jan. 3, 2007

Jan. 3, 2007

Chris Rebert wrote: > In Haskell, foo `baz` bar means (baz foo bar), which translates to > baz(foo, bar) in Python. This allows Haskell programmers to use > functions as infix operators. > If I recall correctly, in Py3k, enclosing something in backticks will no > longer cause it to be repr()-ed, leaving the backtick without a meaning > in Python. Perhaps you're not aware of the "best hack of 2005", which works in current Python and goes a step further by allowing you to … [View More]

2 1