Mailman 3 April 2009 - Python-ideas

Updating PEP 315: do-while loops
by Raymond Hettinger 27 Apr '09

27 Apr '09

Am working on PEP 315 again, simplifying the proposal by focusing on a more standard do-while loop. The two motivating cases are: 1) A pattern of repeated code blocks before and inside a traditional while-loop. Here's an example from random.sample(): j = _int(random() * n) while j in selected: j = _int(random() * n) 2) A pattern of ending a "while True:" loop with an "if not <cond>: break". Here's an example from random.normalvariate(): while 1: u1 = random() u2 = 1.0 - random() z = NV_MAGICCONST*(u1-0.5)/u2 zz = z*z/4.0 if zz <= -_log(u2): break The challenge has been finding a syntax that fits well with the patterns in the rest of the language. It seems that every approach has it's own strengths and weaknesses. 1) One approach uses the standard do-while name but keeps usual python style formatting by putting the condition at the top instead of the bottom where it is in C and Java: do ... while j in selected: j = _int(random() * n) do ... while zz > -_log(u2): u1 = random() u2 = 1.0 - random() z = NV_MAGICCONST*(u1-0.5)/u2 zz = z*z/4.0 This syntax uses the ellipsis for an additional mental cue reminding the reader that the enclosed code block is executed before the condition is evaluated. It is not unlike the with-statement which has both enter and exit behaviors signaled by a single leading keyword and an indented block. Also, code-at-the-top approach fits well with the way the compiler would generate code with the condition test being preceding the loop body: 0 SETUP_LOOP 30 (to 33) 3 JUMP_ABSOLUTE 18 6 LOAD_NAME 0 (x) 9 LOAD_NAME 1 (y) 12 COMPARE_OP 0 (<) 15 POP_JUMP_IF_FALSE 33 18 <loop body> ... 30 JUMP_ABSOLUTE 6 33 LOAD_CONST 1 (None) 36 RETURN_VALUE I like the ellipsis form because of the mental cue it provides, but the proposal still works with any other spelling variant: do_while <cond>: do while <cond>: do_body_first_and_then_loop_if_the_test_condition_is_true <cond>: 2) Another approach is to put the test at the end. Something like: do: j = _int(random() * n) :while j in selected do: j = _int(random() * n) while j in selected These seem syntactically weird to me and feel more like typos than real python code. I'm sure there are many ways to spell the last line, but in the two years since I first worked on the PEP, I haven't found any condition-at-the-end syntax that FeelsRight(tm). Probably everyone has an opinion on these, but I don't any think those could be called bike-shedding. The spelling does matter and hopefully the cleanest solution can be found. It would be really great if we were to get some decent spelling of do-while in the language. Raymond P.S. I've punted on the more complex PEP-315 variants with repeated setup code: <setup code> while <condition>: <loop body> <setup code> It was hard enough to get a decent spelling for do-while. Also, I've punted on the while-cond-with-assignment case that is being handled by other proposals such as: while f.read(20) as data != '': ...

17 38

why not try without except?
by spir 27 Apr '09

27 Apr '09

Hello, In various cases, we need to do something (set or update a var or attribute, launch an action, compute data), or not, according to a condition that is a potential case of exception. Python provides the try...except construct to allow straightforward expression of the non-exceptional case without overloading the code with explicit checkings. Still, in the common case above, the result is try: <do_something> except ErrorType: pass or we have to fall back to a pre-checking construct if not <error_case>: <do_something> For instance try: text += doc.footer except ErrorType: pass or if hasattr(doc, 'footer') text += doc.footer Actually, this shows that 'try' alone can be considered analog to 'if' without 'else' clause, needed when the condition may raise an exception. Both express an optional action. This could be written more directly, because we do not need the condition: ? <do_something> option <do_something> with the meaning: "Try & do this, but if there's an exception just let it down." The same syntax may be reused in other contexts such as for having optional parameters: def f(data, option param): def f(data, ? param): Actually, the meaning is the same: "Try & and read a second argument, but if you step on an exception just let it down." Moreover, the body of the func will probably use precisely the same construct to try and do something using the optional param, e.g.: class Circle: ....... def draw(self, ? fill_color): <draw outline> ? self.fill(fill_color) As a nice side-effect, this would also remove one of the common (mis?)uses of None -- which is often considered problematic because of conflicting various uses. I would then support the introduction of such a syntax, or of 'try' without 'except'. And sincerally thank you for explaining why the latter was not allowed from start. Denis ------ la vita e estrany

8 20

Re: [Python-ideas] Updating PEP 315: do-while loops
by Larry Hastings 26 Apr '09

26 Apr '09

Mike Meyer wrote: >> It's true, "break if <condition>" and "continue if <condition>" are >> redundant constructs. But this debate is over refining our syntactic >> sugar--charting what is arguably a redundant construct. >> > No, it isn't. Well, it wasn't originally, but got hijacked into being > that. More on the original in a minute. > [...] > Minor spelling variants (which is what these are) > are pretty much poison as far as readability is concerned, and are > unPythonic. > > The original proposal - while it was syntactic sugar of sorts - dealt > with a real issue: Python doesn't have a natural loop construct for > the common case of having a conditional at the bottom. The existing > conditional loop can be coerced into that role in a couple of ways, > but all of them have problems. > I don't follow the distinction you draw. In what way is do-while not straight syntactic sugar? We can already express the construct with "while True: ... if not <expression>: break". There's nothing unnatural about spelling it this way, and it's worked for longer than either of us have been using Python. So, from my perspective, the proposal is *asking* for redundant syntax. And therefore describing my suggestions as "unPythonic" and "pure poison" because they're redundant is sophistry. > The proposed solution puts the conditional at the top, which is pretty > much mandated by existing python control flow syntax: control > constructs appear at the top of blocks, control the code indented > beneath them, and they are terminated by a dedent (yes, I'm ignoring > statement stacking). Unfortunately, this solution feels strange, what > with the conditional at the wrong end of the controlled block. > > A syntax that puts the conditional at the bottom end of the controlled > block either uses a dedent to signal the end of the control block, > which clashes with existing constructs, or doesn't use such a dedent, > which means it's trivially turned into conditional in the middle > construct. Which means - as we've just seen - that it's in danger of > being a spelling variant of the existing solutions. > > What's really needed is a completely different looping construct. I don't agree with this conclusion. We're talking about "do/while", which is traditionally different from other looping structures. You call that "clashing", I call that its "raison d'etre". Having slept on it, I think we won't do any better than the usual approach for cooking up new Python syntax: view C syntax through Python-colored glasses. I claim this transformation works as follows: * Start with the C form, in K&R spelling, with curly braces. * Replace the opening curly brace with a colon. * Remove the closing curly brace entirely, and any non-newline whitespace after it. * Remove the parentheses around the conditional expression. * Remove semicolons. Applying that transformatino to C's "if", "if/else", and "while" gives us the Python syntax. And it shows us the way to write "for". So let's apply it to the C do/while loop. This: do { something(); } while (condition); As viewed through my "Python glasses" becomes this: do: something() while condition Yes, it's surprising that there's no colon on the "while" line. But we aren't indenting a new code block, so we shouldn't have a trailing colon. And Python has never used a leading colon to mean "outdent but continue the construct"; if it did we'd spell if-else as "if <condition>: ... :else:". If it looks weird to you to have a "while" not followed by a colon... good! do/while is a weird construct. We should *expect* it to look as weird in Python as it does in C. I therefore assert this syntax is optimal; certainly I doubt anyone will suggest one I'd like more. I still like "break if" and "continue if", but I now see those as separate. And I should not be surprised if they are henceforth ignored--as you point out they are wholly redundant. I think they're pleasing in their redundancy but I'm surely in the minority. /larry/

1 0

Revised**12 PEP on Yield-From
by Greg Ewing 25 Apr '09

25 Apr '09

Draft 13 of the PEP. Adjusted the expansion so as not to suggest that a throw or close attribute with the value None should be treated as a missing method. -- Greg PEP: XXX Title: Syntax for Delegating to a Subgenerator Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg.ewing(a)canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for a generator to delegate part of its operations to another generator. This allows a section of code containing 'yield' to be factored out and placed in another generator. Additionally, the subgenerator is allowed to return with a value, and the value is made available to the delegating generator. The new syntax also opens up some opportunities for optimisation when one generator re-yields values produced by another. Motivation ========== A Python generator is a form of coroutine, but has the limitation that it can only yield to its immediate caller. This means that a piece of code containing a ``yield`` cannot be factored out and put into a separate function in the same way as other code. Performing such a factoring causes the called function to itself become a generator, and it is necessary to explicitly iterate over this second generator and re-yield any values that it produces. If yielding of values is the only concern, this can be performed without much difficulty using a loop such as :: for v in g: yield v However, if the subgenerator is to interact properly with the caller in the case of calls to ``send()``, ``throw()`` and ``close()``, things become considerably more difficult. As will be seen later, the necessary code is very complicated, and it is tricky to handle all the corner cases correctly. A new syntax will be proposed to address this issue. In the simplest use cases, it will be equivalent to the above for-loop, but it will also handle the full range of generator behaviour, and allow generator code to be refactored in a simple and straightforward way. Proposal ======== The following new expression syntax will be allowed in the body of a generator: :: yield from <expr> where <expr> is an expression evaluating to an iterable, from which an iterator is extracted. The iterator is run to exhaustion, during which time it yields and receives values directly to or from the caller of the generator containing the ``yield from`` expression (the "delegating generator"). Furthermore, when the iterator is another generator, the subgenerator is allowed to execute a ``return`` statement with a value, and that value becomes the value of the ``yield from`` expression. The full semantics of the ``yield from`` expression can be described in terms of the generator protocol as follows: * Any values that the iterator yields are passed directly to the caller. * Any values sent to the delegating generator using ``send()`` are passed directly to the iterator. If the sent value is None, the iterator's ``next()`` method is called. If the sent value is not None, the iterator's ``send()`` method is called. If the call raises StopIteration, the delegating generator is resumed. Any other exception is propagated to the delegating generator. * Exceptions other than GeneratorExit thrown into the delegating generator are passed to the ``throw()`` method of the iterator. If the call raises StopIteration, the delegating generator is resumed. Any other exception is propagated to the delegating generator. * If a GeneratorExit exception is thrown into the delegating generator, or the ``close()`` method of the delegating generator is called, then the ``close()`` method of the iterator is called if it has one. If this call results in an exception, it is propagated to the delegating generator. Otherwise, GeneratorExit is raised in the delegating generator. * The value of the ``yield from`` expression is the first argument to the ``StopIteration`` exception raised by the iterator when it terminates. * ``return expr`` in a generator causes ``StopIteration(expr)`` to be raised. Enhancements to StopIteration ----------------------------- For convenience, the ``StopIteration`` exception will be given a ``value`` attribute that holds its first argument, or None if there are no arguments. Formal Semantics ---------------- Python 3 syntax is used in this section. 1. The statement :: RESULT = yield from EXPR is semantically equivalent to :: _i = iter(EXPR) try: _y = next(_i) except StopIteration as _e: _r = _e.value else: while 1: try: _s = yield _y except GeneratorExit as _e: try: _m = getattr(_i, 'close') except AttributeError: pass else: _m() raise _e except BaseException as _e: try: _m = getattr(_i, 'throw') except AttributeError: raise _e else: try: _y = _m(*sys.exc_info()) except StopIteration as _e: _r = _e.value break else: try: if _s is None: _y = next(_i) else: _y = _i.send(_s) except StopIteration as _e: _r = _e.value break RESULT = _r 2. In a generator, the statement :: return value is semantically equivalent to :: raise StopIteration(value) except that, as currently, the exception cannot be caught by ``except`` clauses within the returning generator. 3. The StopIteration exception behaves as though defined thusly: :: class StopIteration(Exception): def __init__(self, *args): if len(args) > 0: self.value = args[0] else: self.value = None Exception.__init__(self, *args) Rationale ========= The Refactoring Principle ------------------------- The rationale behind most of the semantics presented above stems from the desire to be able to refactor generator code. It should be possible to take an section of code containing one or more ``yield`` expressions, move it into a separate function (using the usual techniques to deal with references to variables in the surrounding scope, etc.), and call the new function using a ``yield from`` expression. The behaviour of the resulting compound generator should be, as far as reasonably practicable, the same as the original unfactored generator in all situations, including calls to ``next()``, ``send()``, ``throw()`` and ``close()``. The semantics in cases of subiterators other than generators has been chosen as a reasonable generalization of the generator case. The proposed semantics have the following limitations with regard to refactoring: * A block of code that catches GeneratorExit without subsequently re-raising it cannot be factored out while retaining exactly the same behaviour. * Factored code may not behave the same way as unfactored code if a StopIteration exception is thrown into the delegating generator. With use cases for these being rare to non-existent, it was not considered worth the extra complexity required to support them. Finalization ------------ There was some debate as to whether explicitly finalizing the delegating generator by calling its ``close()`` method while it is suspended at a ``yield from`` should also finalize the subiterator. An argument against doing so is that it would result in premature finalization of the subiterator if references to it exist elsewhere. Consideration of non-refcounting Python implementations led to the decision that this explicit finalization should be performed, so that explicitly closing a factored generator has the same effect as doing so to an unfactored one in all Python implementations. The assumption made is that, in the majority of use cases, the subiterator will not be shared. The rare case of a shared subiterator can be accommodated by means of a wrapper that blocks ``throw()`` and ``close()`` calls, or by using a means other than ``yield from`` to call the subiterator. Generators as Threads --------------------- A motivation for generators being able to return values concerns the use of generators to implement lightweight threads. When using generators in that way, it is reasonable to want to spread the computation performed by the lightweight thread over many functions. One would like to be able to call a subgenerator as though it were an ordinary function, passing it parameters and receiving a returned value. Using the proposed syntax, a statement such as :: y = f(x) where f is an ordinary function, can be transformed into a delegation call :: y = yield from g(x) where g is a generator. One can reason about the behaviour of the resulting code by thinking of g as an ordinary function that can be suspended using a ``yield`` statement. When using generators as threads in this way, typically one is not interested in the values being passed in or out of the yields. However, there are use cases for this as well, where the thread is seen as a producer or consumer of items. The ``yield from`` expression allows the logic of the thread to be spread over as many functions as desired, with the production or consumption of items occuring in any subfunction, and the items are automatically routed to or from their ultimate source or destination. Concerning ``throw()`` and ``close()``, it is reasonable to expect that if an exception is thrown into the thread from outside, it should first be raised in the innermost generator where the thread is suspended, and propagate outwards from there; and that if the thread is terminated from outside by calling ``close()``, the chain of active generators should be finalised from the innermost outwards. Syntax ------ The particular syntax proposed has been chosen as suggestive of its meaning, while not introducing any new keywords and clearly standing out as being different from a plain ``yield``. Optimisations ------------- Using a specialised syntax opens up possibilities for optimisation when there is a long chain of generators. Such chains can arise, for instance, when recursively traversing a tree structure. The overhead of passing ``next()`` calls and yielded values down and up the chain can cause what ought to be an O(n) operation to become, in the worst case, O(n\*\*2). A possible strategy is to add a slot to generator objects to hold a generator being delegated to. When a ``next()`` or ``send()`` call is made on the generator, this slot is checked first, and if it is nonempty, the generator that it references is resumed instead. If it raises StopIteration, the slot is cleared and the main generator is resumed. This would reduce the delegation overhead to a chain of C function calls involving no Python code execution. A possible enhancement would be to traverse the whole chain of generators in a loop and directly resume the one at the end, although the handling of StopIteration is more complicated then. Use of StopIteration to return values ------------------------------------- There are a variety of ways that the return value from the generator could be passed back. Some alternatives include storing it as an attribute of the generator-iterator object, or returning it as the value of the ``close()`` call to the subgenerator. However, the proposed mechanism is attractive for a couple of reasons: * Using a generalization of the StopIteration exception makes it easy for other kinds of iterators to participate in the protocol without having to grow an extra attribute or a close() method. * It simplifies the implementation, because the point at which the return value from the subgenerator becomes available is the same point at which the exception is raised. Delaying until any later time would require storing the return value somewhere. Rejected Ideas -------------- Some ideas were discussed but rejected. Suggestion: There should be some way to prevent the initial call to next(), or substitute it with a send() call with a specified value, the intention being to support the use of generators wrapped so that the initial next() is performed automatically. Resolution: Outside the scope of the proposal. Such generators should not be used with ``yield from``. Suggestion: If closing a subiterator raises StopIteration with a value, return that value from the ``close()`` call to the delegating generator. Resolution: Undesirable for a number of reasons. The purpose of closing a generator is to ensure proper cleanup, not to obtain a meaningful return value. Also, it would be unreliable unless the return value were stored so as to be available to subsequent close calls, which would cause it to persist for longer than expected. Suggestion: If ``close()`` is not to return a value, then raise an exception if StopIteration with a non-None value occurs. Resolution: No clear reason to do so. Ignoring a return value is not considered an error anywhere else in Python. Criticisms ========== Under this proposal, the value of a ``yield from`` expression would be derived in a very different way from that of an ordinary ``yield`` expression. This suggests that some other syntax not containing the word ``yield`` might be more appropriate, but no acceptable alternative has so far been proposed. Rejected alternatives include ``call``, ``delegate`` and ``gcall``. It has been suggested that some mechanism other than ``return`` in the subgenerator should be used to establish the value returned by the ``yield from`` expression. However, this would interfere with the goal of being able to think of the subgenerator as a suspendable function, since it would not be able to return values in the same way as other functions. The use of an exception to pass the return value has been criticised as an "abuse of exceptions", without any concrete justification of this claim. In any case, this is only one suggested implementation; another mechanism could be used without losing any essential features of the proposal. Alternative Proposals ===================== Proposals along similar lines have been made before, some using the syntax ``yield *`` instead of ``yield from``. While ``yield *`` is more concise, it could be argued that it looks too similar to an ordinary ``yield`` and the difference might be overlooked when reading code. To the author's knowledge, previous proposals have focused only on yielding values, and thereby suffered from the criticism that the two-line for-loop they replace is not sufficiently tiresome to write to justify a new syntax. By dealing with the full generator protocol, this proposal provides considerably more benefit. Additional Material =================== Some examples of the use of the proposed syntax are available, and also a prototype implementation based on the first optimisation outlined above. `Examples and Implementation`_ .. _Examples and Implementation: http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/ Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

7 28

For CPython 3.1: make type Check and CheckExact macros consistent? universal?
by Larry Hastings 25 Apr '09

25 Apr '09

There are a bunch of type-checking macros in Include/*.h. Most classes only provide one, Check(), implemented as follows: #define PyWhatever_Check(ob) (Py_TYPE(ob) == &PyWhatever_Type) Examples of this style include PyRange, PyMemoryView, PyFunction, and PySlice. Then there are types with a more sophisticated view of type identity. For example, an object can qualify as a list even if its type is not PyList_Type. For such types, there's a Check() macro that does a more sophisticated check. They'll also have a CheckExact() macro doing an exact type check; that'll look like PyWhatever_Check above. Then there's PyMethod. PyMethod has a CheckExact macro but *doesn't* have a Check macro. Hmm! When I stumbled across that it got me to thinking. Might it be a good idea for all Python types to have both Check and CheckExact macros? The CheckExact form would be consistent, and for types that only need an "exact" check they could define Check in terms of CheckExact: #define PyWhatever_CheckExact(ob) (Py_TYPE(ob) == &PyWhatever_Type) #define PyWhatever_Check PyWhatever_CheckExact This would let you express your intent. If you were expecting a Whatever-like object, you always use Check. If you require that exact object, you always use CheckExact. On the other hand, maybe every type-checking macro that does (Py_TYPE(ob) == &PyWhatever_Type) should be called CheckExact, and for types that don't have an inexact check they don't have a Check. Or maybe the existing subclass-style Check macros should be renamed CheckSubclass (or Implemented or something) and Check should always do a CheckExact-style check. I admit this isn't a stupendous idea, but I thought it was worth typing up. If there's interest I'd be happy to supply a patch for whichever form found favor. Or we could forget the whole thing, you and I, /larry/

3 2

Re: [Python-ideas] Revised**12 PEP on Yield-From
by Greg Ewing 24 Apr '09

24 Apr '09

Erik Groeneveld wrote: > I do not see how a yield-from without support for > splitting boundaries would be combined with my own code to do the > latter. Seems to me you could yield a sequence of values to be pushed back, and use a top-level driver something like this: def pushingback(g): queue = [] try: while 1: queue.append(yield) while queue: pushback = g.send(queue.pop(0)) if pushback: queue.extend(pushback) except StopIteration, e: return e.value > Well, the whole point of using coroutines is to avoid buffering. Arranging processing as a pipeline avoids having to buffer all of the intermediate data. But if lookahead is required, then you need at least a small amount of buffering somewhere, whether it's explicit or implicit. You want to do the buffering implicitly, by reading ahead and pushing back the stuff you don't want, making the caller responsible for holding onto it. While that's certainly possible, I'm far from convinced it's the easiest or clearest way to go about it. I would be more inclined to insert an explicit buffering object, such as a queue, between the two stages of the pipeline. > Thirdly, there seems to be some sort of unspoken 'protocol' with > generators. A next() is actually send(None) and vaguely means 'I want > data'. It the same vein 'x = yield' actually is 'x = yield None' and > also vaguely means 'I want data'. So the None seems to play a special > role. There is a one-to-one relationship between yields in the generator and next/send calls by the caller. Each yield provides an opportunity to transmit one value and receive one value. If either of these opportunities is unused, None is substituted. This is no more special than a function returning None when it has no other value of interest. This lockstep execution between producer and consumer is a feature of any coroutine system in which one coroutine sends values directly to another, without any form of buffering between. This may be an unfamiliar concept to programmers these days, because the modern incarnations of coroutines normally encountered (OS-provided processes and threads) *never* send values to each other directly -- there's always some kind of IPC object in between, such as a pipe or queue. So when seeing the generator send/yield mechanism for the first time, it's tempting to think of it as being like sending data to a process through a pipe. But it's not really like that -- it's a lower-level facility. > JSP requires a lookahead > and the coroutines must have some way to support this. (Introducing a > stream or buffer would put us back to where we started of course). I don't see why. You don't have to buffer all the data, only the minimum required to support the amount of lookahead needed. And I can't see why a pair of coroutines communicating via an IPC object that supports lookahead can't be just as simple and clear as using some kind of pushback mechanism. -- Greg

2 1

A conditional "for" statement
by Michael S. Gilbert 24 Apr '09

24 Apr '09

Hello, I've recently been working on some code where i am processing a list, but excluding certain items. The solution is to use a list comprehension in the "for" statement. An example is: for m in [n for n in range( 0 , 5 ) if n != 2] Determining what's going on here isn't immediately obvious (i.e. what's this new variable n doing?). It would be nice to have a more streamlined syntax such as: for m in range( 0 , 5 ) with m != 2 which is much cleaner and obvious. It is also very much like the list comprehension syntax (although i've changed "if" to "with", and the better of the two is subject to personal opinion and debatable). I would hope that the statements following "with" could be any conditional expression. This is just a thought I had while working today. Thank you for your consideration. Best Regards, Mike

7 8

Re: [Python-ideas] Revised**12 PEP on Yield-From
by Greg Ewing 24 Apr '09

24 Apr '09

Erik Groeneveld wrote: > the readRe generator must be able to > indicate that it has superfluous data, and this data must be processed > by other generators. > > Have you though about this? How would you solve it? I think you're expecting a bit much from yield-from. In a situation like this, I wouldn't use yield to receive the values. I'd read them from some kind of buffering object that allows peeking ahead however far is needed. -- Greg

3 3

An idea for a new pickling tool
by Raymond Hettinger 24 Apr '09

24 Apr '09

Motivation ---------- Python's pickles use a custom format that has evolved over time but they have five significant disadvantages: * it has lost its human readability and editability * is doesn't compress well * it isn't interoperable with other languages * it doesn't have the ability to enforce a schema * it is a major security risk for untrusted inputs New idea -------- Develop a solution using a mix of PyYAML, a python coded version of Kwalify, optional compression using bz2, gzip, or zlib, and pretty printing using pygments. YAML ( http://yaml.org/spec/1.2/ ) is a language independent standard for data serialization. PyYAML ( http://pyyaml.org/wiki/PyYAML ) is a full implementation of the YAML standard. It uses the YAML's application-specific tags and Python's own copy/reduce logic to provide the same power as pickle itself. Kwalify ( http://www.kuwata-lab.com/kwalify/ruby/users-guide.01.html ) is a schema validator written in Ruby and Java. It defines a YAML/JSON based schema definition for enforcing tight constraints on incoming data. The bz2, gzip, and zlib compression libraries are already built into the language. Pygments ( http://pygments.org/ ) is python based syntax highlighter with builtin support for YAML. Advantages ---------- * The format is simple enough to hand edit or to have lightweight applications emit valid pickles. For example: print('Todo: [go to bank, pick up food, write code]') # valid pickle * To date, efforts to make pickles smaller have focused on creating new codes for every data type. Instead, we can use the simple text formatting of YAML and let general purpose data compression utilities do their job (letting the user control the trade-offs between speed, space, and human readability): yaml.dump(data, compressor=None) # fast, human readable, no compression yaml.dump(data, compressor=bz2) # slowest, but best compression yaml.dump(data, compressor=zlib) # medium speed and medium compression * The current pickle tools makes it easy to exchange object trees between two Python processes. The new tool would make it equally easy to exchange object trees between processes running any of Python, Ruby, Java, C/C++, Perl, C#, PHP, OCaml, Javascript, ActionScript, and Haskell. * Ability to use a schema for enforcing a given object model and allowing full security. Which would you rather run on untrusted data: data = yaml.load(myfile, schema=ListOfStrings) or data = pickle.load(myfile) * Specification of a schema using YAML itself ListOfStrings (a schema written in yaml) ........................................ type: seq sequence: - type: str Sample of valid input ..................... - foo - bar - baz Note, schemas can be defined for very complex, nested object models and allow many kinds of constraints (unique items, enumerated list of allowable values, min/max allowable ranges for values, data type, maximum length, and names of regular Python classes that can be constructed). * YAML is a superset of JSON, so the schema validation also works equally well with JSON encoded data. What needs to be done --------------------- * Combine the tools for a single, clean interface to C speed parsing of a data serialization standard, with optional compression, schema validation, and pretty printing.

14 20

Re: [Python-ideas] A conditional "for" statement
by Michael S. Gilbert 24 Apr '09

24 Apr '09

On Thu, 23 Apr 2009 19:09:14 -0400, Mike Meyer wrote: > If you believe this is still true (though as has been pointed out, > this idea has already been considered and rejected), show us the > timings. In particular, with a range of different values for the > percentage of excluded elements: say 0, 10, 50, 90 and 100? Ok, I'll admit when I'm wrong. It actually takes longer to use the list comprehension (for all exclusion percentages), which, for me at least, is completely unintuitive. For 1 million samples: excluded: 0%, comprehension time: 0.840000 excluded: 0%, if time: 0.530000 excluded: 10%, comprehension time: 0.770000 excluded: 10%, if time: 0.520000 excluded: 20%, comprehension time: 0.710000 excluded: 20%, if time: 0.490000 excluded: 30%, comprehension time: 0.670000 excluded: 30%, if time: 0.460000 excluded: 40%, comprehension time: 0.610000 excluded: 40%, if time: 0.450000 excluded: 50%, comprehension time: 0.560000 excluded: 50%, if time: 0.420000 excluded: 60%, comprehension time: 0.510000 excluded: 60%, if time: 0.400000 excluded: 70%, comprehension time: 0.450000 excluded: 70%, if time: 0.380000 excluded: 80%, comprehension time: 0.410000 excluded: 80%, if time: 0.350000 excluded: 90%, comprehension time: 0.360000 excluded: 90%, if time: 0.330000 excluded: 100%, comprehension time: 0.310000 excluded: 100%, if time: 0.300000 I hereby accept the rejection of this idea since it makes sense neither from a syntax nor efficiency point of view. Best Regards, Mike

2 1