Mailman 3 November 2011 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley Dec. 16, 2020

Dec. 16, 2020

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists … [View More]now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich [View Less]

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann Feb. 22, 2020

Feb. 22, 2020

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int … [View More]

14 20

i18n and Python tracebacks
by Andre Roberge Oct. 28, 2012

Oct. 28, 2012

I think it would be a good idea if Python tracebacks could be translated into languages other than English - and it would set a good example. For example, using French as my default local language, instead of >>> 1/0 Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: integer division or modulo by zero I might get something like >>> 1/0 Suivi d'erreur (appel le plus récent en dernier) : Fichier "<stdin>", à la … [View More]

9 15

Re: [Python-ideas] Cofunctions - Back to Basics
by Mark Shannon Oct. 21, 2012

Oct. 21, 2012

Greg Ewing wrote: > Mark Shannon wrote: > >> Why not have proper co-routines, instead of hacked-up generators? > > What do you mean by a "proper coroutine"? > A parallel, non-concurrent, thread of execution. It should be able to transfer control from arbitrary places in execution, not within generators. Stackless provides coroutines. Greenlets are also coroutines (I think). Lua has them, and is implemented in ANSI C, so it can be done portably. See: http://www.jucs.… [View More]

9 25

Cofunctions PEP - Revision 4
by Greg Ewing Oct. 13, 2012

Oct. 13, 2012

Here's an updated version of the PEP reflecting my recent suggestions on how to eliminate 'codef'. PEP: XXX Title: Cofunctions Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg.ewing(a)canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for defining and calling a special type of generator called a 'cofunction'. It is designed to provide a … [View More]streamlined way of writing generator-based coroutines, and allow the early detection of certain kinds of error that are easily made when writing such code, which otherwise tend to cause hard-to-diagnose symptoms. This proposal builds on the 'yield from' mechanism described in PEP 380, and describes some of the semantics of cofunctions in terms of it. However, it would be possible to define and implement cofunctions independently of PEP 380 if so desired. Specification ============= Cofunction definitions ---------------------- A cofunction is a special kind of generator, distinguished by the presence of the keyword ``cocall`` (defined below) at least once in its body. It may also contain ``yield`` and/or ``yield from`` expressions, which behave as they do in other generators. From the outside, the distinguishing feature of a cofunction is that it cannot be called the same way as an ordinary function. An exception is raised if an ordinary call to a cofunction is attempted. Cocalls ------- Calls from one cofunction to another are made by marking the call with a new keyword ``cocall``. The expression :: cocall f(*args, **kwds) is evaluated by first checking whether the object ``f`` implements a ``__cocall__`` method. If it does, the cocall expression is equivalent to :: yield from f.__cocall__(*args, **kwds) except that the object returned by __cocall__ is expected to be an iterator, so the step of calling iter() on it is skipped. If ``f`` does not have a ``__cocall__`` method, or the ``__cocall__`` method returns ``NotImplemented``, then the cocall expression is treated as an ordinary call, and the ``__call__`` method of ``f`` is invoked. Objects which implement __cocall__ are expected to return an object obeying the iterator protocol. Cofunctions respond to __cocall__ the same way as ordinary generator functions respond to __call__, i.e. by returning a generator-iterator. Certain objects that wrap other callable objects, notably bound methods, will be given __cocall__ implementations that delegate to the underlying object. Grammar ------- The full syntax of a cocall expression is described by the following grammar lines: :: atom: cocall | <existing alternatives for atom> cocall: 'cocall' atom cotrailer* '(' [arglist] ')' cotrailer: '[' subscriptlist ']' | '.' NAME Note that this syntax allows cocalls to methods and elements of sequences or mappings to be expressed naturally. For example, the following are valid: :: y = cocall self.foo(x) y = cocall funcdict[key](x) y = cocall a.b.c[i].d(x) Also note that the final calling parentheses are mandatory, so that for example the following is invalid syntax: :: y = cocall f # INVALID New builtins, attributes and C API functions -------------------------------------------- To facilitate interfacing cofunctions with non-coroutine code, there will be a built-in function ``costart`` whose definition is equivalent to :: def costart(obj, *args, **kwds): try: m = obj.__cocall__ except AttributeError: result = NotImplemented else: result = m(*args, **kwds) if result is NotImplemented: raise TypeError("Object does not support cocall") return result There will also be a corresponding C API function :: PyObject *PyObject_CoCall(PyObject *obj, PyObject *args, PyObject *kwds) It is left unspecified for now whether a cofunction is a distinct type of object or, like a generator function, is simply a specially-marked function instance. If the latter, a read-only boolean attribute ``__iscofunction__`` should be provided to allow testing whether a given function object is a cofunction. Motivation and Rationale ======================== The ``yield from`` syntax is reasonably self-explanatory when used for the purpose of delegating part of the work of a generator to another function. It can also be used to good effect in the implementation of generator-based coroutines, but it reads somewhat awkwardly when used for that purpose, and tends to obscure the true intent of the code. Furthermore, using generators as coroutines is somewhat error-prone. If one forgets to use ``yield from`` when it should have been used, or uses it when it shouldn't have, the symptoms that result can be extremely obscure and confusing. Finally, sometimes there is a need for a function to be a coroutine even though it does not yield anything, and in these cases it is necessary to resort to kludges such as ``if 0: yield`` to force it to be a generator. The ``cocall`` construct address the first issue by making the syntax directly reflect the intent, that is, that the function being called forms part of a coroutine. The second issue is addressed by making it impossible to mix coroutine and non-coroutine code in ways that don't make sense. If the rules are violated, an exception is raised that points out exactly what and where the problem is. Lastly, the need for dummy yields is eliminated by making it possible for a cofunction to call both cofunctions and ordinary functions with the same syntax, so that an ordinary function can be used in place of a cofunction that yields zero times. Record of Discussion ==================== An earlier version of this proposal required a special keyword ``codef`` to be used in place of ``def`` when defining a cofunction, and disallowed calling an ordinary function using ``cocall``. However, it became evident that these features were not necessary, and the ``codef`` keyword was dropped in the interests of minimising the number of new keywords required. The use of a decorator instead of ``codef`` was also suggested, but the current proposal makes this unnecessary as well. It has been questioned whether some combination of decorators and functions could be used instead of a dedicated ``cocall`` syntax. While this might be possible, to achieve equivalent error-detecting power it would be necessary to write cofunction calls as something like :: yield from cocall(f)(args) making them even more verbose and inelegant than an unadorned ``yield from``. It is also not clear whether it is possible to achieve all of the benefits of the cocall syntax using this kind of approach. Prototype Implementation ======================== An implementation of an earlier version of this proposal in the form of patches to Python 3.1.2 can be found here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cofunctions.h… If this version of the proposal is received favourably, the implementation will be updated to match. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

11 29

Avoiding nested for try..finally: atexit for functions?
by Nikolaus Rath Dec. 12, 2011

Dec. 12, 2011

Hello, I often have code of the form: def my_fun(): allocate_res1() try: # do stuff allocate_res2() try: # do stuff allocate_res3() try: # do stuff finally: cleanup_res3() finally: cleanup_res2() finally: cleanup_res1() return With increasing number of managed resources, the indentation becomes really annoying, there is lots of line noise, and I don't like … [View More]

8 18

Is there a reason why file.readlines() doesn't/can't return an iterator?
by Giampaolo Rodolà Dec. 2, 2011

Dec. 2, 2011

This is problably too late and I'm probably missing something but given amount of generators/iterators introduced in python 3.X (http://docs.python.org/release/3.0.1/whatsnew/3.0.html#views-and-iterators-…) file.readlines() seems a good case where an iterator can be more appropriate than a list. I realized it while writing this recipe: http://code.activestate.com/recipes/577968-log-watcher-tail-f-log/ In this specific case, having readlines() yield a single line at a time would save a lot of … [View More]

12 21

Making min and max behave more like any and all
by Jakob Bowyer Nov. 30, 2011

Nov. 30, 2011

I just had something pointed out to me in #python that min and max accept *args, I know for a fact that any and all only use a single iterable password. Shouldn't we allow either one of the two ideas? Either max/min take only iterable arguments. OR Allow any/all to use *args In the second case this becomes legal any(1 in a, 2 in b, 3 in c) In the first case this becomes illegal max(1,2,3,4,5)

5 5

PEP 3155 - Qualified name for classes and functions
by Antoine Pitrou Nov. 27, 2011

Nov. 27, 2011

Hello, I would like to propose the following PEP for discussion and, if possible, acceptance. I think the proposal shouldn't be too controversial (I find it quite simple and straightforward myself :-)). I also have a draft implementation that's quite simple (http://hg.python.org/features/pep-3155). Regards Antoine. PEP: 3155 Title: Qualified name for classes and functions Version: $Revision$ Last-Modified: $Date$ Author: Antoine Pitrou <solipsis(a)pitrou.net> Status: Draft Type: … [View More]Standards Track Content-Type: text/x-rst Created: 2011-10-29 Python-Version: 3.3 Post-History: Resolution: TBD Rationale ========= Python's introspection facilities have long had poor support for nested classes. Given a class object, it is impossible to know whether it was defined inside another class or at module top-level; and, if the former, it is also impossible to know in which class it was defined. While use of nested classes is often considered poor style, the only reason for them to have second class introspection support is a lousy pun. Python 3 adds insult to injury by dropping what was formerly known as unbound methods. In Python 2, given the following definition:: class C: def f(): pass you can then walk up from the ``C.f`` object to its defining class:: >>> C.f.im_class <class '__main__.C'> This possibility is gone in Python 3:: >>> C.f.im_class Traceback (most recent call last): File "<stdin>", line 1, in <module> AttributeError: 'function' object has no attribute 'im_class' >>> dir(C.f) ['__annotations__', '__call__', '__class__', '__closure__', '__code__', '__defaults__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__get__', '__getattribute__', '__globals__', '__gt__', '__hash__', '__init__', '__kwdefaults__', '__le__', '__lt__', '__module__', '__name__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__'] This limits again the introspection capabilities available to the user. It can produce actual issues when porting software to Python 3, for example Twisted Core where the issue of introspecting method objects came up several times. It also limits pickling support [1]_. Proposal ======== This PEP proposes the addition of a ``__qname__`` attribute to functions and classes. For top-level functions and classes, the ``__qname__`` attribute is equal to the ``__name__`` attribute. For nested classed, methods, and nested functions, the ``__qname__`` attribute contains a dotted path leading to the object from the module top-level. The repr() and str() of functions and classes is modified to use ``__qname__`` rather than ``__name__``. Example with nested classes --------------------------- >>> class C: ... def f(): pass ... class D: ... def g(): pass ... >>> C.__qname__ 'C' >>> C.f.__qname__ 'C.f' >>> C.D.__qname__ 'C.D' >>> C.D.g.__qname__ 'C.D.g' Example with nested functions ----------------------------- >>> def f(): ... def g(): pass ... return g ... >>> f.__qname__ 'f' >>> f().__qname__ 'f.g' Limitations =========== With nested functions (and classes defined inside functions), the dotted path will not be walkable programmatically as a function's namespace is not available from the outside. It will still be more helpful to the human reader than the bare ``__name__``. As the ``__name__`` attribute, the ``__qname__`` attribute is computed statically and it will not automatically follow rebinding. References ========== .. [1] "pickle should support methods": http://bugs.python.org/issue9276 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: [View Less]

18 57

proto-pep capturing the concurrency discusion
by Mike Meyer Nov. 23, 2011

Nov. 23, 2011

I tried to capture all the ideas generated during the concurrency discussion early this month in the form of an informational PEP. I believe that information is worth preserving where others can find it, and a PEP seems like a logical place. Since it doesn't make a specific proposal, it's informational PEP. Possibly this is an abuse of the PEP mechanism, in which case I'll find another forum for it. The point this time is not to debate the ideas proposed, but to check for errors and omissions. … [View More]

2 1

Except block for with-statement
by Carl M. Johnson Nov. 20, 2011

Nov. 20, 2011

I was looking through the "What's New for Python 3.3" file and I saw this example code: try: with open("document.txt") as f: content = f.read() except FileNotFoundError: print("document.txt file is missing") except PermissionError: print("You are not allowed to read document.txt") and I thought it would be more elegant if you could drop the try. with open("document.txt") as f: content = f.read() except FileNotFoundError: print("document.txt file is missing") … [View More]

5 5

Exceptions thrown from generators.. patch.
by Ron Adam Nov. 20, 2011

Nov. 20, 2011

I was able to create a patch for testing this idea. The hard part was in getting to know cpython well enough to do it. :-) To get it to work, I made the following change in ceval.c so that the main loop will accept a pointer rather than an int for the throwflag. That allows an opcode to set it before it yields an exception. ie... a reverse throw. PyObject * PyEval_EvalFrameEx(PyFrameObject *f, int throwflag) { return PyEval_EvalFrame_Ex(f, &throwflag); } PyObject * … [View More]PyEval_EvalFrame_Ex(PyFrameObject *f, int *throwflag) { ... TARGET(YIELD_EXCEPT) *throwflag = 1; retval = POP(); f->f_stacktop = stack_pointer; why = WHY_YIELD; goto fast_yield; ... The genobject gen_send_ex() function checks the throwflag value after a send to see if it got a thrown out exception back. (Rather than one yielded out.) A new keyword 'throws' was needed to go with the YIELD_EXCEPT opcode. I didn't see anyway to do it with a function or method. It should be possible to set the exception in the ceval loop rather than yielding it out, but I think that would be more complex. Because the exception isn't raised inside the generator code object, but is yielded out first, the generator can be continued as if it was a regular yielded value. No magic required, and no fiddling with the exception stack was needed. It doesn't effect unexpected exceptions, or exceptions raised with raise. Those will terminate a generator as always. Python 3.3.0a0 (qbase qtip tip yield_except:92ac2848438f+, Nov 18 2011, 21:59:57) [GCC 4.6.1] on linux Type "help", "copyright", "credits" or "license" for more information. >>> def G(): ... yield 1 ... throws ValueError ... yield 2 ... >>> g = G() >>> next(g) 1 >>> next(g) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError >>> next(g) 2 >>> next(g) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration The three main benefits of being able to do this are... * To use switch like exception structures for flow control in schedulers and coroutines. * For consumer type coroutines to be able reject and re-request data in a nice way without terminating. ie.. the reverse of throwing in an exception in, in order to change what a generator does. * It creates alternative channels for data input and output by being able to both throw exceptions in and out of generators. Those signals can carry objects in and out and not burden the fast yield data path with testing for special wrapper objects. Here's an example of it being used in a simple scheduler. ----------- class Suspend(Exception): pass def Person(name, count, mode): n = 0 while n < count: if mode == 0: # The normal data path. yield name, count else: # Use an exception as an alternative data path. throws Suspend(name, count) n += 1 # return raise StopIteration(name, n) def main(data, mode): stack = [Person(*(args + (mode,))) for args in data] results = [] while stack: done = [] for ct in stack: try: print('yield', *next(ct)) # from yield except Suspend as exc: print('throws', *exc.args) # from throws except StopIteration as exc: results.append(exc.args) continue done.append(ct) stack = done print(results) return results if __name__ == "__main__": data = [("John", 2), ("Micheal", 3), ("Terry", 4)] results1 = main(data, 0) results2 = main(data, 1) assert(results1 == results2 == data) ------------- The output looks like... yield John 2 yield Micheal 3 yield Terry 4 yield John 2 yield Micheal 3 yield Terry 4 yield Micheal 3 yield Terry 4 yield Terry 4 [('John', 2), ('Micheal', 3), ('Terry', 4)] throws John 2 throws Micheal 3 throws Terry 4 throws John 2 throws Micheal 3 throws Terry 4 throws Micheal 3 throws Terry 4 throws Terry 4 [('John', 2), ('Micheal', 3), ('Terry', 4)] This shows that 'throws' works a lot like 'yield'. Open issues: * A better name than 'throws' might be good. * Should it get the object sent in. <object> = throws <exception> Or should it be ... throws <exception> * What would be the best argument form.. Should it take the same arguments as raise or just a single expression. Python's test suite passes as this doesn't change anything that already works. I haven't tested it with the yield-from patch yet, but I think if it can throw out exceptions in the same way yield-from yields out, that it will make some things easier and nicer to do. If anyone is interested, I can create a tracker item and put the patch there where it can be improved further. Cheers, Ron [View Less]

3 5

Renaming the "test" package
by Yuval Greenfield Nov. 16, 2011

Nov. 16, 2011

For python4, the "test" package should actually be named __pythontest__ or something similar. There are 2 million results for ""test.py"" on google. What do you think? --Yuval

3 3

try-else without except or finally
by Matt Joiner Nov. 15, 2011

Nov. 15, 2011

Hi Ideas, I frequently find myself in the following situations: 1) I wish to do something if no exception is thrown, for instance: try: logger.debug('Fiber starting: %s', self) try: self._result = self._routine() finally: logger.debug('Fiber finished: %s', self) except: raise else: raise FiberExit(self) finally: self._finished.set() … [View More]

18 35

Throwing exceptions from inside coroutines.
by Ron Adam Nov. 11, 2011

Nov. 11, 2011

Idea... Exceptions should be thrown out of coroutines rather than raised inside them and allowed to propagate out. There are several benefits. * It doesn't stop or leave the coroutine in a state that can't be continued. * It maintains the boundary between coroutines and other code in such a way that they may be replaced or wrapped in threads or processes easier. * It creates a new way to communicate to outside framework which simplifies creating schedulers. * It would work with yield-from … [View More]

1 0

Add additional special method lookups to math module
by Case Van Horsen Nov. 10, 2011

Nov. 10, 2011

Currently the functions round(), math.ceil(), math.floor(), and math.trunc() all check for the existence of a special method (__round__, __ceil__, __floor__, and __trunc__). Would it be possible to enhance the math and cmath modules to check for the existence of a special method for (almost) functions? For example, math.sin(obj) would first check for obj.__sin__. Rationale I'm in the final stages of adding support for the MPFR (multiple-precision floating point) and MPC (multiple-precision … [View More]

12 22

Re: [Python-ideas] Add additional special method lookups to math module
by Case Van Horsen Nov. 9, 2011

Nov. 9, 2011

Paul Moore wrote: > On 9 November 2011 20:15, Masklinn <masklinn(a)masklinn.net> wrote: >> This would mean custom numerical types wouldn't be drop-in compatible with existing numerical *code*, >> which I am under the impression is what Case Van Horsen wants (and is a desirable outcome). >> >> Furthermore, your second paragraph is not correct: Case does not propose "changing the behavior of >> built-in modules depending on the presence of another module", … [View More]Case proposes adding *method hooks* >> to existing math and cmath functions. These would be protocols allowing custom numerical types to >> implement `math`/`cmath`'s operation in a sensible manner, as is already possible for four methods he >> mentions, as well as a number of other Python operations[0], including prominent numerical ones[1]. > Correct. If a type does not implement one of the special methods, then there would be no change to the behaviour of the math/cmath modules. > I like the idea of being able to extend the math functions to allow > user-defined numeric types to behave like built in ones. (In fact, if > this were possible, complex could use it and a large proportion of the > cmath module would be unneeded). However, I have to say that for me > this is a purely theoretical issue - I've never had the need for the > functionality myself in real code, and I doubt I ever will. Given the > proliferation of special methods that would be needed, plus the > runtime overhead of extra checking, I think the cost is likely too > high. > > It's also worth noting the preamble text in the math module > documentation - the functions there deliberately only handle floats, > and are thin wrappers over the C standard library. To get any sort of > generic behaviour, users need to use cmath, by design. So as stated > the proposal is unlikely to get much support. > > Some alternative suggestions: > > 1. As already noted, you could include a gmpmath module with the > relevant functions, and users could monkeypatch the math module if > they wanted to do so. > The gmpy2 module provides a superset of the functions in math/cmath. I'm trying to avoid name collisions if someone replaces "from math import *" with "from gmpy2 import *". I know they shouldn't do that.... > 2. To make such monkeypatching easier, publish a module that exposes a > context manager to do the job: > with monkeypatch(math, 'sin', gmpmath.sin): > your code here... > > 3. Write a genericmath module that provides the "generic" versions > you're proposing, and see how much use it gets - if it's sufficiently > popular, you have a better case then for folding the functionality > into at least cmath. I'll try to put a prototype together. One issue with merging math and cmath is that math.functions raise an exception instead of returning a complex result. If it becomes a single module, should it return complex values instead of raising an exception? That could be a major change in someone's code. (gmpy2 uses a context manager to enable/disable the return of a complex result from a real function.) > > 4. As Georg mentioned, numpy's ufuncs have a feature like this, so > support that and people can use your types with numpy. That may be > enough (depending on your expected user base). > I will look at numpy's ufuncs. > Overall, I think the idea of being able to use user-defined types > interchangeably with built-in ones is a good one, but it's not > something Python goes out of its way to support. If the idea of > generic functions had taken off, then this is an area where they would > have fit nicely - but once again, there wasn't much enthusiasm from > the core developers for addressing the types of problem they were > aimed at. > > Paul. > casevh [View Less]

1 0

shutil.tail(file, lines)
by Giampaolo Rodolà Nov. 8, 2011

Nov. 8, 2011

This is something I need to do every once in a while and I think it would be a good addition for shutil module. Here: http://stackoverflow.com/questions/136168/get-last-n-lines-of-a-file-with-p… ...is a nice implementation which appears to be a good compromise in terms of speed and memory consumption (it reads the file in chunks, no more than 1024 bytes per-read). What do you think? --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/

4 6

Cofunctions - Back to Basics
by Greg Ewing Nov. 8, 2011

Nov. 8, 2011

Still feeling in a peppish mood after the last round of PEP 335, I decided to revisit my cofunctions proposal. Last time round, I allowed myself to be excessively swayed by popular opinion, and ended up with something that lacked elegance and failed to address the original issues. So, I've gone right back to my initial idea. The current version of the draft PEP can be found here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cd_current/co… together with updated examples here:… [View More]

13 78

Re: [Python-ideas] raising an exception type doesn't instantiate it until it gets caught
by Gregory P. Smith Nov. 7, 2011

Nov. 7, 2011

On Tue, Nov 1, 2011 at 1:15 AM, Nick Coghlan <ncoghlan(a)gmail.com> wrote: > > No, the traceback info is added by the eval loop itself. Remember that > when you raise an exception *type* (rather than an instance), the > exception doesn't get instantiated until it gets caught somewhere - > the eval loop maintains the unwinding stack for the traceback as part > of the thread state until it is time to attach it to the exception > object. > I did not know that. Is … [View More]

8 14

Re: [Python-ideas] A concurrency survey of sorts
by Weeble Nov. 6, 2011

Nov. 6, 2011

> Regardless, I still think the GIL should be fixed or the threading module > removed. It's disingenuous to have a threading module when it doesn't work > as advertised due to an interpreter "feature" of dubious merit anyway. Is this a serious proposal? Without threads, is there any reasonable cross-platform way to communicate interactively with multiple subprocesses over pipes? Windows doesn't have select. I don't *like* using threads, but as far as I can tell they're the only … [View More]

6 8

Need way to check if a `datetime.timedelta` is positive
by Ram Rachum Nov. 5, 2011

Nov. 5, 2011

(Sorry for possible double-post, Google Groups was doing weird things.) Hey, I'm currently writing code that's manipulating `datetime.timedelta` objects. I get one and I need to check whether it's positive or negative. I can't really think of an elegant way to do this. Here's one: my_timedelta >= datetime.timedelta(0) Another one: my_timedelta.total_seconds() >= 0 Perhaps we should allow a more elegant way? Possibly `my_timedelta >= 0` for 0 only? Thanks, Ram.

4 3

Fwd: Concurrent safety?
by Mike Meyer Nov. 5, 2011

Nov. 5, 2011

Sigh. Sent this only to bruce by accident. Sorry about the duplicates Bruce. ---------- Forwarded message ---------- From: Mike Meyer <mwm(a)mired.org> Date: Mon, Oct 31, 2011 at 10:30 PM Subject: Re: [Python-ideas] Concurrent safety? To: Bruce Leban <bruce(a)leapyear.org> On Mon, 31 Oct 2011 18:07:19 -0700 Bruce Leban <bruce(a)leapyear.org> wrote: > On Mon, Oct 31, 2011 at 4:19 PM, Mike Meyer <mwm(a)mired.org> wrote: > >> That would mean that all of my … [View More]current code is broken. > > Pretty much, yes. It's like adding garbage collection and removing > > alloc*/free. It's going to break a *lot* of code. That's why I said "not in > > 3. and possibly never in cPython." > In order to make concurrent code slightly safer, you're going to break all > existing programs that don't use concurrency. That seems to me like a new > language, not Python. You've been on this list long enough to see the > attention that's paid to backward compatibility. The way adding automatic memory management just made pointer arithmetic slightly safer. But yeah, the first thing I said was "never in 3.x, possibly never in cPython." I've been on this list long enough to know that, while the community pays a lot of attention to backwards compatibility, it is willing to throw it out when there's enough benefit. As you point out, this is a hard problem. I know I haven't covered all the issues. That's why the second thing I said was that I'm hoping to get people smarter than me to look at things. The cpu manufacturers have switched to improving performance by adding more cores instead of cranking clock speeds. As time goes by, more and more programmers are going to want to leverage that in serious ways. I already do, and find that Python makes some design choices unavailable or very fragile(*). I'd like to make those choices available, and help Python get ready for the time when that desire is the norm instead of the exception. > > Furthermore, merely *reading* an object that isn't locked can cause > >> problems. This code is not thread-safe: > >> if element in dictionary: return dictionary[element] > >> so you have to decide how much safety you want and what cost we're > >> willing to pay for this. > > You're right - it's not thread safe. However, it also doesn't suffer from > > the problem I'm trying to deal with, where you mutate an object in a way > > that leaves things broken, but won't be detected at that point. If it > > breaks because someone mutates the object underneath it, it'll throw an > > exception at that point. I know you can construct cases where that isn't > > so. > I think the cases where non-thread-safe code won't throw an exception are > numerous, for example, the equally trivial: Again, I said such cases can be built. I *didn't* say they were exceptions, I proposed a change to deal with them. > If you're going to tackle thread safety, it should address more of the > problem. These bugs are in many ways worse than mutating "an object in a > way that leaves things broken, but won't be detected at that point." The > above bugs may *never* be detected. I've come across bugs like that that > were in code for many years before I found them (and I'm sure that's > happened to others on this list as well). Like me. That's part of why I want to get the interpreter to help find them. > The first thing to do is identify the problems you want to solve and make > sure that the problems are well understood. Then design some solutions. > Starting with a bad solution to a fraction of the problem isn't a good > start. I've identified the problem I want to solve: I want to make concurrent use of python objects "safe by default", so that doing unsafe things causes the programmer to have to do something explicit about making things safe. I believe this can be done at the mutation points (again, clojure shows that it can be done). I also want to preserve as much of Python's existing code as possible. It may be that Python's existing data structures mean my believe about mutation points is wrong. This may be the wrong solution. It may be that such a change is to large to be acceptable. But the only way to find out is to investigate it. This discussion has already generated significant changes in the original proposal, plus some implementation ideas. <mike *) I'll note that some of the alternatives make the choices available in Python unavailable in a similar way. -- Mike Meyer <mwm(a)mired.org> http://www.mired.org/ Independent Software developer/SCM consultant, email for more information. O< ascii ribbon campaign - stop html mail - www.asciiribbon.org [View Less]

9 43

Re: [Python-ideas] PEP 3155 - Qualified name for classes and functions
by Joshua Landau Nov. 5, 2011

Nov. 5, 2011

First of all - hello python-ideas. I liked the look of the archives, so I thought I'd chip in, if it's OK. > And what about: >> >> <f locals>.g >> and >> <<xyzzy locals>.f locals>.g >> > > In the end I like the non-nested version better. > xyzzy.<local f>.<local g> or, preferably, xyzzy.<instance.f>.<instance.g> It's non-nested, shows the locality and maps the locality to the variable. If not showing … [View More]

1 0

A concurrency survey of sorts
by Mike Meyer Nov. 4, 2011

Nov. 4, 2011

In order to get a better idea of where things stand, I'd like to get answers to a few questions. This isn't a traditional broadbased survey, but an attempt to get answers from a few people who might know or have good ideas. This is probably where I should have started, but better late than never. 1) How much of the Python standard library is known to be thread safe? 2) How many packages in PyPI are known to be thread safe? 3) Can you suggest another approach to getting safe high-performance … [View More]

8 8