Mailman 3 July 2012 - Python-ideas

@classproperty, @abc.abstractclasspropery, etc.
by K. Richard Pixley 16 Dec '20

16 Dec '20

There's a whole matrix of these and I'm wondering why the matrix is currently sparse rather than implementing them all. Or rather, why we can't stack them as: class foo(object): @classmethod @property def bar(cls, ...): ... Essentially the permutation are, I think: {'unadorned'|abc.abstract}{'normal'|static|class}{method|property|non-callable attribute}. concreteness implicit first arg type name comments {unadorned} {unadorned} method def foo(): exists now {unadorned} {unadorned} property @property exists now {unadorned} {unadorned} non-callable attribute x = 2 exists now {unadorned} static method @staticmethod exists now {unadorned} static property @staticproperty proposing {unadorned} static non-callable attribute {degenerate case - variables don't have arguments} unnecessary {unadorned} class method @classmethod exists now {unadorned} class property @classproperty or @classmethod;@property proposing {unadorned} class non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract {unadorned} method @abc.abstractmethod exists now abc.abstract {unadorned} property @abc.abstractproperty exists now abc.abstract {unadorned} non-callable attribute @abc.abstractattribute or @abc.abstract;@attribute proposing abc.abstract static method @abc.abstractstaticmethod exists now abc.abstract static property @abc.staticproperty proposing abc.abstract static non-callable attribute {degenerate case - variables don't have arguments} unnecessary abc.abstract class method @abc.abstractclassmethod exists now abc.abstract class property @abc.abstractclassproperty proposing abc.abstract class non-callable attribute {degenerate case - variables don't have arguments} unnecessary I think the meanings of the new ones are pretty straightforward, but in case they are not... @staticproperty - like @property only without an implicit first argument. Allows the property to be called directly from the class without requiring a throw-away instance. @classproperty - like @property, only the implicit first argument to the method is the class. Allows the property to be called directly from the class without requiring a throw-away instance. @abc.abstractattribute - a simple, non-callable variable that must be overridden in subclasses @abc.abstractstaticproperty - like @abc.abstractproperty only for @staticproperty @abc.abstractclassproperty - like @abc.abstractproperty only for @classproperty --rich

10 15

Specify number of items to allocate for array.array() constructor
by Sven Rahmann 22 Feb '20

22 Feb '20

At the moment, the array module of the standard library allows to create arrays of different numeric types and to initialize them from an iterable (eg, another array). What's missing is the possiblity to specify the final size of the array (number of items), especially for large arrays. I'm thinking of suffix arrays (a text indexing data structure) for large texts, eg the human genome and its reverse complement (about 6 billion characters from the alphabet ACGT). The suffix array is a long int array of the same size (8 bytes per number, so it occupies about 48 GB memory). At the moment I am extending an array in chunks of several million items at a time at a time, which is slow and not elegant. The function below also initializes each item in the array to a given value (0 by default). Is there a reason why there the array.array constructor does not allow to simply specify the number of items that should be allocated? (I do not really care about the contents.) Would this be a worthwhile addition to / modification of the array module? My suggestions is to modify array generation in such a way that you could pass an iterator (as now) as second argument, but if you pass a single integer value, it should be treated as the number of items to allocate. Here is my current workaround (which is slow): def filled_array(typecode, n, value=0, bsize=(1<<22)): """returns a new array with given typecode (eg, "l" for long int, as in the array module) with n entries, initialized to the given value (default 0) """ a = array.array(typecode, [value]*bsize) x = array.array(typecode) r = n while r >= bsize: x.extend(a) r -= bsize x.extend([value]*r) return x

14 20

hexdump
by anatoly techtonik 29 Apr '13

29 Apr '13

Just an idea of usability fix for Python 3. hexdump module (function or bytes method is better) as simple, easy and intuitive way for dumping binary data when writing programs in Python. hexdump(bytes) - produce human readable dump of binary data, byte-by-byte representation, separated by space, 16-byte rows Rationale: 1. Debug. Generic binary data can't be output to console. A separate helper is needed to print, log or store its value in human readable format in database. This takes time. 2. Usability. binascii is ugly: name is not intuitive any more, there are a lot of functions, and it is not clear how it relates to unicode. 3. Serialization. It is convenient to have format that can be displayed in a text editor. Simple tools encourage people to use them. Practical example: >>> print(b) � � � �� >>> b '\xe6\xb0\x08\x04\xe7\x9e\x08\x04\xe7\xbc\x08\x04\xe7\xd5\x08\x04\xe7\xe4\x08\x04\xe6\xb0\x08\x04\xe7\xf0\x08\x04\xe7\xff\x08\x04\xe8\x0b\x08\x04\xe8\x1a\x08\x04\xe6\xb0\x08\x04\xe6\xb0\x08\x04' >>> print(binascii.hexlify(data)) e6b00804e79e0804e7bc0804e7d50804e7e40804e6b00804e7f00804e7ff0804e80b0804e81a0804e6b00804e6b00804 >>> >>> data = hexdump(b) >>> print(data) E6 B0 08 04 E7 9E 08 04 E7 BC 08 04 E7 D5 08 04 E7 E4 08 04 E6 B0 08 04 E7 F0 08 04 E7 FF 08 04 E8 0B 08 04 E8 1A 08 04 E6 B0 08 04 E6 B0 08 04 >>> >>> # achieving the same output with binascii is overcomplicated >>> data_lines = [binascii.hexlify(b)[i:min(i+32, len(binascii.hexlify(b)))] for i in xrange(0, len(binascii.hexlify(b)), 32)] >>> data_lines = [' '.join(l[i:min(i+2, len(l))] for i in xrange(0, len(l), 2)).upper() for l in data_lines] >>> print('\n'.join(data_lines)) E6 B0 08 04 E7 9E 08 04 E7 BC 08 04 E7 D5 08 04 E7 E4 08 04 E6 B0 08 04 E7 F0 08 04 E7 FF 08 04 E8 0B 08 04 E8 1A 08 04 E6 B0 08 04 E6 B0 08 04 On the other side, getting rather useless binascii output from hexdump() is quite trivial: >>> data.replace(' ','').replace('\n','').lower() 'e6b00804e79e0804e7bc0804e7d50804e7e40804e6b00804e7f00804e7ff0804e80b0804e81a0804e6b00804e6b00804' But more practical, for example, would be counting offset from hexdump: >>> print( ''.join( '%05x: %s\n' % (i*16,l) for i,l in enumerate(hexdump(b).split('\n')))) Etc. Conclusion: By providing better building blocks on basic level Python will become a better tool for more useful tasks. References: [1] http://stackoverflow.com/questions/2340319/python-3-1-1-string-to-hex [2] http://en.wikipedia.org/wiki/Hex_dump -- anatoly t.

8 10

Yielding through context managers
by Joshua Bartlett 09 Jan '13

09 Jan '13

I'd like to propose adding the ability for context managers to catch and handle control passing into and out of them via yield and generator.send() / generator.next(). For instance, class cd(object): def __init__(self, path): self.inner_path = path def __enter__(self): self.outer_path = os.getcwd() os.chdir(self.inner_path) def __exit__(self, exc_type, exc_val, exc_tb): os.chdir(self.outer_path) def __yield__(self): self.inner_path = os.getcwd() os.chdir(self.outer_path) def __send__(self): self.outer_path = os.getcwd() os.chdir(self.inner_path) Here __yield__() would be called when control is yielded through the with block and __send__() would be called when control is returned via .send() or .next(). To maintain compatibility, it would not be an error to leave either __yield__ or __send__ undefined. The rationale for this is that it's sometimes useful for a context manager to set global or thread-global state as in the example above, but when the code is used in a generator, the author of the generator needs to make assumptions about what the calling code is doing. e.g. def my_generator(path): with cd(path): yield do_something() do_something_else() Even if the author of this generator knows what effect do_something() and do_something_else() have on the current working directory, the author needs to assume that the caller of the generator isn't touching the working directory. For instance, if someone were to create two my_generator() generators with different paths and advance them alternately, the resulting behaviour could be most unexpected. With the proposed change, the context manager would be able to handle this so that the author of the generator doesn't need to make these assumptions. Naturally, nested with blocks would be handled by calling __yield__ from innermost to outermost and __send__ from outermost to innermost. I rather suspect that if this change were included, someone could come up with a variant of the contextlib.contextmanager decorator to simplify writing generators for this sort of situation. Cheers, J. D. Bartlett

6 14

i18n and Python tracebacks
by Andre Roberge 28 Oct '12

28 Oct '12

I think it would be a good idea if Python tracebacks could be translated into languages other than English - and it would set a good example. For example, using French as my default local language, instead of >>> 1/0 Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: integer division or modulo by zero I might get something like >>> 1/0 Suivi d'erreur (appel le plus récent en dernier) : Fichier "<stdin>", à la ligne 1, dans <module> ZeroDivisionError: division entière ou modulo par zéro André

9 15

Re: [Python-ideas] Cofunctions - Back to Basics
by Mark Shannon 21 Oct '12

21 Oct '12

Greg Ewing wrote: > Mark Shannon wrote: > >> Why not have proper co-routines, instead of hacked-up generators? > > What do you mean by a "proper coroutine"? > A parallel, non-concurrent, thread of execution. It should be able to transfer control from arbitrary places in execution, not within generators. Stackless provides coroutines. Greenlets are also coroutines (I think). Lua has them, and is implemented in ANSI C, so it can be done portably. See: http://www.jucs.org/jucs_10_7/coroutines_in_lua/de_moura_a_l.pdf (One of the examples in the paper uses coroutines to implement generators, which is obviously not required in Python :) ) Cheers, Mark.

9 25

Python as a tool to download stuff for bootstrapping
by anatoly techtonik 15 Oct '12

15 Oct '12

This one is practical. I am looking at NaCl SDK download page: https://developers.google.com/native-client/sdk/download "you need Python installed", "download SDK update utility" What makes me sad that update utility is a Python script in a zip file - nacl_sdk.zip which includes shell script and a .bat file for launching this Python script. This makes me kind of sad. You have Python installed. Why can't you just crossplatformly do: mkdir nacl cd nacl python -m urllib get http://commondatastorage.googleapis.com/nativeclient-mirror/nacl/nacl_sdk/u… python update_sdk.py

5 8

Cofunctions PEP - Revision 4
by Greg Ewing 13 Oct '12

13 Oct '12

Here's an updated version of the PEP reflecting my recent suggestions on how to eliminate 'codef'. PEP: XXX Title: Cofunctions Version: $Revision$ Last-Modified: $Date$ Author: Gregory Ewing <greg.ewing(a)canterbury.ac.nz> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 13-Feb-2009 Python-Version: 3.x Post-History: Abstract ======== A syntax is proposed for defining and calling a special type of generator called a 'cofunction'. It is designed to provide a streamlined way of writing generator-based coroutines, and allow the early detection of certain kinds of error that are easily made when writing such code, which otherwise tend to cause hard-to-diagnose symptoms. This proposal builds on the 'yield from' mechanism described in PEP 380, and describes some of the semantics of cofunctions in terms of it. However, it would be possible to define and implement cofunctions independently of PEP 380 if so desired. Specification ============= Cofunction definitions ---------------------- A cofunction is a special kind of generator, distinguished by the presence of the keyword ``cocall`` (defined below) at least once in its body. It may also contain ``yield`` and/or ``yield from`` expressions, which behave as they do in other generators. From the outside, the distinguishing feature of a cofunction is that it cannot be called the same way as an ordinary function. An exception is raised if an ordinary call to a cofunction is attempted. Cocalls ------- Calls from one cofunction to another are made by marking the call with a new keyword ``cocall``. The expression :: cocall f(*args, **kwds) is evaluated by first checking whether the object ``f`` implements a ``__cocall__`` method. If it does, the cocall expression is equivalent to :: yield from f.__cocall__(*args, **kwds) except that the object returned by __cocall__ is expected to be an iterator, so the step of calling iter() on it is skipped. If ``f`` does not have a ``__cocall__`` method, or the ``__cocall__`` method returns ``NotImplemented``, then the cocall expression is treated as an ordinary call, and the ``__call__`` method of ``f`` is invoked. Objects which implement __cocall__ are expected to return an object obeying the iterator protocol. Cofunctions respond to __cocall__ the same way as ordinary generator functions respond to __call__, i.e. by returning a generator-iterator. Certain objects that wrap other callable objects, notably bound methods, will be given __cocall__ implementations that delegate to the underlying object. Grammar ------- The full syntax of a cocall expression is described by the following grammar lines: :: atom: cocall | <existing alternatives for atom> cocall: 'cocall' atom cotrailer* '(' [arglist] ')' cotrailer: '[' subscriptlist ']' | '.' NAME Note that this syntax allows cocalls to methods and elements of sequences or mappings to be expressed naturally. For example, the following are valid: :: y = cocall self.foo(x) y = cocall funcdict[key](x) y = cocall a.b.c[i].d(x) Also note that the final calling parentheses are mandatory, so that for example the following is invalid syntax: :: y = cocall f # INVALID New builtins, attributes and C API functions -------------------------------------------- To facilitate interfacing cofunctions with non-coroutine code, there will be a built-in function ``costart`` whose definition is equivalent to :: def costart(obj, *args, **kwds): try: m = obj.__cocall__ except AttributeError: result = NotImplemented else: result = m(*args, **kwds) if result is NotImplemented: raise TypeError("Object does not support cocall") return result There will also be a corresponding C API function :: PyObject *PyObject_CoCall(PyObject *obj, PyObject *args, PyObject *kwds) It is left unspecified for now whether a cofunction is a distinct type of object or, like a generator function, is simply a specially-marked function instance. If the latter, a read-only boolean attribute ``__iscofunction__`` should be provided to allow testing whether a given function object is a cofunction. Motivation and Rationale ======================== The ``yield from`` syntax is reasonably self-explanatory when used for the purpose of delegating part of the work of a generator to another function. It can also be used to good effect in the implementation of generator-based coroutines, but it reads somewhat awkwardly when used for that purpose, and tends to obscure the true intent of the code. Furthermore, using generators as coroutines is somewhat error-prone. If one forgets to use ``yield from`` when it should have been used, or uses it when it shouldn't have, the symptoms that result can be extremely obscure and confusing. Finally, sometimes there is a need for a function to be a coroutine even though it does not yield anything, and in these cases it is necessary to resort to kludges such as ``if 0: yield`` to force it to be a generator. The ``cocall`` construct address the first issue by making the syntax directly reflect the intent, that is, that the function being called forms part of a coroutine. The second issue is addressed by making it impossible to mix coroutine and non-coroutine code in ways that don't make sense. If the rules are violated, an exception is raised that points out exactly what and where the problem is. Lastly, the need for dummy yields is eliminated by making it possible for a cofunction to call both cofunctions and ordinary functions with the same syntax, so that an ordinary function can be used in place of a cofunction that yields zero times. Record of Discussion ==================== An earlier version of this proposal required a special keyword ``codef`` to be used in place of ``def`` when defining a cofunction, and disallowed calling an ordinary function using ``cocall``. However, it became evident that these features were not necessary, and the ``codef`` keyword was dropped in the interests of minimising the number of new keywords required. The use of a decorator instead of ``codef`` was also suggested, but the current proposal makes this unnecessary as well. It has been questioned whether some combination of decorators and functions could be used instead of a dedicated ``cocall`` syntax. While this might be possible, to achieve equivalent error-detecting power it would be necessary to write cofunction calls as something like :: yield from cocall(f)(args) making them even more verbose and inelegant than an unadorned ``yield from``. It is also not clear whether it is possible to achieve all of the benefits of the cocall syntax using this kind of approach. Prototype Implementation ======================== An implementation of an earlier version of this proposal in the form of patches to Python 3.1.2 can be found here: http://www.cosc.canterbury.ac.nz/greg.ewing/python/generators/cofunctions.h… If this version of the proposal is received favourably, the implementation will be updated to match. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

11 29

itertools.chunks(iterable, size, fill=None)
by anatoly techtonik 03 Sep '12

03 Sep '12

Now that Python 3 is all about iterators (which is a user killer feature for Python according to StackOverflow - http://stackoverflow.com/questions/tagged/python) would it be nice to introduce more first class functions to work with them? One function to be exact to split string into chunks. itertools.chunks(iterable, size, fill=None) Which is the 33th most voted Python question on SO - http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-even… P.S. CC'ing to python-dev@ to notify about the thread in python-ideas.

15 25

Re: [Python-ideas] compatibility triples for binary distributions
by Miki Tebeka 07 Aug '12

07 Aug '12

IMO implementation names should be compatible with PEP-421<http://www.python.org/dev/peps/pep-0421/>. Also instead of a cryptic long string, maybe some structured one?. For example (JSON): {"implementation":"cpython","machine":"x86_64"} It's longer but more readable IMO (and easier to parse). On Wednesday, July 25, 2012 7:55:34 AM UTC-7, Daniel Holth wrote: > > I am working on a specification that might be useful in the PKG-INFO > Platform or Supported-Platform tags. It tries to express which Python > versions a built distribution works on with a triple > (Implementation+Language Version, ABI, Architecture), anticipating an > ABI that is not tied to a single CPython release. > > A typical tag would be py27-noabi-noarch or cp33-cp33dmu-linux_x86_64. > > cp27-noabi-noarch is a package that requires CPython 2.7 but doesn't > include extension modules. > > The Python implementation is abbreviated. Each implementation has a > two-letter code: > > py: Generic Python > cp: CPython > ip: IronPython > pp: PyPy > jy: Jython > > concatenated with py_version_nodot “27”. The minor version can be > omitted “py2” or “py3” when appropriate. > > The ABI tag is an abbreviated SOABI “cp33m”, or, for “pure Python” > packages, “noabi”. For pypy it was more or less suggested that the ABI > would have to be a hash of the source code revision and the > translation/compilation options. > > The platform tag is distutils.util.get_platform() with all periods and > hyphens replaced with underscore, or the string ‘noarch’. > > I use the term "built distribution" instead of "binary distribution" > because it includes pure python packages that for example have had > 2to3 run ahead of time and are installable without setup.py. > > Will it work? The ABI in particular may be so messy that it should be > lumped with architecture. What is the ABI for packages that use ctypes or > cffi? >

3 4

changing sys.stdout encoding
by Rurpy 07 Aug '12

07 Aug '12

In my first foray into Python3 I've encountered this problem: I work in a multi-language environment. I've written a number of tools, mostly command-line, that generate output on stdout. Because these tools and their output are used by various people in varying environments, the tools all have an --encoding option to provide output that meets the needs and preferences of the output's ultimate consumers. In converting them to Python3, I found the best (if not very pleasant) way to do this in Python3 was to put something like this near the top of each tool[*1]: import codecs sys.stdout = codecs.getwriter(opts.encoding)(sys.stdout.buffer) What I want to be able to put there instead is: sys.stdout.set_encoding (opts.encoding) The former I found on the internet -- there is zero probability I could have figured that out from the Python docs. It is obscure to anyone (who has like me generally only needed to deal with .encode() and .decode()) who hasn't encountered it before or dealt much with the codecs module. It is excessively complex for what is conceptually a simple and straight-forward operation. It requires the import of the codecs module in programs that other- wise don't need it [*2], and the reading of the codecs docs (not a shining example of clarity themselves) to understand it. In short it is butt ugly relative to what I generally get in Python. Would it be feasible to provide something like .set_encoding() on textio streams? (Or make .encoding a writeable property?; it seems to intentionally be non-writeable for some reason but is that reason really unavoidable?) If doing this for textio in general is too hard, then what about encapsulating the codecs stuff above in a sys.set_encoding() function? Needing to change the encoding of a sys.std* stream is not an uncommon need and a user should not have to go through the codecs dance above to do so IMO. ---- [*1] There are other ways to change stdout's encoding but they all have problems AFAICT. PYTHONIOENCODING can't easily be changed dynamically within program. Reopening stdout as binary, or using the binary interface to text stdout, requires a explicit encode call at each write site. Overloading print() is obscure because it requires reader to notice print was overloaded. [*2] I don't mean the actual import of the codecs module which occurs anyway; I mean the extra visual and cognitive noise introduced by the presence of the import statement in the source.

16 33

Better support for finalization with weakrefs
by Richard Oudkerk 02 Aug '12

02 Aug '12

I would like to see better support for the use of weakrefs callbacks for object finalization. The current issues with weakref callbacks include: 1. They are rather low level, and working out how to use them correctly requires a bit of head scratching. One must find somewhere to store the weakref till after the referent is dead, and without accidentally keeping the referent alive. Then one must ensure that the callback frees the weakref (without leaving any remnant ref-cycles). When it is an option, using a __del__ method is far less hassle. 2. Callbacks invoked during interpreter shutdown are troublesome. For instance they can throw exceptions because module globals have been replaced by None. 3. Sometimes you want the callback to be called at program exit even if the referent is still alive. 4. Exceptions thrown in callbacks do not produce a traceback. This can make debugging rather awkward. (Maybe this is because printing tracebacks is problematic during interpreter shutdown?) (Note that problems 2-4 also apply to using __del__ methods.) If possible I would like to see the weakref module provide a finalize class to address these issues. Trivial usage might be >>> class Kenny: pass ... >>> kenny = Kenny() >>> finalize(kenny, print, "you killed kenny!") <finalize.finalize object at 0xffed4f2c> >>> del kenny you killed kenny! Prototype at https://gist.github.com/3208245 Cheers, Richard

5 7

only raise ImportError out of imports
by Eric Snow 31 Jul '12

31 Jul '12

Currently, if module 'eggs' has bad syntax and you import it from module 'spam', then you will get a SyntaxError in module spam: --------------------------------------------------------- $ cat > eggs.py << EOF a + EOF $ cat > spam.py << EOF import eggs EOF $ python -c 'import spam' Traceback (most recent call last): File "spam.py", line 2, in <module> import eggs File "/tmp/eggs.py", line 1 a + ^ SyntaxError: invalid syntax --------------------------------------------------------- My proposal is that, instead, an error during import always results in an ImportError. So in the above example, the error would have been an ImportError rather than a SyntaxError. However, rather than ignore the cause of the import error, we would use exception chaining to save the SyntaxError. To get this effect currently would look something like this: --------------------------------------------------------- $ cat > spam.py << EOF try: import eggs except SyntaxError as e: raise ImportError("failed to import eggs") from e EOF $ python -c 'import spam' Traceback (most recent call last): File "spam.py", line 2, in <module> import eggs File "/tmp/eggs.py", line 1 a + ^ SyntaxError: invalid syntax The above exception was the direct cause of the following exception: Traceback (most recent call last): File "spam.py", line 4, in <module> raise ImportError("failed to import eggs") from e ImportError: failed to import eggs --------------------------------------------------------- With exception chaining the SyntaxError is stored on the ImportError: --------------------------------------------------------- $ cat > ham.py << EOF try: import spam except ImportError as e: print("cause: {}".format(e.__cause__)) print("context: {}".format(e.__context__)) EOF $ python ham.py cause: invalid syntax (eggs.py, line 1) context: invalid syntax (eggs.py, line 1) --------------------------------------------------------- The main benefit of this change would be to isolate errors during import to the module where they happen. The main challenge would be with backward compatibility, so I won't hold my breath to see this in Python 3. While all feedback is welcome, I'd particularly like to hear from those that actually use non-ImportError exceptions that come out of the import statement. Thanks! -eric

3 3

Python Environment Initialization Scripts
by James William Pye 31 Jul '12

31 Jul '12

I posted a question on c.l.py[1] looking for a feature and I suspect it doesn't exist. Some quick searches on -ideas and -dev didn't reveal anything either. Has the idea of [python] environment initialization been pitched/rejected before or does it already exist? Simply, I'm looking for a mechanism to automatically load some meta_path hooks after site initialization. Without modifying libpython, I can achieve what I'm trying to do by: * Mangling the site.py file to run the code. (in my case, "import c.lib; c.lib.install()") * Overriding the python3 executable (not desirable; consider embedding) * Provide an alternate executable. (not desirable; again embedding and lack of availability in scripts pointing directly to python3) * LD_PRELOAD overrides? (portability problems abound, I imagine. mangling seems preferable) * Perform installation in the dependent (duplication of installation in each dependent) [1] https://groups.google.com/forum/?fromgroups#!topic/comp.lang.python/CCHe8_W…

3 7

Marvelous movable memoryview
by Matt Chaput 28 Jul '12

28 Jul '12

By which I mean a memoryview that lets you change the start and end offsets of its view of the underlying object (not modifying the underlying object). Let's say I want to slice a long string s into trigrams and write them to disk (or something). for i in xrange(0, len(s) - 3): x = s[i:i + 3] myfile.write(x) At each step the slice copies the bytes of the string, even though all I do is write them to disk. I could avoid copying with memoryviews... for i in xrange(0, len(s) - 3): x = memoryview(s)[i:i + 3] myfile.write(x) ...but this is actually much slower (3x slower in some quick tests). I'm guessing it's because of all the object creation (while string slicing probably uses fast paths). Shouldn't I be able to do this? m = memoryview(s) for i in xrange(0, len(s) - 3): m.start = i m.end = i + 3 myfile.write(m) Cheers, Matt

5 4

Package collections
by Daniel Holth 27 Jul '12

27 Jul '12

See http://good-py.appspot.com

3 3

Python Package Collections
by Daniel Fetchinson 24 Jul '12

24 Jul '12

Hi folks, here is an idea that came out of (not quite) thin air: Pypi, setuptools, distribute & Co are all very useful for finding, installing, upgrading, etc. of python packages. They even handle dependencies! Yaaaay! However, there is one more thing that could be built on top of them that would make them even more useful. This is the <drum roll please> Python Package Collections! This would be an online service to organize packages into collections allowing one to download, install, build, upgrade, etc. many packages at once. Say somebody finds package1, package2, package3, package4 very useful and figure they naturally belong together by topic. And so our test person always wants to have them all installed so when he downloads a new python version which will make his python arsenal not only 2.4 2.5 2.6 2.7 3.1 3.2 3.3 but also 3.4, he won't have to install all these packages separately again but he can just install the collection with one command or one click. You might ask why can't our test person just do "install package1" then "install package2" then "install package3" then "install package4"? Well, certainly he could do that but if he found that these four packages nicely go together chances are others will find it handy to have them all too. So our brilliant test person will want to share his collection with others telling them essentially "if you like working with X" or "you frequently develop for Y" then "you can just install my collection and you will be ready to go"! Anybody could organize packages into collections and I bet there would be very popular ones like "mathematics" including all sorts of math packages from pypi, or "pdfgoodies" including everything pdf related that one typically needs for work with pdf. These don't have to depend on each other (dependencies are handled quite well by existing tools), they just need to have the property that they are very handy to have together. To be honest the idea is not mine, take a look at https://addons.mozilla.org/en-US/firefox/collections/ how add-ons for firefox are organized into collections. It's basically the same idea but for python packages. This whole thing could be integrated into pypi or initially the idea could be tested on a separate site to see if it catches on. What do you all think? Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown

6 5

How can I get own instance of generator in itself code
by Đinh Nho Nam 24 Jul '12

24 Jul '12

Hi Python Software Foundation! I have learned Python for 2 years, and I love this language very much. The structure and grammar are simple and beautiful. I used this language in most of my projects. Now, I learn to use generator. This architecture is very intelligent. It allows programmer can pause and resume function. I think that this feature is not supported in classic language as C++, Java, C#.. In my design, we use 1 generator in both tasks: invoke and callback when download one url. But I have a small problem when getting instance of generator. How can I get own instance of generator in itself code? detail pseudo code example: def invoke_callback(url): do something here self = get myself (invoke_callback generator) instance #how can we do as this line???? data = yield asynchronous_dowload(url, callback = self.next) do something with data How can we do as this code. How can I get self variable?. Does the language support this feature? I cannot find it in any document in python.org. So, I have to solve it by wrapper it in a list. def invoke_callback(url, wrapper): self = wrapper[0] data = yield asynchronous_dowload(url, callback = self.next) do something with data wrapper = [] instance = invoke_callback(url, wrapper) wrapper.append(instance) instance.next() The other way is sending instance to itself, and we invoke twice. def invoke_callback(url): self = yield data = yield asynchronous_dowload(url, callback = self.next) do something with data instance = invoke_callback(url) instance.next() #first instance.send(instance) #second But the code in both of ways is not nice. Can you add this feature in the next version of python??? Regards Namdn

8 11

Atomic file.get(offset, length)
by Matt Chaput 23 Jul '12

23 Jul '12

I wish Python binary file objects had an atomic seek-read method, so I wouldn't have to perform my own locking everywhere to prevent other threads from moving the file pointer between seek and read. Is this something that can be bubbled up from the underlying platform? I think the Linux C equivalent is pread. I also think Java has something like this but can't find a reference now. Or does this exist and I missed it? (On a mmap file this is trivial, of course.) Has this been discussed before? Cheers, Matt

8 11

Provide a 'key' argument for all() and any() builtins
by Giampaolo Rodolà 23 Jul '12

23 Jul '12

This would be similar to 'key' argument already available for min(), max() and sorted() and would let user decide what must be considered True and what not. Some use cases: all(a, b, c, ... key=callable) all(a, b, c, ... key=lambda x: isinstance(x, int)) any(a, b, c, ... key=lambda x: x < 10) a = b = c = None for x in foo: if cond1: a = 0 elif cond2: b = 0 elif cond3: c = 0 # meaning all cond* has been satisfied and it's ok to consider 0 a meaningful value assert all(a, b, c, key=lambda x: x != None) Thoughts? --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/

5 5

Re: [Python-ideas] Where has packaging gone?
by Michele Lacchia 22 Jul '12

22 Jul '12

> > Thoughts attached. > Yeah that one is great! :) And true. No, the distutils2 project continues and is still being actively > developed. (distutils2 *is* packaging - the name on PyPI is just > different from the name that will be used when it is added to the > standard library). Bento is a project that aims to tackle part of > distutils that distutils2 hasn't really looked at yet: creating a more > flexible build system (or, more accurately, making it easier to hook > into *existing* build systems rather than defining yet another one). Ok. Now the distinction is clear. Good to hear disutils2 is still actively developed. Those PEPs are *already* standards and are actively supported by > external projects today. I just found the wheel project: https://bitbucket.org/dholth/wheel It tries to implement those standards by patching distribute and pip. I run the demo and I think the resulting package format is just amazing. In a nutshell is a binary format (extensions get compiled) which in my opinion is really flexible. I hope it will become standard and will be integrated in the major software someday.

1 0

__sort__ special member
by David Townshend 22 Jul '12

22 Jul '12

I was quite surprised to find that nobody seems to have suggested this before, because it seems like an obvious idea. Basically, add a special method __sort__ which, if specified, is used when sorted() is called. I can think of two immediate use cases: 1. When an object wants sorted() to return something other than a list, e.g. dict.__sort__ could return an OrderedDict. 2. When there is a more efficient method of sorting a specific sequence. E.g. sorting a range object should be trivial. Is there some obvious reason why nobody has suggested this before? Is it worth pursuing? David.

7 10

'' in 'abc' == True
by anatoly techtonik 21 Jul '12

21 Jul '12

I've just spotted inconsistency between string and lists handling: >>> '' in 'abc' True >>> '' in 'abc'.split() False >>> [] in ['a', 'b', 'c'] False Why strings here behave differently than other sequence types? Is that by design?

10 16

Introduce some obvious way to encode and decode filenames from Python code
by Sven Marnach 18 Jul '12

18 Jul '12

Currently, there is no obvious way to encode a filename in the default filesystem encoding. To pipe some filenames to the stdin of a subprocess, I effectively used encoded_name = file_name.encode(sys.getfilesystemencoding()) which mostly worked. There are cases where this fails, though: on Linux with LANG=C and filenames that contain non-ASCII characters, for example, or in any situation where the default filesystem encoding can't decode a filename. The correct way to do this seems to be something like if sys.platform == "nt": errors = "strict" else: errors = "surrogateescape" encoded_name = file_name.encode(sys.getfilesystemencoding() errors=errors) I think there should be (1) some documentation on the issue and (2) a more obvious way to do encode filenames. 1. The most useful reference I could find in the docs is http://docs.python.org/dev/c-api/unicode.html#file-system-encoding and there is a short paragraph at http://docs.python.org/dev/library/os.html#file-names-command-line-argument… The filename encoding applies to basically all Python library functions (including built-ins like `open()`) and should probably be documented at a more prominent spot. The "surrogateescape" error handler isn't mentioned here http://docs.python.org/dev/howto/unicode.html#unicode-filenames 2. There should be some way to access the C API functions for decoding and encoding filenames from Python. I don't have a good idea how to do this – maybe by adding a meta-encoding "filesystem", or by adding functions to the standard library. Did I miss something? Any thoughts? Cheers, Sven

8 13

Where has packaging gone?
by Nick Coghlan 16 Jul '12

16 Jul '12

(the list address on the original email I replied to was broken) On Mon, Jul 16, 2012 at 6:55 PM, Michele Lacchia <michelelacchia(a)gmail.com> wrote: > Hi, > I read this: > http://www.gossamer-threads.com/lists/python/dev/992979 > > but at the end nobody posted to it anymore. I would like to know what will > now happen, since as far as I understood, the development of packaging > has stopped and Bento (David Cournapeau library) has appeared. No, the distutils2 project continues and is still being actively developed. (distutils2 *is* packaging - the name on PyPI is just different from the name that will be used when it is added to the standard library). Bento is a project that aims to tackle part of distutils that distutils2 hasn't really looked at yet: creating a more flexible build system (or, more accurately, making it easier to hook into *existing* build systems rather than defining yet another one). (In fact, the lack of attention to the build system flaws is one of the big reasons distutils2/packaging was deemed unready for stdlib inclusion in 3.3) > So what will happen now? Packaging will continue the development? Or > will be Bento replace it? Personally I don't care which tool will become > standard, > but I do feel the need for a standard to be created. The point of having documented standards in PEP form is that there can (and will) be multiple projects that produce and consume those standard formats. > - PEP 345 (http://www.python.org/dev/peps/pep-0345/) > - PEP 376 (http://www.python.org/dev/peps/pep-0376/) > - PEP 386 (http://www.python.org/dev/peps/pep-0386/) Those PEPs are *already* standards and are actively supported by external projects today. One key missing piece is standard library support, which the packaging module is intended to provide. However, it was deemed that the project was not quite mature enough for 3.3, thus it will continue to be developed (with a faster iteration cycle) as distutils2, with a view to including a more mature version of the API for 3.4. This will likely take the form of additional PEPs that more clearly define what is to be added and *why* it is being added, with one focus being making it easier to build additional tools that produce and consume metadata in the standard formats, rather then being a comparatively unwieldy monolithic system like the original distutils. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia

1 0