Mailman 3 July 2012 - Python-ideas

changing sys.stdout encoding
by Rurpy 07 Aug '12

07 Aug '12

In my first foray into Python3 I've encountered this problem: I work in a multi-language environment. I've written a number of tools, mostly command-line, that generate output on stdout. Because these tools and their output are used by various people in varying environments, the tools all have an --encoding option to provide output that meets the needs and preferences of the output's ultimate consumers. In converting them to Python3, I found the best (if not very pleasant) way to do this in Python3 was to put something like this near the top of each tool[*1]: import codecs sys.stdout = codecs.getwriter(opts.encoding)(sys.stdout.buffer) What I want to be able to put there instead is: sys.stdout.set_encoding (opts.encoding) The former I found on the internet -- there is zero probability I could have figured that out from the Python docs. It is obscure to anyone (who has like me generally only needed to deal with .encode() and .decode()) who hasn't encountered it before or dealt much with the codecs module. It is excessively complex for what is conceptually a simple and straight-forward operation. It requires the import of the codecs module in programs that other- wise don't need it [*2], and the reading of the codecs docs (not a shining example of clarity themselves) to understand it. In short it is butt ugly relative to what I generally get in Python. Would it be feasible to provide something like .set_encoding() on textio streams? (Or make .encoding a writeable property?; it seems to intentionally be non-writeable for some reason but is that reason really unavoidable?) If doing this for textio in general is too hard, then what about encapsulating the codecs stuff above in a sys.set_encoding() function? Needing to change the encoding of a sys.std* stream is not an uncommon need and a user should not have to go through the codecs dance above to do so IMO. ---- [*1] There are other ways to change stdout's encoding but they all have problems AFAICT. PYTHONIOENCODING can't easily be changed dynamically within program. Reopening stdout as binary, or using the binary interface to text stdout, requires a explicit encode call at each write site. Overloading print() is obscure because it requires reader to notice print was overloaded. [*2] I don't mean the actual import of the codecs module which occurs anyway; I mean the extra visual and cognitive noise introduced by the presence of the import statement in the source.

16 33

Better support for finalization with weakrefs
by Richard Oudkerk 02 Aug '12

02 Aug '12

I would like to see better support for the use of weakrefs callbacks for object finalization. The current issues with weakref callbacks include: 1. They are rather low level, and working out how to use them correctly requires a bit of head scratching. One must find somewhere to store the weakref till after the referent is dead, and without accidentally keeping the referent alive. Then one must ensure that the callback frees the weakref (without leaving any remnant ref-cycles). When it is an option, using a __del__ method is far less hassle. 2. Callbacks invoked during interpreter shutdown are troublesome. For instance they can throw exceptions because module globals have been replaced by None. 3. Sometimes you want the callback to be called at program exit even if the referent is still alive. 4. Exceptions thrown in callbacks do not produce a traceback. This can make debugging rather awkward. (Maybe this is because printing tracebacks is problematic during interpreter shutdown?) (Note that problems 2-4 also apply to using __del__ methods.) If possible I would like to see the weakref module provide a finalize class to address these issues. Trivial usage might be >>> class Kenny: pass ... >>> kenny = Kenny() >>> finalize(kenny, print, "you killed kenny!") <finalize.finalize object at 0xffed4f2c> >>> del kenny you killed kenny! Prototype at https://gist.github.com/3208245 Cheers, Richard

5 7

only raise ImportError out of imports
by Eric Snow 01 Aug '12

01 Aug '12

Currently, if module 'eggs' has bad syntax and you import it from module 'spam', then you will get a SyntaxError in module spam: --------------------------------------------------------- $ cat > eggs.py << EOF a + EOF $ cat > spam.py << EOF import eggs EOF $ python -c 'import spam' Traceback (most recent call last): File "spam.py", line 2, in <module> import eggs File "/tmp/eggs.py", line 1 a + ^ SyntaxError: invalid syntax --------------------------------------------------------- My proposal is that, instead, an error during import always results in an ImportError. So in the above example, the error would have been an ImportError rather than a SyntaxError. However, rather than ignore the cause of the import error, we would use exception chaining to save the SyntaxError. To get this effect currently would look something like this: --------------------------------------------------------- $ cat > spam.py << EOF try: import eggs except SyntaxError as e: raise ImportError("failed to import eggs") from e EOF $ python -c 'import spam' Traceback (most recent call last): File "spam.py", line 2, in <module> import eggs File "/tmp/eggs.py", line 1 a + ^ SyntaxError: invalid syntax The above exception was the direct cause of the following exception: Traceback (most recent call last): File "spam.py", line 4, in <module> raise ImportError("failed to import eggs") from e ImportError: failed to import eggs --------------------------------------------------------- With exception chaining the SyntaxError is stored on the ImportError: --------------------------------------------------------- $ cat > ham.py << EOF try: import spam except ImportError as e: print("cause: {}".format(e.__cause__)) print("context: {}".format(e.__context__)) EOF $ python ham.py cause: invalid syntax (eggs.py, line 1) context: invalid syntax (eggs.py, line 1) --------------------------------------------------------- The main benefit of this change would be to isolate errors during import to the module where they happen. The main challenge would be with backward compatibility, so I won't hold my breath to see this in Python 3. While all feedback is welcome, I'd particularly like to hear from those that actually use non-ImportError exceptions that come out of the import statement. Thanks! -eric

3 3

Python Environment Initialization Scripts
by James William Pye 01 Aug '12

01 Aug '12

I posted a question on c.l.py[1] looking for a feature and I suspect it doesn't exist. Some quick searches on -ideas and -dev didn't reveal anything either. Has the idea of [python] environment initialization been pitched/rejected before or does it already exist? Simply, I'm looking for a mechanism to automatically load some meta_path hooks after site initialization. Without modifying libpython, I can achieve what I'm trying to do by: * Mangling the site.py file to run the code. (in my case, "import c.lib; c.lib.install()") * Overriding the python3 executable (not desirable; consider embedding) * Provide an alternate executable. (not desirable; again embedding and lack of availability in scripts pointing directly to python3) * LD_PRELOAD overrides? (portability problems abound, I imagine. mangling seems preferable) * Perform installation in the dependent (duplication of installation in each dependent) [1] https://groups.google.com/forum/?fromgroups#!topic/comp.lang.python/CCHe8_W…

3 7

Marvelous movable memoryview
by Matt Chaput 29 Jul '12

29 Jul '12

By which I mean a memoryview that lets you change the start and end offsets of its view of the underlying object (not modifying the underlying object). Let's say I want to slice a long string s into trigrams and write them to disk (or something). for i in xrange(0, len(s) - 3): x = s[i:i + 3] myfile.write(x) At each step the slice copies the bytes of the string, even though all I do is write them to disk. I could avoid copying with memoryviews... for i in xrange(0, len(s) - 3): x = memoryview(s)[i:i + 3] myfile.write(x) ...but this is actually much slower (3x slower in some quick tests). I'm guessing it's because of all the object creation (while string slicing probably uses fast paths). Shouldn't I be able to do this? m = memoryview(s) for i in xrange(0, len(s) - 3): m.start = i m.end = i + 3 myfile.write(m) Cheers, Matt

5 4

Package collections
by Daniel Holth 27 Jul '12

27 Jul '12

See http://good-py.appspot.com

3 3

Python Package Collections
by Daniel Fetchinson 25 Jul '12

25 Jul '12

Hi folks, here is an idea that came out of (not quite) thin air: Pypi, setuptools, distribute & Co are all very useful for finding, installing, upgrading, etc. of python packages. They even handle dependencies! Yaaaay! However, there is one more thing that could be built on top of them that would make them even more useful. This is the <drum roll please> Python Package Collections! This would be an online service to organize packages into collections allowing one to download, install, build, upgrade, etc. many packages at once. Say somebody finds package1, package2, package3, package4 very useful and figure they naturally belong together by topic. And so our test person always wants to have them all installed so when he downloads a new python version which will make his python arsenal not only 2.4 2.5 2.6 2.7 3.1 3.2 3.3 but also 3.4, he won't have to install all these packages separately again but he can just install the collection with one command or one click. You might ask why can't our test person just do "install package1" then "install package2" then "install package3" then "install package4"? Well, certainly he could do that but if he found that these four packages nicely go together chances are others will find it handy to have them all too. So our brilliant test person will want to share his collection with others telling them essentially "if you like working with X" or "you frequently develop for Y" then "you can just install my collection and you will be ready to go"! Anybody could organize packages into collections and I bet there would be very popular ones like "mathematics" including all sorts of math packages from pypi, or "pdfgoodies" including everything pdf related that one typically needs for work with pdf. These don't have to depend on each other (dependencies are handled quite well by existing tools), they just need to have the property that they are very handy to have together. To be honest the idea is not mine, take a look at https://addons.mozilla.org/en-US/firefox/collections/ how add-ons for firefox are organized into collections. It's basically the same idea but for python packages. This whole thing could be integrated into pypi or initially the idea could be tested on a separate site to see if it catches on. What do you all think? Cheers, Daniel -- Psss, psss, put it down! - http://www.cafepress.com/putitdown

6 5

How can I get own instance of generator in itself code
by Đinh Nho Nam 24 Jul '12

24 Jul '12

Hi Python Software Foundation! I have learned Python for 2 years, and I love this language very much. The structure and grammar are simple and beautiful. I used this language in most of my projects. Now, I learn to use generator. This architecture is very intelligent. It allows programmer can pause and resume function. I think that this feature is not supported in classic language as C++, Java, C#.. In my design, we use 1 generator in both tasks: invoke and callback when download one url. But I have a small problem when getting instance of generator. How can I get own instance of generator in itself code? detail pseudo code example: def invoke_callback(url): do something here self = get myself (invoke_callback generator) instance #how can we do as this line???? data = yield asynchronous_dowload(url, callback = self.next) do something with data How can we do as this code. How can I get self variable?. Does the language support this feature? I cannot find it in any document in python.org. So, I have to solve it by wrapper it in a list. def invoke_callback(url, wrapper): self = wrapper[0] data = yield asynchronous_dowload(url, callback = self.next) do something with data wrapper = [] instance = invoke_callback(url, wrapper) wrapper.append(instance) instance.next() The other way is sending instance to itself, and we invoke twice. def invoke_callback(url): self = yield data = yield asynchronous_dowload(url, callback = self.next) do something with data instance = invoke_callback(url) instance.next() #first instance.send(instance) #second But the code in both of ways is not nice. Can you add this feature in the next version of python??? Regards Namdn

8 11

Atomic file.get(offset, length)
by Matt Chaput 24 Jul '12

24 Jul '12

I wish Python binary file objects had an atomic seek-read method, so I wouldn't have to perform my own locking everywhere to prevent other threads from moving the file pointer between seek and read. Is this something that can be bubbled up from the underlying platform? I think the Linux C equivalent is pread. I also think Java has something like this but can't find a reference now. Or does this exist and I missed it? (On a mmap file this is trivial, of course.) Has this been discussed before? Cheers, Matt

8 11

Provide a 'key' argument for all() and any() builtins
by Giampaolo Rodolà 24 Jul '12

24 Jul '12

This would be similar to 'key' argument already available for min(), max() and sorted() and would let user decide what must be considered True and what not. Some use cases: all(a, b, c, ... key=callable) all(a, b, c, ... key=lambda x: isinstance(x, int)) any(a, b, c, ... key=lambda x: x < 10) a = b = c = None for x in foo: if cond1: a = 0 elif cond2: b = 0 elif cond3: c = 0 # meaning all cond* has been satisfied and it's ok to consider 0 a meaningful value assert all(a, b, c, key=lambda x: x != None) Thoughts? --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/

5 5