Mailman 3 June 2012 - Python-ideas

functools.partial
by Yury Selivanov 09 Jun '12

09 Jun '12

Hello, While I was working on adding support for 'functools.partial' in PEP 362, I discovered that it doesn't do any sanity check on passed arguments upon creation. Example: def foo(a): pass p = partial(foo, 1, 2, 3) # this line will execute p() # this line will fail Is it a bug? Or is it a feature, because we deliberately don't do any checks because of performance issues? If the latter - I think it should be at least documented. - Yury

3 3

SysLogHandler: gratuitous data loss
by Jacek Masiulaniec 08 Jun '12

08 Jun '12

Hello, In logging.handlers, SysLogHandler defaults to localhost:514. http://docs.python.org/py3k/library/logging.handlers.html#logging.handlers.… http://docs.python.org/library/logging.handlers.html#logging.handlers.SysLo… In practice, there are systems out there that offer local syslog service via additional endpoints, for example: /dev/log /var/run/syslog Some systems even ship with UDP endpoint disabled by default, in which case Python's default is to drop data despite the availability of these other endpoints. The /dev/log path in particular is so commonplace that many system-level utils default to it. Other languages' syslog libraries provide support for it, too. [1] [2] I propose a change to SysLogHandler's default behavior: 1) Try connect(2) against the socket files. 2) Use localhost:514 as a fallback. I believe it's possible to change this interface while remaining backwards-compatible. Thoughts? Jacek [1] http://hackage.haskell.org/packages/archive/hslogger/1.0.7/doc/html/src/Sys… [2] http://golang.org/src/pkg/log/syslog/syslog_unix.go

1 0

Supporting already opened sockets in our socket-based server classes
by Tarek Ziadé 06 Jun '12

06 Jun '12

Hello What about allowing all our socket servers -- from SocketServer to WSGIServer, to run with an existing socket. The use case is to make it easier to write applications that use the pre-fork model to run several processes against the same socket. Basically: - the main process creates a socket, binds it and listen to it - the main process forks some subprocesses and pass them the socket fd value - each subprocess recreates a socket object using socket.fromfd() -- so it does not bind it - each subprocess can accept() connection on the socket I have a working prototype here : https://github.com/tarekziade/chaussette/blob/master/chaussette/server.py (don't look at the code I made it quickly just as a proof of concept) What I am proposing is the following syntax: if the host passed to the class is of the form: fd://12 The class will try to create a socket object against the file descriptor 12, and will not bind() it neither accept() it. How does that sounds ? If people like the idea I can try to build a patch for 3.x, and I can certainly release a backport for 2.x Cheers Tarek

3 5

Re: [Python-ideas] changing sys.stdout encoding
by Rurpy 06 Jun '12

06 Jun '12

On 06/05/2012 05:56 PM, MRAB wrote: > On 06/06/2012 00:34, Victor Stinner wrote: >> 2012/6/5 Rurpy<rurpy-/E1597aS9LQAvxtiuMwx3w(a)public.gmane.org>: >>> In my first foray into Python3 I've encountered this problem: >>> I work in a multi-language environment. I've written a number >>> of tools, mostly command-line, that generate output on stdout. >>> Because these tools and their output are used by various people >>> in varying environments, the tools all have an --encoding option >>> to provide output that meets the needs and preferences of the >>> output's ultimate consumers. >> >> What happens if the specified encoding is different than the encoding >> of the console? Mojibake? >> >> If the output is used as in the input of another program, does the >> other program use the same encoding? >> >> In my experience, using an encoding different than the locale encoding >> for input/output (stdout, environment variables, command line >> arguments, etc.) causes various issues. So I'm curious of your use >> cases. >> >>> In converting them to Python3, I found the best (if not very >>> pleasant) way to do this in Python3 was to put something like >>> this near the top of each tool[*1]: >>> >>> import codecs >>> sys.stdout = codecs.getwriter(opts.encoding)(sys.stdout.buffer) >> >> In Python 3, you should use io.TextIOWrapper instead of >> codecs.StreamWriter. It's more efficient and has less bugs. >> >>> What I want to be able to put there instead is: >>> >>> sys.stdout.set_encoding (opts.encoding) >> >> I don't think that your use case merit a new method on >> io.TextIOWrapper: replacing sys.stdout does work and should be used >> instead. TextIOWrapper is generic and your use case if specific to >> sys.std* streams. >> >> It would be surprising to change the encoding of an arbitrary file >> after it is opened. At least, I don't see the use case. >> > [snip] > > And if you _do_ want multiple encodings in a file, it's clearer to open > the file as binary and then explicitly encode to bytes and write _that_ > to the file. But is it really? The following is very simple and the level of python expertise required is minimal. It (would) works fine with redirection. One could substitute any other ordinary open (for write) text file for sys.stdout. [off the top of my head] text = 'This is %s text: 世界へ、こんにちは！' sys.stdout.set_encoding ('sjis') print (text % 'sjis') sys.stdout.set_encoding ('euc-jp') print (text % 'euc-jp') sys.stdout.set_encoding ('iso2022-jp') print (text % 'iso2022-jp') As for your suggestion, how do I reopen sys.stdout in binary mode? I don't need to do that often and don't know off the top of my head. (And it's too late for me to look it up.) And what happens to redirected output when I close and reopen the stream? I can open a regular filename instead. But remember to make the last two opens with "a" rather than "w". And don't forget the "\n" at the end of the text line. Could you show me an code example of your suggestion for comparison? Disclaimer: As I said before, I am not particularly advocating for a for a set_encoding() method -- my primary suggestion is a programatic way to change the sys.std* encodings prior to first use. Here I am just questioning the claim that a set_encoding() method would not be clearer than existing alternatives.

1 0

Re: [Python-ideas] changing sys.stdout encoding
by Rurpy 06 Jun '12

06 Jun '12

On 06/05/2012 05:34 PM, Victor Stinner wrote: > 2012/6/5 Rurpy <rurpy(a)yahoo.com>: >> In my first foray into Python3 I've encountered this problem: >> I work in a multi-language environment. I've written a number >> of tools, mostly command-line, that generate output on stdout. >> Because these tools and their output are used by various people >> in varying environments, the tools all have an --encoding option >> to provide output that meets the needs and preferences of the >> output's ultimate consumers. > > What happens if the specified encoding is different than the encoding > of the console? Mojibake? When output is directed to te console, yes. Would one expect something else? > If the output is used as in the input of another program, does the > other program use the same encoding? Yes of course (when not misused). That's why they have --encoding options. (Obviously details vary depending on requirements of the various tools.) > In my experience, using an encoding different than the locale encoding > for input/output (stdout, environment variables, command line > arguments, etc.) causes various issues. So I'm curious of your use > cases. I gave the use case in my original post: + I work in a multi-language environment. I've written a number + of tools, mostly command-line, that generate output on stdout. + Because these tools and their output are used by various people + in varying environments, the tools all have an --encoding option + to provide output that meets the needs and preferences of the + output's ultimate consumers. They are often used like: ./extractor.py --encoding=euc-jp dataset >somefile <send somefile to some user who uses euc-jp data> And of course some tools require something similar for stdin encodings. >> In converting them to Python3, I found the best (if not very >> pleasant) way to do this in Python3 was to put something like >> this near the top of each tool[*1]: >> >> import codecs >> sys.stdout = codecs.getwriter(opts.encoding)(sys.stdout.buffer) > > In Python 3, you should use io.TextIOWrapper instead of > codecs.StreamWriter. It's more efficient and has less bugs. Thanks, I'll do that. But surely this is a strong argument for encapsulating the ability to change (or reinitialize) the std* encodings. I did fair amount of searching on the internet (many orders of magnitude more time that it would have taken to look up sys.stdout.set_encoding() in the documentation) and *still* ended up with a suboptimal solution. >> What I want to be able to put there instead is: >> >> sys.stdout.set_encoding (opts.encoding) > > I don't think that your use case merit a new method on > io.TextIOWrapper: replacing sys.stdout does work and should be used > instead. TextIOWrapper is generic and your use case if specific to > sys.std* streams. > > It would be surprising to change the encoding of an arbitrary file > after it is opened. At least, I don't see the use case. I gave a couple that I encountered in the past, in my response to Steven Turnbull. However, now I am more concerned with just resetting the encoding at the beginning of the program. > For example, tokenize.open() opens a Python source code file with the > right encoding. It starts by reading the file in binary mode to detect > the encoding, and then use TextIOWrapper to get a text file without > having to reopen the file. It would be possible to start with a text > file and then change the encoding, but it would be less elegant. That's a rather different use case than mine, yes? >> sys.stdout = codecs.getwriter(opts.encoding)(sys.stdout.buffer) > > You should also flush sys.stdout (and maybe also sys.stdout.buffer) > before replacing it. > >> It requires the import of the codecs module in programs that other- >> wise don't need it [*2], and the reading of the codecs docs (not >> a shining example of clarity themselves) to understand it. > > It's maybe difficult to change the encoding of sys.stdout at runtime > because it is NOT a good idea :-) Why would that be? My tools already do that, they meet their usability requirements and I have noticed no ill effects. The code (except for the piece I am complaining about) is about as simple and obvious as it is possible to get. Am I missing something? >> Needing to change the encoding of a sys.std* stream is not an >> uncommon need and a user should not have to go through the >> codecs dance above to do so IMO. > > Replacing sys.std* works but has issues: output written before the > replacement is encoded to a different encoding for example. The best > way is to change your locale encoding (using LC_ALL, LC_CTYPE or LANG > environment variable on UNIX), or simply to set PYTHONIOENCODING > environment variable. Those solutions are not only NOT the best solution (IMO) -- they are completely unacceptable. If I had to build my programs as shell scripts that manipulate environment variables before calling my Python program, I would dump Python for some other language. >> [*1] There are other ways to change stdout's encoding but they >> all have problems AFAICT. PYTHONIOENCODING can't easily be >> changed dynamically within program. > > Ah? Detect if PYTHONIOENCODING is present (or if sys.stdout.encoding > is the requested encoding), if not: restart the program with > PYTHONIOENCODING=encoding. For what I need to do (print() to sys.stdout with a different encoding than what Python guessed I'd want), your proposal seems absurdly convoluted to me. sys.stdout is set to encoding A. I want it to write using encoding B. The obvious, simplest, most desirable solution (barring technical difficulties) is just change the encoding. >> Overloading print() is obscure >> because it requires reader to notice print was overloaded. > > Why not writing the output into a file, instead of stdout? Because the interface for these tools already exists and the users of the tools are happy with them the way they are. And even if that weren't the case, it is not the role of a general purpose programming language to say a standard convention such as file redirection should be relegated to second-class status simply because the programmer needs a different output encoding than the language designers thought he would.

1 0

shutil.run (Was: shutil.runret and shutil.runout)
by anatoly techtonik 05 Jun '12

05 Jun '12

Hello again, I've finally found some time to partially process the replies and came up with a better solution than subprocess.* and shutil.runret/runout Disclaimer: I don't say that suprocess suxx - it is powerful and very awesome under the hood. What I want to say that its final user interface is awful - for such complex thing as this it should have been passed through several iteration cycles before settling down. Therefore, inspired by Fabric API, I've finally found the solution - shutil.run() function: https://bitbucket.org/techtonik/shutil-run/src run(command, combine_stderr=True): Run command through a system shell, return output string with additional properties: output.succeeded - result of the operation True/False output.return_code - specific return code output.stderr - stderr contents if combine_stderr=False `combine_stderr` if set, makes stderr merged into output string, otherwise it will be available as `output.stderr` attribute. Example: from shellrun import run output = run('ls -la') if output.succeeded: print(output) else: print("Error %s" % output.return_code) That's the most intuitive way I found so far. Objective advantages: 1. Better than subprocess.call(cmd, shell=true) subprocess.check_call(cmd, shell=true) subprocess.check_output(cmd, shell=True) because it is just shutil.run(cmd) i.e. short, simple and _easy to remember_ 2. With shutil.run() you don't need to rewrite your check_call() or check_output() with Popen() if you need to get return_code in addition to stderr contents on error 3. shutil.run() is predictable and consistent - its arguments are not dependent on each other, their combination doesn't change the function behavior over and over requiring you iterate over the documentation and warnings again and again 4. shutil.run() is the correct next level API over subprocess base level. subprocess executes external process - that is its role, but automatic ability to execute external process inside another external process (shell) looks like a hack to me. Practical, but still a hack. 5. No required exception catching, which doesn't work for shell=True anyway 6. No need to learn subprocess.PIPE routing magic (not an argument for hackers, I know) Subjective advantages: 1. More beautiful 2. More simple 3. More readable 4. Practical 5. Obvious 6. It easy to explain Hopefully, it can find its way in stdlib instead of http://shell-command.readthedocs.org/ -- anatoly t.

6 13

Expose Linux-specific APIs in resource module
by Giampaolo Rodolà 04 Jun '12

04 Jun '12

>From "man getrlimit" we have 5 linux-specific constants which are currently not exposed by resource module: RLIMIT_MSGQUEUE RLIMIT_NICE RLIMIT_RTPRIO RLIMIT_RTTIME RLIMIT_SIGPENDING Also, we have prlimit(), which is useful to get/set resources in a per-process fashion based on process PID. If desirable I can submit a patch for this. --- Giampaolo http://code.google.com/p/pyftpdlib/ http://code.google.com/p/psutil/ http://code.google.com/p/pysendfile/

1 0

Adding list.pluck()
by Cenk Altı 02 Jun '12

02 Jun '12

Hello All, pluck() is a beautiful function which is in underscore.js library. Described as "A convenient version of what is perhaps the most common use-case for map: extracting a list of property values." http://documentcloud.github.com/underscore/#pluck What about it implementing for python lists? And maybe for other iterables?

12 15

setprofile and settrace inconsistency
by Alon Horev 02 Jun '12

02 Jun '12

Hi, When setting a trace function with settrace, the trace function when called with a new scope can return another trace function or None, indicating the inner scope should not be traced. I used settrace for some time but calling the trace function for every line of code is a performance killer. So I moved on to setprofile, which calls a trace function every function entry/exit. now here's the problem: the return value from the trace function is ignored (intentionally), denying the possibility to skip tracing of 'hot' or 'not interesting' code. I would like to propose two alternatives: 1. setprofile will not ignore the return value and mimic settrace's behavior. 2. setprofile is just a wrapper around settrace that limits it's functionality, lets make settrace more flexible so setprofile will be redundant. here's how: settrace will recieve an argument called 'events', the trace function will fire only on events contained in that list. for example: setprofile = partial(settrace, events=['call', 'return']) I personally prefer the second. Some context to this issue: I'm building a python tracer - a logger that records each and every function call. In order for it to run in production systems, the overhead should be minimal. I would like to allow the user to say which function/module/classes to trace or skip, for example: the user will skip all math/cpu intensive operations. another example: the user will want to trace his django app code but not the django framework. your thoughts? Thanks, Alon Horev

2 1