Mailman 3 November 2012 - Python-ideas

CLI option for isolated mode
by Christian Heimes 15 Nov '12

15 Nov '12

Hi everybody, I like to propose a new option for the Python interpreter: python -I It shall start the interpreter in isolated mode which ignores any environment variables set by the user and any files installed by the user. The mode segregate a Python program from anything an unpriviliged user is able to modify and uses only files that are installed by a system adminstrator. The isolated mode implies -E (ignore all PYTHON* environment vars) and -s (don't add user site directory). It also refrains from the inclusion of '' or getcwd() to sys.path. TKinter doesn't load and execute Python scripts from the user's home directory. Other parts of the stdlib should be checked, too. The option is intended for OS and application scripts that doesn't want to become affected by user installed files or files in the current working path of a user. The idea is motivated by a couple of bug reports, for example: https://bugs.launchpad.net/bugs/938869 lsb_release crashed with SIGABRT in Py_FatalError() http://bugs.python.org/issue16202 sys.path[0] security issues http://bugs.python.org/issue16248 Security bug in tkinter allows for untrusted, arbitrary code execution. Regards, Christian

11 12

CPS transform for Python
by Sam Rushing 10 Nov '12

10 Nov '12

The discussion last week with Greg Ewing got me to thinking about the CPS transform, and how it might be a useful technique for callback/event-driven code like asyncore & Twisted. I'm pretty sure when I first thought about this eons ago it was before Python had closures. They definitely make it a bit easier! I put some simple demo code together, I think it demonstrates that the idea is feasible, I'm curious to know if anyone is interested. Don't get hung up on the poor quality of the generated code, big improvements could be made with a little bit of work. https://github.com/samrushing/cps-python/ -Sam

3 3

sys.py3k
by anatoly techtonik 08 Nov '12

08 Nov '12

if sys.py3k: # some py2k specific code pass Why? 1. readable 2. traceable Explained: 1. self-explanatory 2. sys.version_info >= (3, 0) or sys.version[0] == '3' is harder to trace when you need to find all python 3 related hacks -- anatoly t.

9 11

Windows temporary file association for Python files
by anatoly techtonik 07 Nov '12

07 Nov '12

I wonder if it will make the life easier if Python was installed with .py association to "%PYTHON_HOME%\python.exe" "%1" %* It will remove the need to run .py scripts in virtualenv with explicit 'python' prefix. Example how it doesn't work right now E:\virtenv32\Scripts>echo import sys; print(sys.version) > test.py E:\virtenv32\Scripts>test.py 3.3.0 (v3.3.0:bd8afb90ebf2, Sep 29 2012, 10:55:48) [MSC v.1600 32 bit (Intel)] E:\virtenv32\Scripts>python test.py 3.2.3 (default, Apr 11 2012, 07:15:24) [MSC v.1500 32 bit (Intel)] If Python file association was specified with "%PYTHON_HOME%\python.exe" "%1" %* then virtualenv could override this variable when setting the environment to set correct executable for .py files. -- anatoly t.

5 6

Publishing ideas on ActiveState recipe site
by anatoly techtonik 07 Nov '12

07 Nov '12

On Tue, Nov 6, 2012 at 1:22 AM, Steven D'Aprano <steve(a)pearwood.info> wrote: > On 05/11/12 23:52, Ned Batchelder wrote: >> >> Get everything the way you want it, and then propose it. > > > +1 > > Also consider publishing it as a recipe on ActiveState, where many > people will view it, use it, and offer feedback. This has many > benefits: > > * You will gauge community interest; > > * Many eyeballs make bugs shallow; > > * You are providing a useful recipe that others can use, even > if it doesn't get included in the std lib. > > Some of the most useful parts of the std lib, like namedtuple, > started life on ActiveState. > > http://code.activestate.com/recipes/langs/python/ Why I don't use ActiveState: 1. StackOverflow is much easier to access - just one click to login with Google Account versus several clicks, data entry and copy/paste operations to remind the password on ActiveState - I want to login there with Python account 2. StackOverflow is problem search oriented - not recipe catalog oriented, which makes it better for solving problems, which I do more often than reading the recipe book (although I must admin when I was starting Python - the Cookbook from O'Reilly in CHM format was mega awesome) 3. I post the code as gists as it includes the notion of history, unlike ActiveState, which interface looks a little outdated - it was not obvious for me that recipes have history until today 4. Recipes are licensed, which is a too much of a burden for a snippet 5. ActiveState site makes it clear that it is ActiveState site - the 20% of my screen is taken by ActiveState header, so it looks like company site - not a site for community Otherwise the idea of community recipe site is very nice.

4 6

Async API: some code to review
by Guido van Rossum 07 Nov '12

07 Nov '12

I am finally ready to show the code I worked on for the past two weeks. This is definitely not ready for anything except as a quick demo, but I learned enough while writing it to feel comfortable with the PEP 380 paradigm. I've set up a Hg repo on code.google.com, and I picked a codename: tulip. View the code here: http://code.google.com/p/tulip/source/browse/ It runs on Linux and OSX; I have no easy access to Windows but I'd be happy to take contributions. Key files in the directory: - main.py: the main program for testing, and a rough HTTP client - sockets.py: transports for sockets and SSL, and a buffering layer - scheduling.py: a Task class and related stuff; this is where the PEP 380 scheduler is implemented - polling.py: an event loop and basic polling implementations for: select(), poll(), epoll(), kqueue() Other junk: .hgignore, Makefile, README, p3time.py (benchmark yield from vs. plain functions), longlines.py (stupid style checker) More detailed discussions per file follows; please read the code along with my description (separately they may not make much sense): polling.py: http://code.google.com/p/tulip/source/browse/polling.py I found it remarkably easy to come up with polling implementations using all those different system calls. I ended up mixing in the pollster class with the event loop class, although I'm not sure that's the best design -- perhaps it's better if the event loop just references the pollster as a separate object. The pollster has a very simple API: add_reader(fd, callback, *args), add_writer(<ditto>), remove_reader(fd), remove_writer(fd), and poll(timeout) -> list of events. (fd means file descriptor.) There's also pollable() which just checks if there are any fds registered. My implementation requires fd to be an int, but that could easily be extended to support other types of event sources. I'm not super happy that I have parallel reader/writer APIs, but passing a separate read/write flag didn't come out any more elegant, and I don't foresee other operation types (though I may be wrong). The event list started out as a tuple of (fd, flag, callback, args), where flag is 'r' or 'w' (easily extensible); in practice neither the fd nor the flag are used, and one of the last things I did was to wrap callback and args into a simple object that allows cancelling the callback; the add_*() methods return this object. (This could probably use a little more abstraction.) Note that poll() doesn't call the callbacks -- that's up to the event loop. The event loop has two basic ways to register callbacks: call_soon(callback, *args) causes callback(*args) to be called the next time the event loop runs; call_later(delay, callback, *args) schedules a callback at some time (relative or absolute) in the future. It also inherits add_reader() and add_writer() from the pollster. Then there is run(), which runs the event loop until there's nothing left to do (no readers, no writers, no soon or later callbacks), and run_once(), which goes through the entire list of event sources once. (I think the order in which I do this isn't quite right but it works for now.) Finally, there's a helper class (ThreadRunner) here which lets you run something in a separate thread using the features of concurrent.futures. It uses the "self-pipe trick" (Google it :-) to ensure that the poll() call wakes up -- this is needed by call_in_thread() at the next layer (scheduling.py). (There may be a race condition here, but I think it can be fixed.) Note that there are no yields (or yield froms) here; that's for the next layer: scheduling.py: http://code.google.com/p/tulip/source/browse/scheduling.py This is the scheduler for PEP-380 style coroutines. I started with a Scheduler class and operations along the lines of Greg Ewing's design, with a Scheduler instance as a global variable, but ended up ripping it out in favor of a Task object that represents a single stack of generators chained via yield-from. There is a Context object holding the event loop and the current task in thread-local storage, so that multiple threads can (and must) have independent event loops. Most user (and much library) code in this system should be written as generators invoking other generators directly using yield from. However to run something as an independent task, you wrap the generator call in a Task() constructor, possibly giving it a timeout, and then calling its start() method. A Task also acts a little like a future -- you can wait() for it, add done-callbacks, and it preserves the return value of the generator call. This can be used to introduce concurrency or to give something a separate timeout. (There are also primitives to wait for the first N completed of a bunch of Tasks.) To invoke a primitive I/O operation, you call the current task's block() method and then immediately yield (similar to Greg Ewing's approach). There are helpers block_r() and block_w() that arrange for a task to block until a file descriptor is ready for reading/writing. Examples of their use are in sockets.py. There is also call_in_thread() which integrates with polling.ThreadRunner to run a function in a separate thread and wait for it. Also used in sockets.py. In the docstrings I use the prefix "COROUTINE:" to indicate public APIs that should be invoked using yield from. sockets.py: http://code.google.com/p/tulip/source/browse/sockets.py This implements some internet primitives using the APIs in scheduling.py (including block_r() and block_w()). I call them transports but they are different from transports Twisted; they are closer to idealized sockets. SocketTransport wraps a plain socket, offering recv() and send() methods that must be invoked using yield from. SslTransport wraps an ssl socket (luckily in Python 2.6 and up, stdlib ssl sockets have good async support!). Then there is a BufferedReader class that implements more traditional read() and readline() coroutines (i.e., to be invoked using yield from), the latter handy for line-oriented transports. Finally there are some functions for connecting sockets, the highest-level one create_transport(). These use call_in_thread() to run socket.getaddrinfo() in a thread (this provides IPv6 support). I don't particularly care about the exact abstractions in this module; they are convenient and I was surprised how easy it was to add SSL, but still these mostly serve as somewhat realistic examples of how to use scheduling.py. (Afterthought: I think the SocketTransport's recv() and send() methods could be made more similar to SslTransport.) More examples in the final file: main.py: http://code.google.com/p/tulip/source/browse/main.py There is a simplistic HTTP client here built on top of the sockets.*Transport abstractions. And the main code exercises this by spawning four tasks fetching a variety of URLs (more when you uncomment a block of code) and waiting for their results. The code is a bit of a mess because I used it as a place to try out various APIs. I'm most interested in feedback on the design of polling.py and scheduling.py, and to a lesser extent on the design of sockets.py; main.py is just an example of how this style works out in practice. Sorry for the brain-dump style; I would like to write it all up better, but at the same time waiting longer doesn't necessarily make it better, so here it is, for all to see. (I also have a list of problems I had to debug during the development and what I learned from that; but that's too raw to post right now.) -- --Guido van Rossum (python.org/~guido)

20 96

os.path.commonpath()
by Serhiy Storchaka 07 Nov '12

07 Nov '12

See http://bugs.python.org/issue10395. os.path.commonpath() should be a function which returns right longest common sub-path for specified paths (os.path.commonprefix() is completely useless for this). There are some open questions about details of *right* behavior. What should be a common prefix of '/var/log/apache2' and '/var//log/mysql'? What should be a common prefix of '/usr' and '//usr'? What should be a common prefix of '/usr/local/' and '/usr/local/'? What should be a common prefix of '/usr/local/' and '/usr/local/bin'? What should be a common prefix of '/usr/bin/..' and '/usr/bin'? Please, those who are interested in this feature, give consistent answers to these questions.

6 10

The "in"-statement
by Markus Unterwaditzer 07 Nov '12

07 Nov '12

This is my first post on this mailinglist and i haven't lurked a long time yet, so please be gentle. While mocking objects, i got annoyed by the following code pattern i had to use when modifying multiple attributes on a single object:: obj.first_attr = "value" obj.second_attr = "value2" some_other = "lel" I thought it would be neat if i could do:: in obj: first_attr = "value" second_attr = "value2" some_other = "lel" # indenting this would cause it to appear as an attribute of obj Just a vague idea. Tell me what you think. -- Markus

13 14

Code snippet signature (Was: Publishing ideas on ActiveState recipe site)
by anatoly techtonik 07 Nov '12

07 Nov '12

Speaking about snippets in the thread about ActiveState recipe site reminded me of this idea. Snippet signature is a checksum over normalized source code with stripped comments. The signature looks like `# snippet:site:hash` added before function as a comment. Normalized means that code is unindented and PEPified. Comments are also stripped as they may be custom. Signatures are useful to: 1. identify piece of code repeatably used in Python apps for potential inclusion in Python library 2. measure popularity of snippets using automated code analysis 3. automatically check for updates using lookup on the given hash 4. automatically check if local code has improvements over the original snippet by recalculating and comparing hash 5. lookup if snippets were integrated into standard (or some other library) 6. compile a Python library with mostly reused snippets on the server side -- anatoly t. On Wed, Nov 7, 2012 at 9:46 AM, anatoly techtonik <techtonik(a)gmail.com> wrote: > On Tue, Nov 6, 2012 at 1:22 AM, Steven D'Aprano <steve(a)pearwood.info> wrote: >> On 05/11/12 23:52, Ned Batchelder wrote: >>> >>> Get everything the way you want it, and then propose it. >> >> >> +1 >> >> Also consider publishing it as a recipe on ActiveState, where many >> people will view it, use it, and offer feedback. This has many >> benefits: >> >> * You will gauge community interest; >> >> * Many eyeballs make bugs shallow; >> >> * You are providing a useful recipe that others can use, even >> if it doesn't get included in the std lib. >> >> Some of the most useful parts of the std lib, like namedtuple, >> started life on ActiveState. >> >> http://code.activestate.com/recipes/langs/python/ > > Why I don't use ActiveState: > > 1. StackOverflow is much easier to access - just one click to login > with Google Account versus several clicks, data entry and copy/paste > operations to remind the password on ActiveState - I want to login > there with Python account > 2. StackOverflow is problem search oriented - not recipe catalog > oriented, which makes it better for solving problems, which I do more > often than reading the recipe book (although I must admin when I was > starting Python - the Cookbook from O'Reilly in CHM format was mega > awesome) > 3. I post the code as gists as it includes the notion of history, > unlike ActiveState, which interface looks a little outdated - it was > not obvious for me that recipes have history until today > 4. Recipes are licensed, which is a too much of a burden for a snippet > 5. ActiveState site makes it clear that it is ActiveState site - the > 20% of my screen is taken by ActiveState header, so it looks like > company site - not a site for community > > Otherwise the idea of community recipe site is very nice.

1 0

os.path.split(path, maxsplit=1)
by anatoly techtonik 07 Nov '12

07 Nov '12

Why? Because it is critical when comparing paths and non-trivial. http://stackoverflow.com/questions/4579908/cross-platform-splitting-of-path… http://stackoverflow.com/questions/3167154/how-to-split-a-dos-path-into-its… http://www.gossamer-threads.com/lists/python/dev/654410 -- anatoly t.

3 5