Mailman 3 April 2003 - Python-Dev

RE: [Python-checkins] python/dist/src/Modules gcmodule.c,2.33.6.5,2.33.6.6
by Tim Peters April 4, 2003

April 4, 2003

[jhylton(a)users.sourceforge.net] > Modified Files: > Tag: release22-maint > gcmodule.c > Log Message: > Fix memory corruption in garbage collection. > ... > The problem with the previous revision is that it followed > gc->gc.gc_next before calling has_finalizer(). If has_finalizer() > gc->happened to deallocate the object FROM_GC(gc->gc.gc_next), then > the next time through the loop gc would point to freed memory. The > fix is to always follow the next pointer after calling > has_finalizer(). Oops! I didn't see this before posting my "Boom" msg. > Note that Python 2.3 does not have this problem, because > has_finalizer() checks the tp_del slot and never runs Python code. That part isn't so, alas: the program I posted in the "Boom" msg crashes 2.3, via the same mechanism: return PyInstance_Check(op) ? PyObject_HasAttr(op, delstr) : PyType_HasFeature(op->ob_type, Py_TPFLAGS_HEAPTYPE) ? op->ob_type->tp_del != NULL : 0; It's the PyInstance_Check(op) path there that's still vulnerable. I'll poke at that. > Tim, Barry, and I peed away the better part of two days tracking this > down. > ! next = gc->gc.gc_next; > if (has_finalizer(op)) { > gc_list_remove(gc); > gc_list_append(gc, finalizers); > gc->gc.gc_refs = GC_MOVED; > } > } > } > --- 277,290 ---- > for (; gc != unreachable; gc=next) { > PyObject *op = FROM_GC(gc); > ! /* has_finalizer() may result in arbitrary Python > ! code being run. */ > if (has_finalizer(op)) { > + next = gc->gc.gc_next; > gc_list_remove(gc); > gc_list_append(gc, finalizers); > gc->gc.gc_refs = GC_MOVED; > } > + else > + next = gc->gc.gc_next; > } > } Are we certain that has_finalizer() can't unlink gc itself from the unreachable list? If it can, then > + else > + next = gc->gc.gc_next; will set next to the content of free()ed memory. In fact, I believe the Boom program will suffer this fate ... yup, it does. "The problem" isn't yet really fixed in any version of Python, although I agree it's a lot better with the change above.

1 0

Boom
by Tim Peters April 4, 2003

April 4, 2003

While enduring dental implant surgery earlier today, I thought to myself "oops -- I bet this program will crash Python". Turns out it does, in current CVS, and almost certainly in every version of Python since cyclic gc was added: """ import gc class C: def __getattr__(self, attr): del self.attr raise AttributeError a = C() b = C() a.attr = b b.attr = a del a, b gc.collect() """ Short course: a and b are in a trash cycle. gcmodule's move_finalizers() finds one of them and calls has_finalizer() to see whether it's collectible. Say it's b. has_finalizer() calls (in effect) hasattr(b, "__del__"), and b.__getattr__() deletes b.attr as a side effect before saying b.__del__ doesn't exist. That drops the refcount on a to 0, which in turn drops the refcount on a.__dict__ to 0. Those two are the killers: a and a.__dict__ become untracked (by gc) as part of cleaning them up, but the move_finalizers() "next" local still points to one of them (to the __dict__, in the run I happened to step thru). As a result, the next trip around the move_finalizer() loop calls has_finalizer() on memory that's already been free()ed. Hilarity ensues. The anesthesia is wearing off and I won't speculate about solutions now. I suspect it's easy, or close to intractable. PLabs folks, I'm unsure whether this relates to the ZODB test failure we've been bashing away at. All, ZODB is a persistent database, and at one point in this test gc determines that "a ghost" is unreachable. When gc's has_finalizer() asks whether the ghost has a __del__ method, the persistence machinery kicks in, sucking the ghost's state off of disk, and executing a lot of Python code as a result. Part of the Python code executed does appear (if hazy memory serves) to delete some previously unreachable objects that were also in (or hanging off of) the ghost's cycle, and so in the unreachable list gc's move_finalizers() is crawling over. The kind of blowup above could be one bad effect, and Jeremy was seeing blowups with move_finalizers() in the traceback. Unfortunately, the test doesn't blow up under CVS Python, and 2.2.2 doesn't have the telltale 0xdbdbdbdb filler 2.3's debug PyMalloc sprays into free()ed memory.

1 0

Distutils documentation amputated in 2.2 docs?
by Greg Ewing April 3, 2003

April 3, 2003

I was looking at the Distributing Python Modules section of the distutils docs for 2.2 the other day, and it mentioned a section about extending the distutils, but there did not appear to be any such section. Further investigation revealed that the 1.6 version of the docs *does* have this section, as section 8, but somewhere between the 1.6 and 2.2 docs, this section has disappeared, along with almost all of section 9, "Reference", which now appears as section 7, but with only a small part of what it should contain. What's the proper way of submitting a bug report about this? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg(a)cosc.canterbury.ac.nz +--------------------------------------+

6 14

fwd: Dan Sugalski on continuations and closures
by Kevin Altis April 3, 2003

April 3, 2003

via Simon Willison's blog: http://simon.incutio.com/archive/2003/04/03/#closuresAndContinuations " Thanks to Dan Sugalski (designer of Parrot, the next generation Perl VM) I finally understand what continuations and closures actually are. He explains them as part of a comparison between the forthcoming Parrot and two popular virtual machines already in existence: * (Perl|python|Ruby) on (.NET|JVM) leads in to the explanation. http://www.sidhe.org/~dan/blog/archives/000151.html * The reason for Parrot, part 2 explains closures. http://www.sidhe.org/~dan/blog/archives/000152.html * Continuations and VMs explains continuations. http://www.sidhe.org/~dan/blog/archives/000156.html * Continuations and VMs, part 2 rounds things off by explaining why the JVM and the CLR are unsuitable environments for supporting these language features. http://www.sidhe.org/~dan/blog/archives/000157.html " ka ps. In order to focus on Python promotion and site-redesign efforts I've suspended delivery of python-dev email in the short-term and will only be scanning the archives as time permits. If you need to flame me, please address your emails to me directly or /dev/null, your choice ;-)

1 0

Re: [Python-Dev] python-dev Summary for 2003-03-16 through 2003-03-31
by Zooko April 3, 2003

April 3, 2003

> > > One point made about capabilities is that they partially go against the > > > Pythonic grain. ... > > If capabilities were implemented as Python references, you could inherit > > capabilities (== references) from superclasses, just as you can currently do. > > That's why it says "shouldn't" instead of "couldn't". I could re-word > this to go more along the way Ping phrased it in how the class statement > does not make perfect sense for capabilities but it can be used. I can't speak for Ping, but I would be quite surprised if he thought that capabilities were un-Pythonic. (I wouldn't be surprised if he disapproved of the notion of classes in a programming language, regardless of security considerations...) Speaking for myself, capabilities have two main advantages: they fit with the Zen of Python, they enable higher-order least-privilege, and they fit with the principle of unifying designation and authority. But seriously, I feel that capabilities fit with normal Python programming as it is currently practiced. Regards, Zooko http://zooko.com/ ^-- under re-construction: some new stuff, some broken links

2 1

Capabilities
by Guido van Rossum April 3, 2003

April 3, 2003

[Moving a discussion about capabilities to where it arguably belongs] [Ben Laurie] > The point about capabilities is that mere possession of a capability is > all that is required to exercise it. If you start adding security > checkers to them, then you don't have capabilities anymore. But the > point is somewhat deeper that than - given capabilities, you can > implement proxies without requiring any more infrastructure - you can > also implement security schemes that don't really correspond to any kind > of security checking at all (ok, you can probably find some convoluted > way to achieve the same effect, but I'll bet it comes down to having > tokens that correspond to proxies, and security checkers that allow you > to proceed if you have the appropriate token - in other words, > capabilities, but very hard to use). > > So, it seems to me, its simpler and more powerful to start with > capabilities and build proxies on top of them (or whatever alternate > scheme you want to build). > > Once more, my apologies for not just getting straight to the point. > > BTW, if you would like to explain why you don't think bound methods are > the way to go on python-dev, I'd love to hear it. It seems to e a matter of convenience. Often objects have many methods to which you want to provide access as a group. E.g. I might have a service configuration registry object. The object behaves roughly like a dictionary. A certain user may be given read-only access to the registry. Using capabilities, I would have to hand her a bunch of capabilities for various methods: __getitem__, has_key, get, keys, items, values, and many more. Using proxies I can simply give her a read-only proxy for the object. So proxies are more powerful. Before you start saying that we should use capabilities as the more fundamental mechanism and build proxies on top of that: as you point out, we already have an equivalent more fundamental mechanism, bound methods, which is equivalent to capabilities. It's just that raw capabilities aren't very usable, so one way or another we've got to build something on top of that. --Guido van Rossum (home page: http://www.python.org/~guido/)

12 67

Re: [Python-Dev] Capabilities
by Zooko April 3, 2003

April 3, 2003

(I, Zooko, wrote the lines prepended with "> > ".) Ben Laurie wrote: > > > In the capability way of life, it is still the case that access to the ZipFile > > class gives you the ability to open files anywhere in the system! (That is: I'm > > assuming for now that we implement capabilities without re-writing every > > dangerous class in the Library.) ... > It would probably be helpful to explain what you (or, at least, I) would > do if you (I) were writing from scratch, rather then "taming" the > existing libraries. In this case, Zipfile would require a file > capability to be passed to it at construction time, and so would become > non-dangerous, which is, I think, where Guido is coming from. Thank you. You are right about how I would do it, and I think you are right that this fits with Guido's approach, too. I would make the constructor of the ZipFile class take a file object, and hide (at least from unprivileged code) the option of passing a filename to the constructor. This would make it so that no authority is gained by importing the zipfile module. Regards, Zooko http://zooko.com/ ^-- under re-construction: some new stuff, some broken links

1 0

Re: [Python-Dev] Capabilities (we already got one)
by Zooko April 3, 2003

April 3, 2003

(I, Zooko, wrote the lines prepended with "> > ".) Guido wrote: > > Yes. That may be why the demand for capabilities has been met with > resistance: to quote the French in "Monty Python and the Holy Grail", > "we already got one!" :-) ;-) Such skepticism is of course perfectly appropriate for proposed changes to your beautiful language. More on the one you already got below. (I agree: you already got one.) > > Here's a two sentence definition of capabilities: > > I've heard too many of these. They are all too abstract. There may have been a terminological problem. The word "capabilities" has been used for three different systems -- "capabilities-as-rows-of-the-Lampson-access- control-matrix", "capabilities-as-keys", and "capabilities-as-references". Unfortunately, the distinction is rarely made explicit, so people often assert things about "capabilities" which are untrue of capabilities-as-references. (Ping has just written a paper about this.) The former two kinds of capabilities have major problems and are disliked by almost everybody. The last one is the one that Ping, Ben Laurie and I are advocating, and the one that you already got. Anyway, if someone gave a definition of capabilities-as-references and it didn't match with the two-sentence definition I gave (and with the diagram), then it was wrong. Here's the two-sentence definition again: > > Authority originates in C code (in the interpreter or C extension > > modules), and is passed from thing to thing. > > This part I like. > > > A given thing "X" -- an instance of ZipFile, for example -- has the > > authority to use a given authority -- to invoke the real open(), for > > example -- if and only if some thing "Y" previously held both the > > "open()" authority and the "authority to extend authorities to X" > > authority, and chose to extend the "open()" authority to X. > > But the instance of ZipFile is not really a protection domain. > Methods on the instance may have different authority. Okay, ZipFile was the wrong example. Here it is without examples: Abstract version: A given thing "X" can use a given authority "S" if and only if some thing "Y" has previously held both the authority and the "authority to extend authorities to X" and chose to extend "S" to X. To make it concrete, I will use the word "object" to mean "anything referenced by a Python reference". This includes class instances, closures, bound methods, stack frames, etc. When I mean Python's instance-of-a-class "object", I'll say "instance" instead of "object". So the concrete version is: Concrete version: An object "X" can use an object "S" if and only if some object "Y" has previously held references to both S and X, and chose to give a reference to S to X. (Quoting out of order:) > > Hm. Reviewing the rexec docs, I being to suspect that the "access > > control system with unified designation and authority" *is* how > > Python does access control in restricted mode, and that rexec itself > > is just to manage module import and certain dangerous builtins. > > Yes. [...] > Sure. The question is, what exactly are Alice, Bob and Carol? I > claim that they are not specific class instances but they are each a > "workspace" as I tried to explain before. A workspace is more or less > the contents of a particular "sys.modules" dictionary. I believe I understand the motivation for rexec now. I think that in restricted-execution-mode (hereafter: "REM", as per Greg Ewing's suggestion [1]), Python objects have encapsulation -- one can't access their private data without their permission. Once this is done, Python references are capabilities. So if you have a Python object such as a wxWindow instance, and you want to control access to it, the natural way to do that is to control how references to it are passed around. This is why you've already got one. The natural and Pythonic way to control access to Python objects is with capabilities, and that's what you've been doing all along. However, you don't use the same technique to control access to Python *modules* such as the zipfile module, because the "import zipfile" statement will give the current scope access to the zipfile module even if nobody has granted such access to the current scope. This is a violation of the two-sentence definition and of the graph: the current scope just gained authority ex nihilo. So your solution to this, to prevent code from grabbing privileges willy nilly via "import" and builtins, is rexec, which creates a scope in which code executes (now called a "workspace"), and allows you to control which builtins and modules are available for code executing in that "workspace". Now access to modules conforms to the definition of capabilities: an object X can access a module S if and only if some object Y previously had access to X's workspace and to S, and Y chose to give X access to S. So unless I've missed something, rexec conforms to the definition of capabilities as well. (Of course, one can always build other access-control mechanisms on top of capabilities. In particular, the rexec "hooks" mechanism seems intended for that.) Regards, Zooko http://zooko.com/ ^-- under re-construction: some new stuff, some broken links [1] http://mail.python.org/pipermail/python-dev/2003-March/034311.html

5 4

Re: [Python-Dev] Capabilities
by Zooko April 3, 2003

April 3, 2003

Guido wrote: > > I understand how class ZipFile could exercise authority in a > rexec-based world, if the zipfile module was trusted code. But I > thought that a capability view of the world doesn't distinguish > between trusted and untrusted code. I guess I need to understand > better what kind of "barriers" the capability way of life *does* use. I think you are on track with regard to the deeper question you are grappling with. Almost all dangerous things come ultimately from C code. (I can think of one danger that can come from pure Python code: it can provide an illicit communications channel between other objects.) So in the "separate policy language" way of life, access to the ZipFile class gives you the ability to open files anywhere in the filesystem. The ZipFile class therefore has the "dangerous" flag set, and when you run code that you think might misuse this feature, you set the "can't use dangerous things" flag on that code. In the capability way of life, it is still the case that access to the ZipFile class gives you the ability to open files anywhere in the system! (That is: I'm assuming for now that we implement capabilities without re-writing every dangerous class in the Library.) In this scheme, there are no flags, and when you run code that you think might misuse this feature, you simply don't give that code a reference to the ZipFile class. (Also, we have to arrange that it can't acquire a reference by "import zipfile".) So far the two approaches have the same effect, and the difference, for better or for worse, is that the policy of "this code can't use ZipFile" is encoded in Python reference-management code in the latter and encoded in a pair of flags in the former. Now, we might want to allow certain code to use something else dangerous (such as the socket module) while simultaneously disallowing it from using ZipFile. As we add N more dangerous modules, and M more objects of untrusted code that we want to control, we have an N*M access control matrix to configure which code can use which modules. (In an access control matrix, rows are "subjects" -- things that can exercise authority and columns are "resources" -- things that might require authority when used.) In a system where designation is not unified with authority, you tell this untrusted code "I want you to do this action X.", and then you also have to go update the policy specification to say that the code in question is allowed to do the action X. This "say it twice if you really mean it" overhead puts a practical limit on how fine-grained your policies can be, and it adds a source of accidents that lead to security holes. So now with a large or fine-grained access control matrix, we see the "unify designation and authority" maxim really shines, and really matches well with the Zen of Python. But there is still another advantage that capabilities offer over other access control systems. With normal access control (and an extremely diligent and patient programmer and user) you can in theory achieve the Principle of Least Privilege -- that the untrusted code runs with the minimal set of authorities necessary to do its job. However, this is implemented by creating a new "principal" -- a new row in the access control matrix, setting the access control bits in each element of that row, and preventing any other code from setting the bits in that row. Now, observe that only maximally trusted code -- with "root" authority -- is allowed to make these kinds of updates to the access control matrix. This means that all code is divided into two kinds: the kind that can impose Least-Privilege on code that it invokes (this code has root authority), and the kind that can be constrained by Least-Privilege when it is invoked (this code doesn't). With capabilities there is no such distinction. All code can be constrained to have access to only the privileges that it requires, and at the same time all code can constrain other code that it invokes. This feature, which I call "Higher-Order Principle of Least Privilege" [*] enables new applications. For example, using first-order Least-Privilege a web browser which runs cap-Python "caplets" could extend selective privileges to the caplets, such as permission to read a certain file, while withholding others, such as permission to write to that file, or permission to send the contents of the file to a remote computer. In addition, if cap-Python supports Higher-Order Least-Privilege, those caplets could themselves use other caplets ("web services"?) without unnecessarily exposing their privileges to those sub-caplets. One could imagine, for example, a web browser written in cap-Python, which runs inside the first web browser (e.g. Mozilla with a cap-Python plug-in), and uses cap-Python caplets to extend its (the cap-Python web browser's) functionality. If people already had the cap-Python plug-in installed in their local Mozilla, then simply visiting the "cap-python-browser.com" site would be sufficient to launch the cap-Python web browser. Of course, this could lead straight to a fully functional desktop, making good on Marc Andreesen's old threat to turn the browser into the operating system and the operating system into the device driver. This would be effectively the "virtualization" of access control. I regard it as a kind of holy Grail for internet computing. Regards, Zooko [*] I call it that because it is the application of the Principle of Least Privilege to the implementation of the Principle of Least Privilege. One should be able to impose least-privilege constraints on the code one uses without requiring full root privileges oneself! http://zooko.com/ ^-- under re-construction: some new stuff, some broken links

4 3

Re: [Python-Dev] Capabilities (we already got one)
by Zooko April 2, 2003

April 2, 2003

(I, Zooko, wrote the lines prepended with "> > ".) Ping wrote: > > > I think that in restricted-execution-mode (hereafter: "REM", as per Greg Ewing's > > suggestion [1]), Python objects have encapsulation -- one can't access their > > private data without their permission. > > > > Once this is done, Python references are capabilities. > > Aaack! I wish you would *stop* saying that! > > There is no criterion by which a reference is or is not a capability. > To talk in such terms only confuses the issue. Let me be a little more precise. Once Python objects are encapsulated, then possession of a reference is constrained in the following way: you can have a reference only if another object that had it chose to give it to you (or if you create something yourself, in which case you get the first-ever reference to it). This constraint happens to be the same constraint that the rule of capabilities imposes on the transmission of capabilities: you can have a capability only if someone else who had it chose to give it to you (or if you create something yourself, in which case you get the first-ever capability to it). Therefore, if you wish to use capability access control to manage access to resources in Python you can use the following technique: 1. Encapsulate the resource that you wish to control in a Python object. 2. Say to yourself "References are capabilities!". 3. Control the way references to that object are shared. Doing it this way will yield the advantages that capability access control enjoys over alternative access control models. It also has the advantage that your skills at Python programming can be applied directly to the problem of managing access control, without requiring you to learn any new policy language or new concepts. You are quite right, Ping, that capability access control could be enforced in other ways in Python. I didn't mean to say "capabilities are Python references", which would imply that capability access control could not be implemented in any other way. I'm deliberately refraining from posting about the issue of controlling import of modules and builtins in an attempt to "slow down" the discussion until Guido returns from Python UK. Regards, Zooko http://zooko.com/ ^-- under re-construction: some new stuff, some broken links

1 0