Mailman 3 April 2003 - Python-Dev

Re: [Python-Dev] python-dev Summary for 2003-03-16 through 2003-03-31
by Zooko 03 Apr '03

03 Apr '03

Brett Cannon <bac(a)OCF.Berkeley.EDU> wrote: > > One point made about capabilities is that they partially go against the > Pythonic grain. Since you have to pass capabilities specifically and > shouldn't allow them to be inherited, it does not go with the way you tend > to write Python code. This doesn't make sense to me, and I don't recall a message which asserted it. If capabilities were implemented as Python references, you could inherit capabilities (== references) from superclasses, just as you can currently do. The rest looks like a good summary! Regards, Zooko http://zooko.com/ ^-- under re-construction: some new stuff, some broken links

2 1

Re: [Python-Dev] Capabilities (we already got one)
by Phillip J. Eby 03 Apr '03

03 Apr '03

>However, you don't use the same technique to control access to Python *modules* >such as the zipfile module, because the "import zipfile" statement will give the >current scope access to the zipfile module even if nobody has granted such >access to the current scope. >... >So your solution to this, to prevent code from grabbing privileges willy nilly >via "import" and builtins, is rexec, which creates a scope in which code >executes (now called a "workspace"), and allows you to control which builtins >and modules are available for code executing in that "workspace". Almost. I think you may be confusing module *code* and module *objects*. Guido pointed this out earlier. A Python module object is populated by executing a body of *code* against the module *object* dictionary. The module object dictionary contains a '__builtins__' entry that gives it its "base" capabilities. Module *objects* possess capabilities, which are in their dictionary or reachable from it. *Code* doesn't possess capabilities except to constants used in the code. So access to *code* only grants you capabilities to the code and its constants. So, in order to provide a capability-safe environment, you need only provide a custom __import__ which uses a different 'sys.modules' that is specific to that environment. At that point, a "workspace" consists of an object graph rooted in the supplied '__builtins__', locals(), globals(), and initially executing code. We can then see that the standard Python environment is in fact a capability system, wherein everything is reachable from everything else. The "holes" in this capability system, then, are: 1. introspective abilities that allow "breaking out" of the workspace (such as the ability to 'sys._getframe()' or examine tracebacks to "reach up" to higher-level stack frames) 2. the structuring of the library in ways that equate creating an instance of a class with an "unsafe" capability. (E.g., creating instances of 'file()') coupled with instance->class introspection 3. Lack of true "privacy" for objects. (Proxies are a useful way to address this issue, because they allow more than one "capability" to exist for the same object.)

2 1

python-dev Summary for 2003-03-16 through 2003-03-31
by Brett Cannon 02 Apr '03

02 Apr '03

You guys have 24 hours to correct my usual bunch of mistakes. Also give me feedback on the new format for the Quickies section. ----------- +++++++++++++++++++++++++++++++++++++++++++++++++++++ python-dev Summary for 2003-03-16 through 2003-03-31 +++++++++++++++++++++++++++++++++++++++++++++++++++++ .. _last summary: http://www.python.org/dev/summary/2003-03-01_2003-03-15.html ====================== Summary Announcements ====================== PyCon is now over! It was a wonderful experience. Getting to meet people from python-dev in person was great. The sprint was fun and productive (work on the AST branch, caching where something is found in an inheritence tree, and a new CALL_ATTR opcode were all worked on). Definitely was worth it. I am trying a new way of formatting the Quickies_ section. I am trying non-inline implicit links instead of inlined ones. I am hoping this will read better in the text version of the summary. If you have an opinion on whether the new or old version is better let me know. And remember, the last time I asked for an opinion Michael Chermside was the only person to respond and thus ended up making an executive decision. .. _PyCon: http://www.python.org/pycon/ ======================== `Re: lists v. tuples`__ ======================== __ http://mail.python.org/pipermail/python-dev/2003-March/034029.html Splinter threads: - `Re: Re: lists v. tuples <http://mail.python.org/pipermail/python-dev/2003-March/034070.html>`__ This developed from a thread from covered in the `last summary`_ that discussed the different uses of lists and tuples. By the start date for this summary, though, it had turned into a discussion on comparisons. This occured when sorting heterogeneous objects came up. Guido commented that having anything beyond equality and non-equality tests for non-related objects does not make sense. This also led Guido to comment that "TOOWTDI makes me want to get rid of __cmp__" (TOOWTDI is "There is Only One Way to Do It"). Now before people start screaming bloody murder over the possible future loss of __cmp__() (which probably won't happen until Python 3), realize that all comparisons can be done using the six other rich comparisons (__lt__(), __eq__(), etc.). There is some possible code elegance lost if you have to use two rich comparisons instead a single __cmp__() comparison, but it is nothing that will prevent you from doing something that you couldn't do before. This all led Guido to suggest introducing the function before(). This would be used for arbitrary ordering of objects. Alex Martelli said it would "be very nice if before(x,y) were the same as x<y whenever the latter doesn't raise an exception, if feasible". He also said that it should probably "define a total ordering, i.e. the implied equivalence being equality". ================================ `Fast access to __builtins__`__ ================================ __ http://mail.python.org/pipermail/python-dev/2003-March/034243.html There has been rumblings on the list as of late of disallowing shadowing of built-ins. Specifically, the idea of someone injecting something into a module's namespace that overrides a global (by doing something like ``socket.len = lambda x: 42`` from the socket module) is slightly nasty, rarely done, and prevents the core from optimizing for built-ins. Raymond Hettinger, in an effort to see how to speed up built-in access, came up with the idea of replacing opcode calls of LOAD_GLOBAL and replace them with LOAD_CONST after putting the built-in being called into the constants table. This would leave shadowing of built-ins locally unaffected but prevent shadowing at the module. Raymond suggested turning on this behavior for when running Python -O. The idea of turning this on when running with the -O option was shot down. The main argument is that semantics are changed and thus is not acceptable for the -O flag. It was mentioned that -OO can change semantics, but even that is questionable. So this led to some suggestions of how to turn this kind of feature on. Someone suggested something like a pragma (think Perl) or some other mechanism at the module level. Guido didn't like this idea since he does not want modules to be riddled with code to turn on module-level optimizations. But all of this was partially shot down when Guido stepped in and reiterated he just wanted to prevent outside code from shadowing built-ins for a module. The idea is that if it can be proven that a module does not shadow a built-in it can output an opcode specific for that built-in, e.g. len() could output opcode for calling PyOject_Size() if the compiler can prove that len() is not shadowed in the module at any point. Neil Schemanauer suggested adding a warning for when this kind of shadowing is done. Guido said fine as long as extension modules are exempt. Now no matter how well the warning is coded, it would be *extremely* difficult to catch something like ``import X; d = X__dict__; d["len"] = lambda x: 42``. How do you deal with this? By Guido saying he has not issue saying something like this "is always prohibited". He said you could still do ``setattr(X, "len", lambda x: 42)``, though, and that might give you a warning. ================================ `capability-mediated modules`__ ================================ __ http://mail.python.org/pipermail/python-dev/2003-March/034149.html Splinter threads: - `Capabilities <http://mail.python.org/pipermail/python-dev/2003-March/034152.html>`__ The thread that will not die (nor does it look like it will in the near future; Guido asked to postpone discussing it until he gets back from `Python UK`_ which will continue the discussion into the next summary. I am ending up an expert at capabilities against my will. =) In case you have not been following all of this, capabilities as being discussed here is the idea that security is based on passing around references to objects. If you have a reference you can use it with no restrictions. Security comes in by controlling who you give references to. So I might ask for a reference to file(), but I won't necessarily get it. I could, instead, be handed a reference to a restrictive version of file() that only opens files in an OSs temporary file directory. If that is not clear, read the `last summary`_ on this thread. And now, on to the new stuff... One point made about capabilities is that they partially go against the Pythonic grain. Since you have to pass capabilities specifically and shouldn't allow them to be inherited, it does not go with the way you tend to write Python code. There were also suggestions to add arguments to import statements to give a more fine-grained control over them. But it was pointed out that classes fit this bill. The idea of limiting what modules are accessible by some code by not using a universally global scope (i.e., not using sys.modules) but by having a specific scope for each function was suggested. As Greg Ewing put it, "it would be dynamic scoping of the import namespace". While trying to clarify things (which were at PyCon thanks to the Open Space discussion held there on this subject), a good distinction between a rexec_ world (as in the module) and a capabilities was made by Guido. In capabilities, security is based on passing around references that have the amount of power you are willing for it to have. In a rexec world, it is based on what powers the built-ins give you; there is no worry about passing around code. Also, in the rexec world, you can have the idea of a "workspace" where __builtin__ has very specific definitions of built-ins that are used when executing untrusted code. Ka-Ping Yee wrote up an example of some code of what it would be like to code with capabilities (can be found at XXX ). .. _Python UK: http://www.python-uk.org/ .. _rexec: http://www.python.org/dev/doc/devel/lib/module-rexec.html ========= Quickies ========= `tzset`__ time.tzset() is going to be kept in Python, but only on UNIX. The testing suite was also loosened so as to not throw as many false-negatives. __ http://mail.python.org/pipermail/python-dev/2003-March/034062.html `Windows IO`__ stdin and stdout on Windows are TTYs. You can get 3rd-party modules to get more control over the TTY. __ http://mail.python.org/pipermail/python-dev/2003-March/034102.html `Who approved PyObject_GenericGetIter()???`__ Splinter threads: `Re: [Python-checkins] python/dist/src/Modules _hotshot.c,...`__; `PyObject_GenericGetIter()`__ Raymond Hettinger wrote a function called PyObject_GenericGetIter() that returned self for objects that were an iterator themselves. Thomas Wouters didn't like the name and neither did Guido since it was generic at all; it worked specifically with objects that were iterators themselves. Thus the function was renamed to PyObject_SelfIter(). __ http://mail.python.org/pipermail/python-dev/2003-March/034107.html __ http://mail.python.org/pipermail/python-dev/2003-March/034103.html __ http://mail.python.org/pipermail/python-dev/2003-March/034110.html `test_posix failures?`__ A test for posix.getlogin() was failing for Barry Warsaw under XEmacs (that is what he gets for not using Vim_ =). Thomas Wouters pointed out it only works when there is a utmp file somewhere. Basically it was agreed the test that was failing should be removed. __ http://mail.python.org/pipermail/python-dev/2003-March/034120.html .. _Vim: http://www.vim.org/ `Shortcut bugfix`__ Raymond Hettinger reported that a change in `_tkinter.c`_ for a function led to it returning strings or ints which broke PMW_ (although having a function return two different things was disputed in the thread; I think it used to return a string and now returns an int). The suggestion of making string.atoi() more lenient on its accepted arguments was made but shot down since it changes semantics. If you want to keep old way of having everything in Tkinter return strings instead of more proper object types (such as ints where appropriate), you can put teh line ``Tkinter.wantobjects = 0`` before the first creation of a tkapp object. __ http://mail.python.org/pipermail/python-dev/2003-March/034138.html .. __tkinter.c: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Modul… .. _PMW: http://pmw.sourceforge.net/ `csv package ready for prime-time?`__ Related: `csv package stitched into CVS hierarchy`__ Skip Montanaro: Okay to move csv_ package from the sandbox into the stdlib? Guido van Rossum: Yes. __ http://mail.python.org/pipermail/python-dev/2003-March/034162.html __ http://mail.python.org/pipermail/python-dev/2003-March/034179.html .. _csv: http://www.python.org/dev/doc/devel/lib/module-csv.html `string.strip doc vs code mismatch`__ Neal Norwitz asked for someone to look at http://python.org/sf/697220 which updates string.strip() from the string_ module to take an optional second argument. The patch is still open. __ http://mail.python.org/pipermail/python-dev/2003-March/034167.html .. _string: http://www.python.org/dev/doc/devel/lib/module-string.html `Re: More int/long integration issues`__ The point was made that it would be nice if the statement ``if num in range(...): ...`` could be optimized by the compiler if range() was only the built-in by substituting it with something like xrange() and thus skip creating a huge list. This would allow the removal of xrange() without issue. Guido suggested a restartable iterator (generator would work wonderfully if you could just get everything else to make what range() returns look like the list it should be). __ http://mail.python.org/pipermail/python-dev/2003-March/034019.html `socket timeouts fail w/ makefile()`__ Skip Montanaro discovered that using the makefile() method on a socket cause the file-like object to not observe the new timeout facility introduced in Python 2.3. He has since patched it so that it works properly and that sockets always have a makefile() (wasn't always the case before). __ http://mail.python.org/pipermail/python-dev/2003-March/034177.html `New Module? Tiger Hashsum`__ Tino Lange implemented a wrapper for the `Tiger hash sum`_ for Python and asked how he could get it added to the stdlib. He was told that he would need community backing before his module could be added in order to make sure that there is enough demand to warrant the edition. __ http://mail.python.org/pipermail/python-dev/2003-March/034191.html .. _Tiger hash sum: http://www.cs.technion.ac.il/~biham/Reports/Tiger/ `Icon for Python RSS Feed?`__ Tino Lange asked if an XML RSS feed icon could be added at http://www.python.org/ for http://www.python.org/channews.rdf . It has been added. __ http://mail.python.org/pipermail/python-dev/2003-March/034196.html `How to suppress instance __dict__?`__ David Abrahams asked if there was an easy way to suppress an instance __dict__'s creation from a metaclass. The answer turned out to be no. __ http://mail.python.org/pipermail/python-dev/2003-March/034197.html `Weekly Python Bug/Patch Summary`__ Another summary can be found at http://mail.python.org/pipermail/python-dev/2003-March/034286.html Skip Montanaro's weekly reminder how Python ain't perfect. __ http://mail.python.org/pipermail/python-dev/2003-March/034200.html `[ot] offline`__ Samuele Pedroni is off relaxing is is going to be offline for two weeks starting March 23. __ http://mail.python.org/pipermail/python-dev/2003-March/034204.html `funny leak`__ Christian Tismer discovered a memory leak in a funky def statement he came up with. The leak has since been squashed (done at PyCon_ during the sprint, actually). __ http://mail.python.org/pipermail/python-dev/2003-March/034212.html `Checkins to Attic?`__ CVS_ uses something called the Attic to put files that are only in a branch but not the HEAD of a tree. __ http://mail.python.org/pipermail/python-dev/2003-March/034230.html .. _CVS: http://www.cvshome.org/ `ossaudiodev tweak needs testing`__ Greg Ward asked people who are running Linux or FreeBSD to execute ``Lib/test/regrtest.py -uaudio test_ossaudiodev`` so as to test his latest change to ossaudiodev_. __ http://mail.python.org/pipermail/python-dev/2003-March/034233.html .. _ossaudiodev: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Modul… `cvs.python.sourceforge.net fouled up`__ Apparently when you get that nice message from SourceForge_ telling you that recv() has aborted because of server overloading you can rest assured that people with checkin rights get to continue to connect since they get priority. __ http://mail.python.org/pipermail/python-dev/2003-March/034234.html .. _SF: .. _SourceForge: http://www.sf.net/ `Doc strings for typeslots?`__ You can't add custom docstrings to things stored in typeobject slots at the C level. __ http://mail.python.org/pipermail/python-dev/2003-March/034239.html `Compiler treats None both as a constant and variable`__ As of now the compiler outputs opcode that treats None as both a global and a constant. That will change as some point when assigning to None becomes an error instead of a warning as it is in Python 2.3; possibly 2.4 the change will be made. __ http://mail.python.org/pipermail/python-dev/2003-March/034281.html `iconv codec`__ M.A. Lemburg stated that he questioned whether the iconv codec was ready for prime-time. There have been multiple issues with it and most seem to stem from a platform's codec and not ones that come with Python. This affects all u"".encode() calls when the codec does not come with Python. Hye-Shik Chang said he would get his iconv codec NG patch up on SF in the next few days and that would be applied. __ http://mail.python.org/pipermail/python-dev/2003-March/034300.html

2 1

Re: [Python-Dev] How to suppress instance __dict__?
by David Abrahams 02 Apr '03

02 Apr '03

Guido van Rossum <guido(a)python.org> writes: >> > I think you could subclass the metaclass, override __new__, and delete >> > the bogus __getstate__ from the type's __dict__. Then you'll get the >> > default pickling behavior which ignores slots; that should work just >> > fine in your case. :-) >> >> Ooh, that's sneaky! But I can't quite see how it works. The error >> message I quoted at the top about __getstate__ happens when you try to >> pickle an instance of the class. If I delete __getstate__ during >> __new__, it won't be there for pickle to find when I try to do the >> pickling. What will keep it from inducing the same error? > > Just try it. There are many ways to customize pickling, and if > __getstate__ doesn't exist, pickling is done differently. Since this doesn't work: >>> d = type('d', (object,), { '__slots__' : ['foo'] } ) >>> pickle.dumps(d()) I'm still baffled as to why this works: >>> class mc(type): ... def __new__(self, *args): ... x = type.__new__(self, *args) ... del args[2]['__getstate__'] ... return x ... >>> c = mc('c', (object,), { '__slots__' : ['foo'], '__getstate__' : lambda self: tuple() } ) >>> pickle.dumps(c()) 'ccopy_reg\n_reconstructor\np0\n(c__main__\nc\np1\nc__builtin__\nobject\np2\nNtp3\nRp4\n.' especially since: >>> dir(d) == dir(c) 1 I don't see the logic in the source for object.__reduce__(), so where is it? OK, I see it in typeobject.c. But now: >>> c.__getstate__ <unbound method c.<lambda>> OK, this seems to indicate that my attempt to remove __getstate__ from the class __dict__ was a failure. That explains why pickling c works, but not why you suggested that I remove __getstate__ inside of __new__. Did you mean for me to do something different? I note that c's __slots__ aren't pickled at all, which I guess was the point of the __getstate__ requirement: >>> x = c() >>> x.foo = 1 >>> pickle.dumps(x) == pickle.dumps(c()) 1 Fortunately, in our case the __slots__ are empty so it doesn't matter. -- Dave Abrahams Boost Consulting www.boost-consulting.com

3 4

Python Technical Lead, New York, NY
by beau＠nyc-search.com 02 Apr '03

02 Apr '03

http://www.nyc-search.com/jobs/python.html

1 0

Python Programmers, NYC
by beau＠nyc-search.com 02 Apr '03

02 Apr '03

Python Programmers, NYC http://www.nyc-search.com/jobs/python.html

1 0

Re: [Python-Dev] Capabilities
by Zooko 02 Apr '03

02 Apr '03

It's apparent that I didn't explain capabilities clearly enough. Also I misunderstood something about rexec in general and ZipFile in particular. Once we succeed at understanding each other, I'll then inquire whether you agree with my Big Word Proofs. (I, Zooko, wrote lines prepended with "> > ".) Guido wrote: > > > So in the "separate policy language" way of life, access to the > > ZipFile class gives you the ability to open files anywhere in the > > filesystem. The ZipFile class therefore has the "dangerous" flag > > set, and when you run code that you think might misuse this feature, > > you set the "can't use dangerous things" flag on that code. > > But that's not how rexec works. In the rexec world, the zipfile > module has no special privileges; when it is imported by untrusted > code, it is reloaded from disk as if it were untrusted itself. The > zipfile.ZipFile class is a client of "open", an implementation of > which is provided to the untrusted code by the trusted code. <Zooko reads the zipfile module docs.> How is the implementation of "open" provided by the trusted code to the untrusted code? Is it possible to provide a different "open" implementation to different "instances" of the zipfile module? (I think not, as there is no such thing as "a different instance of a module", but perhaps you could have two rexec "workspaces" each of which has a zipfile module with a different "open"?) > > In this scheme, there are no flags, and when you run code > > that you think might misuse this feature, you simply don't give that > > code a reference to the ZipFile class. (Also, we have to arrange > > that it can't acquire a reference by "import zipfile".) > > The rexec world solves this very nicely IMO. Can't the capability > world do it the same way? The only difference might be that 'open' > would have to be a capability. I don't understand exactly how rexec works yet, but so far it sounds like capabilities. Here's a two sentence definition of capabilities: Authority originates in C code (in the interpreter or C extension modules), and is passed from thing to thing. A given thing "X" -- an instance of ZipFile, for example -- has the authority to use a given authority -- to invoke the real open(), for example -- if and only if some thing "Y" previously held both the "open()" authority and the "authority to extend authorities to X" authority, and chose to extend the "open()" authority to X. That rule could be enforced with the rexec system, right? Here is a graphical representation of this rule. (Taken from [1].) http://www.erights.org/elib/capability/ode/images/fundamental.gif In the diagram, the authority is "Carol", the thing that started with the authority is "Alice", and Alice is in the process of extending to Bob the authority to use Carol. This act -- the extending of authority from Alice to Bob -- is the only way that Bob can gain authority, and it can only happen if Alice has both the authority to use Carol and the authority to extend authorities to Bob. Those two sentences above (and equivalently the graph) completely define capabilities, in the abstract. They don't say how they are implemented. A particular implementation that I find deeply appealing is to make "has a reference to 'open'" be the determiner of whether a thing has the authority to use "open", and to make "has a reference to X" be the determiner of whether a thing has the authority to extend authorities to X. That's "unifying designation with authority", and that's what the E language does. > But I think "this code can't use ZipFile" is the wrong thing to say. > You should only have to say "this code can't write files" (or > something more specific). I agree. I incorrectly inferred from previous messages that the current problem under discussion was allowing or denying access to the ZipFile class. But whatever resource we wish to control access to, these same techniques will apply. > > In a system where designation is not unified with authority, you > > tell this untrusted code "I want you to do this action X.", and then > > you also have to go update the policy specification to say that the > > code in question is allowed to do the action X. > > Sorry, you've lost me here. Which part is the "designation" (new word > for me) and which part is the "authority"? Sorry. First let me point out that the issue of unifying designation with authority is separable from "the capability access control rule" described above. The two have good synergy, but aren't identical. By "designation" I meant "naming". For example... Let's see, I think I'll go back to my toy tictactoe example from [2]. In the tictactoe example, you have to specify which wxWindow the tictactoe game object should draw into. This is "designation" -- you pass a reference, which designates which specific window you are talking about. If you use the principle of unifying designation and authority, then this same act -- passing a reference to this particular wxWindows object -- conveys both the identification of which window to draw into and the authority to draw into it. # access control system with unified designation and authority game = TicTacToeGame() game.display(wxPython.wxWindow()) If you have separate designation and authority, then the same code has to look something like this: # access control system with separate designation and authority game = TicTacToeGame() window = wxPython.wxWindow() def policy(subject, resource, operation): if (subject is game) and (resource is window) and \ (operation == "invoke methods of"): return True return False rexec.register_policy_hook(policy) game.display(window) This is what I call "say it twice if you really mean it". Hm. Reviewing the rexec docs, I being to suspect that the "access control system with unified designation and authority" *is* how Python does access control in restricted mode, and that rexec itself is just to manage module import and certain dangerous builtins. > It really sounds to me like at least one of our fundamental (?) > differences is the autonomicity of code units. I think of code (at > least Python code) as a passive set of instructions that has no > inherent authority but derives authority from the built-ins passed to > it; you seem to describe code as having inherent authority. I definitely don't intend for code to have inherent authority (other than the Trusted Code Base -- the interpreter -- which can't help but have it). The word "thing" in my two-sentence definition (a white circle in the diagram) are "computational things that can have state and behavior". (This includes Python objects, closures, stack frames, etc... In another context I would call them "objects", but Python uses the word "object" for something more specific -- an instance of a class.) > > This would be effectively the "virtualization" of access control. I > > regard it as a kind of holy Grail for internet computing. > > How practical is this dream? How useful? Let's revisit the issue once we understand one another's access control schemes. ;-) Regards, Zooko [1] http://www.erights.org/elib/capability/ode/overview.html [2] http://mail.python.org/pipermail/python-dev/2003-March/033938.html http://zooko.com/ ^-- under re-construction: some new stuff, some broken links

4 3

PEP 269 once more.
by Jonathan Riehl 01 Apr '03

01 Apr '03

Hey all, FYI, Guido closed the patch I had on SourceForge (599331), but I have just put an updated patch there. I have added some documentation on how my pgen module may be used, and the interface is much more consistent and useful than the prior upload. If anyone is interested in playing with pgen from Python, check it out and let me know what you think. Thanks! -Jon

1 0