Mailman 3 June 2011 - Python-Dev

[RELEASE] Python 2.7.2
by Benjamin Peterson June 12, 2011

June 12, 2011

On behalf of the Python development team, I'm rosy to announce the immediate availability of Python 2.7.2. Since the release candidate 2 weeks ago, there have been 2 changes: 1. pyexpat.__version__ has be changed to be the Python version. 2. A regression from 3.1.3 in the handling of comments in the netrc module has been resolved. (see issue #12009). 2.7.2 is the second in bugfix release for the Python 2.7 series. 2.7 is the last major verison of the 2.x line and will be receiving only bug … [View More]

2 1

Re: [Python-Dev] [Python-checkins] cpython (2.7): allow "fake" filenames in findsource (closes #9284)
by Nick Coghlan June 12, 2011

June 12, 2011

On Sun, Jun 12, 2011 at 6:56 AM, benjamin.peterson <python-checkins(a)python.org> wrote: > summary: > allow "fake" filenames in findsource (closes #9284) > > This allows findsource() to work in doctests. > > A patch from Dirkjan Ochtman. Either this exception should be mentioned in the inspect.getsource() documentation or else doctest should be monkeypatching inspect as well as linecache. Cheers, Nick. -- Nick Coghlan | ncoghlan(a)gmail.… [View More]

2 2

Re: [Python-Dev] [Python-checkins] cpython (3.1): onto 3.1.5
by Benjamin Peterson June 11, 2011

June 11, 2011

2011/6/11 Terry Reedy <tjreedy(a)udel.edu>: > >> +What's New in Python 3.1.5? >> +=========================== >> + >> +*Release date: XXXX-XX-XX* >> + >> +Core and Builtins >> +----------------- >> + >> +Library >> +------- >> + >> + > > I presume that only security patches should be added. Indeed. -- Regards, Benjamin

1 0

Python jails
by Sam Edwards June 11, 2011

June 11, 2011

Hello! This is my first posting to the python-dev list, so please forgive me if I violate any unspoken etiquette here. :) I was looking at Python 2.x's f_restricted frame flag (or, rather, the numerous ways around it) and noticed that most (all?) of the attacks to escape restricted execution involved the attacker grabbing something he wasn't supposed to have. IMO, Python's extensive introspection features make that a losing battle, since it's simply too easy to forget to blacklist something … [View More]and the attacker finding it. Not only that, even with a perfect vacuum-sealed jail, an attacker can still bring down the interpreter by exhausting memory or consuming excess CPU. I think I might have a way of securely sealing-in untrusted code. It's a fairly nascent idea, though, and I haven't worked out all of the details yet, so I'm posting what I have so far for feedback and for others to try to poke holes in it. Absolutely nothing here is final. I'm just framing out what I generally had in mind. Obviously, it will need to be adjusted to be consistent with "the Python way" - my hope is that this can become a PEP. :) >>> # It all starts with the introduction of a new type, called a jail. (I haven't yet worked out whether it should be a builtin type, ... # or a module.) Unjailed code can create jails, which will run the untrusted code and keep strict limits on it. ... >>> j = jail() >>> dir(j) ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'acquire', 'getcpulimit', 'getcpuusage', 'getmemorylimit', 'getmemoryusage', 'gettimelimit', 'gettimeusage', 'release', 'setcpulimit', 'setmemorylimit', 'settimelimit'] >>> # The jail monitors three things: Memory (in bytes), real time (in seconds), and CPU time (also in seconds) ... # and it also allows you to impose limits on them. If any limit is non-zero, code in that jail may not exceed its limit. ... # Exceeding a memory limit will result in a MemoryError. I haven't decided what CPU/real time limits should raise. ... # The other two calls are "acquire" and "release," which allow you to seal (any) objects inside the jail, or bust them # out. Objects inside the jail (i.e. created by code in that jail) contribute their __sizeof__() to the j.getmemoryusage() ... >>> def stealPasswd(): ... return open('/etc/passwd','r').read() ... >>> j.acquire(stealPasswd) >>> j.getmemoryusage() # The stealPasswd function, its code, etc. are now locked away within the jail. 375 >>> stealPasswd() Traceback (most recent call last): File "<stdin>", line 1, in <module> JailError: tried to access an object outside of the jail The object in question is, of course, 'open'. Unlike the f_restricted model, the jail was freely able to grab the open() function, but was absolutely unable to touch it: It can't call it, set/get/delete attributes/items, or pass it as an argument to any functions. There are three criteria that determine whether an object can be accessed: a. The code accessing the object is not within a jail; or b. The object belongs to the same jail as the code accessing the object; or c. The object has an __access__ function, and theObject.__access__(theJail) returns True. For the jail to be able to access 'open', it needs to be given access explicitly. I haven't quite decided how this should work, but I had in mind the creation of a "guard" (essentially a proxy) that allows the jail to access the object. It belongs to the same jail as the guarded object (and is therefore impossible to create within a jail unless the guarded object belongs to the same jail), has a list of jails (or None for 'any') that the guard will allow to __access__ it (the guard is immutable, so jails can't mess with it even though they can access it), and what the guard will allow though it (read-write, read-only, call-within-jail, call-outside-jail). I have a couple remaining issues that I haven't quite sussed out: * How exactly do guards work? I had in mind a system of proxies (memory usage is a concern, especially in memory-limited jails - maybe allow __access__ to return specific modes of access rather than all-or-nothing?) that recursively return more guards after operations. (e.g., if I have a guard allowing read+call on sys, sys.stdout would return another guard allowing read+call on sys.stdout, likewise for sys.stdout.write) * How are objects sealed in the jail? j.acquire can lead to serious problems with lots of references getting recursively sealed in. Maybe disallow sealing in anything but code objects, or allow explicitly running code within a jail like j.execute(code, globals(), locals()), which works fine since any objects created by jailed code are also jailed. * How do imports work? Should __import__ be modified so that when a jail invokes it, the import runs normally (unjailed), and then returns the module with a special guard that allows read-only+call-within, but not on builtins? This has a nice advantage, since jailed code can import e.g. socket, and maybe even create a socket, but won't be able to do sock.connect(...), since socket.connect (which is running with jailed permissions) can't touch the builtin _socket module. * Is obj.__access__(j) the best way to decide access? It doesn't allow programmers much freedom to customize the jail policy since they can't modify __access__ for builtins. Maybe the jail should have the first chance (such as j.__access__(obj)), which allows programmers to subclass the jail, and the jail can fallback to obj.__access__(j) * How does Python keep track of what jail each frame is in? Maybe each frame can have a frame.f_jail, which references the jail object restricting that frame (or None for unjailed code) - frames' jails default to the jail holding the code object, or can be explicitly overridden (as in j.execute(code, globals(), locals())) * When are jails switched? Obviously, jailed code called from unjailed code (or even from other unjailed code) should be executed in the callee jail... But if a jailed caller is calling unjailed code, does the jail follow, or does the unjailed code run in an unjailed frame? How do programmers specify that? ...that's pretty much my two (erm, twenty) cents on the matter. Again, any feedback/adversarial reasoning you guys can offer is much appreciated. [View Less]

5 5

Re: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to
by Georg Brandl June 10, 2011

June 10, 2011

On 06/07/11 05:20, brett.cannon wrote: > http://hg.python.org/cpython/rev/fc282e375703 > changeset: 70695:fc282e375703 > user: Brett Cannon <brett(a)python.org> > date: Mon Jun 06 20:20:36 2011 -0700 > summary: > Remove some extraneous parentheses and swap the comparison order to > prevent accidental assignment. > > Silences a warning from LLVM/clang 2.9. Swapping the comparison order here seems a bit inconsistent to me. There are lots of … [View More]

12 14

Summary of Python tracker Issues
by Python tracker June 10, 2011

June 10, 2011

ACTIVITY SUMMARY (2011-06-03 - 2011-06-10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 2826 (+11) closed 21268 (+47) total 24094 (+58) Open issues with patches: 1236 Issues opened (45) ================== #5906: Risk of confusion in multiprocessing module - daemonic process http://bugs.python.org/issue5906 reopened by pakal #9516: sysconfig: $… [View More]

1 0

Re: [Python-Dev] cpython (3.2): Fix #11583. Changed os.path.isdir to use GetFileAttributes instead of os.stat.
by Georg Brandl June 9, 2011

June 9, 2011

On 06/09/11 02:00, brian.curtin wrote: > http://hg.python.org/cpython/rev/88e318166eaf > changeset: 70713:88e318166eaf > branch: 3.2 > parent: 70700:0aa3064d1cef > user: Brian Curtin <brian(a)python.org> > date: Wed Jun 08 18:17:18 2011 -0500 > summary: > Fix #11583. Changed os.path.isdir to use GetFileAttributes instead of os.stat. > > By changing to the Windows GetFileAttributes API in nt._isdir we can figure > out if the … [View More]

3 2

3.2.1 and Issue 12291
by Vinay Sajip June 9, 2011

June 9, 2011

I just filed an issue which shows a serious breakage in the marshal module: "file written using marshal in 3.2 can be read by 2.7, but not 3.2 or 3.3" http://bugs.python.org/issue12291 Perhaps this issue should be marked as a release blocker for 3.2.1 - what do others (particularly Georg, of course) think? While it does appear to have been broken for some time, it seems a bit too serious to leave until a 3.2.2 release. Regards, Vinay Sajip

2 1

Can we improve support for abstract base classes with desciptors
by Darren Dale June 9, 2011

June 9, 2011

I would like to try to address some shortfalls with the way python deals with abstract base classes containing descriptors. I originally was just concerned with improving support for defining abstract properties with the decorator syntax and converting between abstract and concrete properties, but recently realized that the problem extends to descriptors in general. ABCs ---- First, a bit of background may be in order. An abstract base class is defined by specifying its metaclass as ABCMeta (… [View More]or a subclass thereof):: class MyABC(metaclass=ABCMeta): @abstractmethod def foo(self): pass When trying to instantiate MyABC or any of its subclasses, ABCMeta inspects the current class namespace for items tagged with __isabstractmethod__=True:: class ABCMeta(type): #[...] def __new__(mcls, name, bases, namespace): cls = super().__new__(mcls, name, bases, namespace) # Compute set of abstract method names abstracts = {name for name, value in namespace.items() if getattr(value, "__isabstractmethod__", False)} ABCMeta then checks if any of the base classes define any items tagged with __isabstractmethod__ and whether they remain abstract in the current class namespace:: for base in bases: for name in getattr(base, "__abstractmethods__", set()): value = getattr(cls, name, None) if getattr(value, "__isabstractmethod__", False): abstracts.add(name) cls.__abstractmethods__ = frozenset(abstracts) In Objects/typeobject.c, __abstractmethods__ is actually a descriptor, and setting it gives the type a chance to set an internal flag specifying if it has any abstract methods defined. When object_new is called in typeobject.c, the flag is checked and an error is raised if any abstract methods were identified. Issues with ABCs and descriptors -------------------------------- In order for this scheme to work, ABCMeta needs to identify all of the abstract methods, but there are some limitations when we consider descriptors. For example, Python's property is a composite object, whose behavior is defined by the getter, setter, and deleter methods with which it is composed. Since there is already an @abstractmethod decorator, I would have suspected that defining abstract properties would be intuitive:: class MyABC(metaclass=ABCMeta): @abstractmethod def _get_foo(self): pass @abstractmethod def _set_foo(self, val): pass foo = property(_get_foo, _set_foo) @property @abstractmethod def bar(self): pass @bar.setter @abstractmethod def bar(self, val): pass Ideally, one would want the flexibility of defining a concrete getter and an abstract setter, for example. However, ABCMeta does not inspect the descriptors of a class to see if they contain any abstract methods. It only inspects the descriptor itself for a True __isabstractmethod__ attribute. This places the burdon on every descriptor implementation to provide its own support for ABC compatibility. For example, support for abstract properties was attempted by adding abstractproperty to the abc module. abstractproperty subclasses the property builtin (as opposed to the relationship between every other abstract and concrete class in the python language). Here is the definition of abstractproperty, in its entirety (modulo docstrings):: class abstractproperty(property): __isabstractmethod__ = True A number of problems manifest with this approach, and I think they all can be traced to the fact that the abstractedness of a descriptor is currently not dependent upon the abstractedness of the methods with which it is composed. The documentation for abstractproperty doesn't suggest using @abstractmethod:: class C(metaclass=ABCMeta): def getx(self): ... def setx(self, value): ... x = abstractproperty(getx, setx) which leads to Issue #1: What is abstract about C.x? How does a subclass of C know whether it needs to override the getter or setter? Issue #2: The decorator syntax cannot be used to convert an abstract property into a concrete one. (This relates to Issue #1: how would a descriptor even know when such a conversion would be appropriate?) Running the following code:: from abc import ABCMeta, abstractmethod, abstractproperty class AbstractFoo(metaclass=ABCMeta): @abstractproperty def bar(self): return 1 @bar.setter def bar(self, val): pass class ConcreteFoo(AbstractFoo): @AbstractFoo.bar.getter def bar(self): return 1 @bar.setter def bar(self, val): pass foo = ConcreteFoo() yields:: TypeError: Can't instantiate abstract class ConcreteFoo with abstract methods bar Issue #3: The following class *is* instantiable, even though AbstractFoo declared that a setter for bar is required:: class ConcreteFoo(AbstractFoo): @property def bar(self): pass Previous attempt to improve abc.abstractproperty ------------------------------------------------ It seems to me that the strategy used by abc.abstractproperty is fundamentally ill-advised. I explored the possibility of extending abstractproperty, redefining its getter, setter, and deleter methods such that they would work in conjunction with the @abstractmethod decorator and yield an instance of the builtin property once all abstract methods were replaced with concrete ones (http://bugs.python.org/issue11610). Issues #1 and #2 were addressed, but there were still problems with that approach. It did not address Issue #3, and it also introduced a new issue, #4:: class AbstractFoo(metaclass=ABCMeta): @abstractproperty # bar would be an abstractproperty, even though the getter is concrete def bar(self): return 1 @bar.setter # bar.setter inspected the getter and the new setter, did not identify # any abstract methods, and thus returned an instance of the built-in # property def bar(self, val): pass @bar.deleter # bar is a concrete property, its deleter decorator does not know it # is supposed to check for abstract methods, so it will return an # instance of the built-in property: @abstractmethod def bar(self): pass By the time the deleter was specified, bar was a concrete property, which does not know it should return an instance of abstractproperty (in part because the inheritance diagram for property/abstractproperty is inverted). Thus, AbstractFoo was instantiable, even though it shouldn't be. Finally, issue #5: the current approach taken by ABCMeta and abstractproperty places the burdon on descriptors to identify themselves to ABCMeta as abstract. Considering the issues encountered with abstractproperty, this may be an onerous requirement. There has been a fair amount of discussion at http://bugs.python.org/issue11610 , which can be summarized as a) concerns about maintaining backward compatibility, and b) objections to requiring @abstractmethod to specify that a method being passed to abstractproperty is abstract. Extending ABCMeta: A Promising Way Forward ------------------------------------------ I think the key is to focus on Issue #3. ABCMeta needs to be improved to recognize descriptor objects, both in the current namespace as well as the base classes, and to identify any abstract methods associated with the descriptors. I suggest the following approach in ABCMeta:: def __new__(mcls, name, bases, namespace): cls = super().__new__(mcls, name, bases, namespace) # Compute set of abstract method names def isdescriptor(val): return hasattr(val, '__get__') or hasattr(val, '__set__') \ or hasattr(val, '__delete__') def getabstracts(ns): return [name for name, value in ns.items() if getattr(value, "__isabstractmethod__", False)] abstracts = getabstracts(namespace) for item, val in namespace.items(): # further inspect descriptors for abstract methods: if isdescriptor(val): ## unfortunately, can't import inspect: #from inspect import getmembers #d = dict(getmembers(val)) ## so instead, use the following: d = dict((k, getattr(val, k, None)) for k in dir(val)) for name in getabstracts(d): # add the abstract descriptor methods to the list: abstracts.append('%s.%s'%(item, name)) for base in bases: for name in getattr(base, "__abstractmethods__", set()): if '.' in name: # base class identified a descriptor abstract method: k, v = name.split('.') desc = getattr(cls, k, None) val = getattr(desc, v, None) else: val = getattr(cls, name, None) if val is None or getattr(val, "__isabstractmethod__", False): abstracts.append(name) cls.__abstractmethods__ = frozenset(abstracts) I rolled everything into __new__ just to keep it as simple as possible for the sake of discussion. Python already provides the rest of the framework needed for descriptors to work properly with ABCs. This implementation actually works; I've tested it with an existing python-3.2 install:: from abc import ABCMeta, abstractmethod class AbstractFoo(metaclass=ABCMeta): @property @abstractmethod def bar(self): return 1 @bar.setter @abstractmethod def bar(self, val): pass >>> abstractfoo = AbstractFoo() Traceback (most recent call last): File "temp.py", line 17, in <module> abstractfoo = AbstractFoo() TypeError: Can't instantiate abstract class AbstractFoo with abstract methods bar.fget, bar.fset as expected. Note the more informative error message indicating what about the bar property is abstract. Also:: class ConcreteFoo(AbstractFoo): @AbstractFoo.bar.getter def bar(self): return 1 >>> foo = ConcreteFoo() Traceback (most recent call last): File "temp.py", line 24, in <module> foo = ConcreteFoo() TypeError: Can't instantiate abstract class ConcreteFoo with abstract methods bar.fset So issue #1 is addressed, since we are explicitly specifying which descriptor methods are abstract. Issue #2 has been addressed, since the following class is instantiable:: class ConcreteFoo(AbstractFoo): @AbstractFoo.bar.getter def bar(self): return 1 @bar.setter def bar(self, val): pass Issue #3 is also addressed. In the following example, even though I redefine the bar property as a readonly property, ABCMeta recognizes that a setter is needed:: class ConcreteFoo(AbstractFoo): @property def bar(self): return 1 >>> foo = ConcreteFoo() Traceback (most recent call last): File "temp.py", line 24, in <module> foo = ConcreteFoo() TypeError: Can't instantiate abstract class ConcreteFoo with abstract methods bar.fset Issue #4 (introduced in a previous attempt to solve the problem using abstractproperty) is also addressed:: class AbstractFoo(metaclass=ABCMeta): @property def bar(self): return 1 @bar.setter def bar(self, val): pass @bar.deleter @abstractmethod def bar(self): pass >>> abstractfoo = AbstractFoo() Traceback (most recent call last): File "temp.py", line 15, in <module> abstractfoo = AbstractFoo() TypeError: Can't instantiate abstract class AbstractFoo with abstract methods bar.fdel Finally, this approach addresses Issue #5 by holding ABCMeta responsible for identifying the abstractedness of descriptor methods, rather than placing that burdon on the desciptor objects to identify themselves as abstract. If ABCMeta were extended as above to identify abstract methods associated with descriptors, third parties would simply decorate methods used to compose the descriptors with @abstractmethod. This change to ABCMeta would not effect the behavior of abstractproperty, so backward compatibility would be maintained in that respect. But I think abstractproperty should be deprecated, or at the very least removed from the documentation. The documentation for @abstractmethod in >=python-3.3 should be extended to provide examples with properties/descriptors. The syntax would be backward compatible with older python versions, but with <python-3.3 ABCMeta would simply not recognize descriptors' abstract methods. This leads to one source of potential forward compatibility:: class AbstractFoo(metaclass=ABCMeta): @property @abstractmethod def bar(self): return 1 class ConcreteFoo(AbstractFoo): pass Both above classes would be instantiable with <python-3.3, but not with >=python3.3. In my opinion, this is a feature: python-3.3 has identified a bug in ConcreteFoo. The developer would not have tagged that method as abstract unless they had intended for consumers of AbstractFoo to provide a concrete implementation. Darren [View Less]

2 5

The socket HOWTO
by Antoine Pitrou June 8, 2011

June 8, 2011

Hello, I would like to suggest that we remove the socket HOWTO (currently at http://docs.python.org/dev/howto/sockets.html) My main issue with this document is that it doesn't seem to have a well-defined destination: - people who know sockets won't learn anything from it - but people who don't know sockets will probably find it clear as mud (for example, what's an "INET" or "STREAM" socket? what's "select"?) I have other issues, such as the style/tone it's written in. I'm sure the author had … [View More]

17 35