As of Python 3.5.0rc1, the canonical repository for Python 3.5.0 is
*no longer* on hg.python.org. Instead, it's hosted on Bitbucket on
my personal account, here:
https://bitbucket.org/larry/cpython350
Since 3.5.0rc1 isn't out yet I'm keeping the repository private for now.
Once 3.5.0 rc1 is released (hopefully Monday) I'll flip the switch and make
the repository public. (I'll email python-dev and python-committers when
that happens.)
Putting it succinctly, here's a table of versions and where you'd check in
for your change to go there:
3.5.0 : https://bitbucket.org/larry/cpython350 (branch "3.5")
3.5.1 : hg.python.org/cpython (branch "3.5")
3.6.0 : hg.python.org/cpython (branch "default")
You'll notice nobody but myself has checkin permissions for my 3.5.0 repo on
Bitbucket. That's on purpose. The only way you can get changes in to 3.5.0
now is by sending me a Bitbucket "pull request". This is a workflow
experiment, to see if we as a community like this sort of new-fangled gizmo.
For now, we're only using Bitbucket for the actual final checking-in stage.
Requests for fixes to be accepted into 3.5.0 and code review will all still
happen on the Python issue tracker.
Also, I'm officially now asking you folks to do the forward-merge into 3.5.1
and 3.6.0 yourselves.
Here's how to get a fix checked in for 3.5.0, starting with 3.5.0rc1+ and
continuing through until 3.5.0 final.
Pre-requisites:
* You must have a Bitbucket account.
* You must have commit rights to the CPython repository.
1. Create an issue on the Python issue tracker for the problem.
2. Submit a patch that fixes the problem.
3. Add me to the issue and get me to agree that it needs fixing in 3.5.0.
(You can attempt this step before 2 if you prefer.)
4. Fork my repository into your Bitbucket account using their web GUI.
To do that, go to Bitbucket, log in, then go to my 3.5.0 repo:
https://bitbucket.org/larry/cpython350
and press the "Fork" link in the left column. Bitbucket has a tutorial
on how to do this, here:
https://confluence.atlassian.com/display/BITBUCKET/Fork+a+teammate%27s+repo…
Note: DO NOT start with a conventional CPython trunk cloned from
hg.python.org. The 3.5 branch in my repo and the 3.5 branch in normal
CPython trunk have intentionally diverged and *need* to stay
out-of-sync.
5. Make a local clone of your fork on your computer.
Bitbucket has a tutorial on how to do that, here:
https://confluence.atlassian.com/display/BITBUCKET/Copy+your+Mercurial+repo…
Reminder: switch to the 3.5 branch!
6. Apply your change to the 3.5 branch and check in.
Reminder: check in to the 3.5 branch!
7. Make sure you checked in your change to the 3.5 branch.
Reminder: Seriously. I keep messing this up. I say, the more
reminders,
the better.
8. Push your change back to *your* fork on *your* Bitbucket account.
Just normal "hg push" should work here.
In case it helps, I recommend using the "https" protocol for this
step, as
it sidesteps ssh authentication and prompts you for your Bitbucket
username
and password.
9. Create a pull request using Bitbucket's web GUI.
Bitbucket has a tutorial on how to create a pull request, here:
https://confluence.atlassian.com/display/BITBUCKET/Create+a+pull+request
On the "Create pull request" web GUI, make sure that you specify
branch "3.5" for *both* your repo *and* my repo. Also, make sure
you *don't* check the "Close 3.5 after the pull request is merged"
check box.
(If you use the "Compare" page, you also need to select "3.5" in both
drop-down lists--one for my repo, and one for yours.)
10. Paste a link to the pull request into the issue tracker issue for this
change request.
11. Wait for confirmation that I've accepted your pull request into the
3.5.0 repo.
12. Pull your accepted change from your local Bitbucket fork repo into
a normal hg.cpython.org CPython repo, merge into 3.5, then merge
into 3.6, then push.
For the record, here's what *my* workflow looks like when I accept your
pull request:
1. Click on the URL you pasted into the pull request.
2. Visually check that the diff matches the approved diff in the issue
on the issue tracker.
3. Click on the "Merge" button.
Frequently Asked Questions
==========================
Q: What if someone sends me a "pull request" for a change that doesn't
merge cleanly?
A: Then I'll decline it, and ask you on the issue tracker to rebase
and resubmit.
Q: What if someone sends me a "pull request" but they don't have commit
rights to CPython?
A: I'll ignore it. I'll only pay attention to pull requests pasted into
the issue tracker by someone with commit rights.
Q: Whose name goes on the commit?
A: It gets the name the checkin was made with. Don't worry, your name
will stay on your commit.
Q: This seems like a lot more work than the old way.
A: For you guys, yes. But notice how little work it is for *me*! Seriously.
Q: Can I reuse my fork / my local copy? Or do I have to create
a fresh one each time?
A: I don't care either way. All I care about are clean pull requests.
If you're careful you should have no trouble reusing forks and local
checkouts. If it were me, I'd probably use a fresh fork each time.
Forks are cheap and this way is cleaner.
I'll add these instructions to the Python Dev Guide in the next day or two.
/arry
p.s. Remember to use the 3.5 branch!
On behalf of the Python development community and the Python 3.5 release
team, I'm relieved to announce the availability of Python 3.5.0rc1, also
known as Python 3.5.0 Release Candidate 1.
Python 3.5 has now entered "feature freeze". By default new features
may no longer be added to Python 3.5.
This is a preview release, and its use is not recommended for production
settings.
You can find Python 3.5.0rc1 here:
https://www.python.org/downloads/release/python-350rc1/
Windows and Mac users: please read the important platform-specific
"Notes on this release" section near the end of that page.
Happy hacking,
/arry
This just went by this morning on reddit's /r/programming. It's a paper
that analyzed Python--among a handful of other languages--to answer the
question "are branch predictors still that bad at the big switch
statement approach to interpreters?" Their conclusion: no.
Our simulations [...] show that, as long as the payload in the
bytecode remains limited and do not feature significant amount of
extra indirect branches, then the misprediction rate on the
interpreter can be even become insignificant (less than 0.5 MPKI).
(MPKI = missed predictions per thousand instructions)
Their best results were on simulated hardware with state-of-the-art
prediction algorithms ("TAGE" and "ITTAGE"), but they also demonstrate
that branch predictors in real hardware are getting better quickly.
When running the Unladen Swallow test suite on Python 3.3.2, compiled
with USE_COMPUTED_GOTOS turned off, Intel's Nehalem experienced an
average of 12.8 MPKI--but Sandy Bridge drops that to 3.5 MPKI, and
Haswell reduces it further to a mere *1.4* MPKI. (AFAICT they didn't
compare against Python 3.3.2 using computed gotos, either in terms of
MPKI or in overall performance.)
The paper is here:
https://hal.inria.fr/hal-01100647/document
I suppose I wouldn't propose removing the labels-as-values opcode
dispatch code yet. But perhaps that day is in sight!
//arry/
We retagged Python 3.5.0rc1 today to fix two bugs that popped up late in
the process. Release candidates are supposed to be software you
genuinely would release, and I couldn't release Python with both those
bugs. This delay rippled through the whole process, so it just isn't
going out tonight (late Sunday / early Monday in my timezone). I have
every expectation it'll go out Monday.
In case you're interested, the bugs are (were!):
http://bugs.python.org/issue24745http://bugs.python.org/issue24835
My thanks to the folks who stepped up and fixed the bugs on short
notice, and my apologies to the community for the delay. We're just
trying to make the best Python we can, for yooooooooou!
See you tomorrow,
//arry/
As I write this email I'm tagging Python 3.5.0 release candidate 1. This
is the moment that we switch over to our new experimental workflow,
where we use Bitbucket and pull requests for all future changesets that
will get applied to 3.5.0.
The Bitbucket repository isn't ready yet, and I'm still putting the
final touches on the documentation for the process. I'll have
everything ready no later than a day from now. It'll be posted here,
and will also go into the Python Dev Guide. For now changes for
3.5.0rc2 will just have to wait.
Again: any revisions you check in to the "3.5" branch on
hg.python.org/cpython right now will go automatically into 3.5.1. They
will *not* automatically go in to 3.5.0. The only way to get changes
into 3.5.0 from this moment forward is by creating a pull request on
Bitbucket which you convince me to accept.
//arry/
p.s. In case you're curious, the last revision to make it in to 3.5.0rc1
(apart from the revisions I generate as part of my build engineering
work) was 202a7aabd4fe.
Is hg.python.org okay for others? I'm getting the following output from all hg commands:
sending hello command
sending between command
abort: no suitable response from remote hg!
I don't know of any more verbose or debugging options than that (--debug, -v's added nothing). I can SSH normally into hg.p.o, not that there's anything I can do without a terminal, but I guess it's not completely down...
Bitbucket has a mirror of cpython, here:
https://bitbucket.org/mirror/cpython
It was last updated on May 7 and still says it's Python 3.5.0a4.
It's not clear to me who owns the "mirror" account--is it Atlassian
themselves? Anyway it'd be nice if it were, y'know, fresher.
//arry/
To pass by reference or by copy of - that is the question from hamlet.
("hamlet" - a community of people smaller than a village python3.4-linux64)
xlist = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
i = 0
for x in xlist:
print(xlist)
print("\txlist[%d] = %d" % (i, x))
if x%2 == 0 :
xlist.remove(x)
print(xlist, "\n\n")
i = i + 1
So, catch the output and help me, PLEASE, improve the answer:
Does it appropriate ALWAYS REevaluate the terms of the expression list
in FOR-scope on each iteration?
But if I want to pass ONCE a copy to FOR instead of a reference (as seen
from an output) and reduce unreasonable reevaluation, what I must to do
for that?
Hi,
Another summer with another EuroPython, which means its time again to try to revive PEP 447…
I’ve just pushes a minor update to the PEP and would like to get some feedback on this, arguably fairly esoteric, PEP.
The PEP proposes to to replace direct access to the class __dict__ in object.__getattribute__ and super.__getattribute__ by calls to a new special method to give the metaclass more control over attribute lookup, especially for access using a super() object. This is needed for classes that cannot store (all) descriptors in the class dict for some reason, see the PEP text for a real-world example of that.
Regards,
Ronald
The PEP text (with an outdated section with benchmarks removed):
PEP: 447
Title: Add __getdescriptor__ method to metaclass
Version: $Revision$
Last-Modified: $Date$
Author: Ronald Oussoren <ronaldoussoren(a)mac.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 12-Jun-2013
Post-History: 2-Jul-2013, 15-Jul-2013, 29-Jul-2013, 22-Jul-2015
Abstract
========
Currently ``object.__getattribute__`` and ``super.__getattribute__`` peek
in the ``__dict__`` of classes on the MRO for a class when looking for
an attribute. This PEP adds an optional ``__getdescriptor__`` method to
a metaclass that replaces this behavior and gives more control over attribute
lookup, especially when using a `super`_ object.
That is, the MRO walking loop in ``_PyType_Lookup`` and
``super.__getattribute__`` gets changed from::
def lookup(mro_list, name):
for cls in mro_list:
if name in cls.__dict__:
return cls.__dict__
return NotFound
to::
def lookup(mro_list, name):
for cls in mro_list:
try:
return cls.__getdescriptor__(name)
except AttributeError:
pass
return NotFound
The default implemention of ``__getdescriptor__`` looks in the class
dictionary::
class type:
def __getdescriptor__(cls, name):
try:
return cls.__dict__[name]
except KeyError:
raise AttributeError(name) from None
Rationale
=========
It is currently not possible to influence how the `super class`_ looks
up attributes (that is, ``super.__getattribute__`` unconditionally
peeks in the class ``__dict__``), and that can be problematic for
dynamic classes that can grow new methods on demand.
The ``__getdescriptor__`` method makes it possible to dynamically add
attributes even when looking them up using the `super class`_.
The new method affects ``object.__getattribute__`` (and
`PyObject_GenericGetAttr`_) as well for consistency and to have a single
place to implement dynamic attribute resolution for classes.
Background
----------
The current behavior of ``super.__getattribute__`` causes problems for
classes that are dynamic proxies for other (non-Python) classes or types,
an example of which is `PyObjC`_. PyObjC creates a Python class for every
class in the Objective-C runtime, and looks up methods in the Objective-C
runtime when they are used. This works fine for normal access, but doesn't
work for access with `super`_ objects. Because of this PyObjC currently
includes a custom `super`_ that must be used with its classes, as well as
completely reimplementing `PyObject_GenericGetAttr`_ for normal attribute
access.
The API in this PEP makes it possible to remove the custom `super`_ and
simplifies the implementation because the custom lookup behavior can be
added in a central location.
.. note::
`PyObjC`_ cannot precalculate the contents of the class ``__dict__``
because Objective-C classes can grow new methods at runtime. Furthermore
Objective-C classes tend to contain a lot of methods while most Python
code will only use a small subset of them, this makes precalculating
unnecessarily expensive.
The superclass attribute lookup hook
====================================
Both ``super.__getattribute__`` and ``object.__getattribute__`` (or
`PyObject_GenericGetAttr`_ and in particular ``_PyType_Lookup`` in C code)
walk an object's MRO and currently peek in the class' ``__dict__`` to look up
attributes.
With this proposal both lookup methods no longer peek in the class ``__dict__``
but call the special method ``__getdescriptor__``, which is a slot defined
on the metaclass. The default implementation of that method looks
up the name the class ``__dict__``, which means that attribute lookup is
unchanged unless a metatype actually defines the new special method.
Aside: Attribute resolution algorithm in Python
-----------------------------------------------
The attribute resolution proces as implemented by ``object.__getattribute__``
(or PyObject_GenericGetAttr`` in CPython's implementation) is fairly
straightforward, but not entirely so without reading C code.
The current CPython implementation of object.__getattribute__ is basicly
equivalent to the following (pseudo-) Python code (excluding some house keeping and speed tricks)::
def _PyType_Lookup(tp, name):
mro = tp.mro()
assert isinstance(mro, tuple)
for base in mro:
assert isinstance(base, type)
# PEP 447 will change these lines:
try:
return base.__dict__[name]
except KeyError:
pass
return None
class object:
def __getattribute__(self, name):
assert isinstance(name, str)
tp = type(self)
descr = _PyType_Lookup(tp, name)
f = None
if descr is not None:
f = descr.__get__
if f is not None and descr.__set__ is not None:
# Data descriptor
return f(descr, self, type(self))
dict = self.__dict__
if dict is not None:
try:
return self.__dict__[name]
except KeyError:
pass
if f is not None:
# Non-data descriptor
return f(descr, self, type(self))
if descr is not None:
# Regular class attribute
return descr
raise AttributeError(name)
class super:
def __getattribute__(self, name):
assert isinstance(name, unicode)
if name != '__class__':
starttype = self.__self_type__
mro = startype.mro()
try:
idx = mro.index(self.__thisclass__)
except ValueError:
pass
else:
for base in mro[idx+1:]:
# PEP 447 will change these lines:
try:
descr = base.__dict__[name]
except KeyError:
continue
f = descr.__get__
if f is not None:
return f(descr,
None if (self.__self__ is self.__self_type__) else self.__self__,
starttype)
else:
return descr
return object.__getattribute__(self, name)
This PEP should change the dict lookup at the lines starting at "# PEP 447" with
a method call to perform the actual lookup, making is possible to affect that
lookup both for normal attribute access and access through the `super proxy`_.
Note that specific classes can already completely override the default
behaviour by implementing their own ``__getattribute__`` slot (with or without
calling the super class implementation).
In Python code
--------------
A meta type can define a method ``__getdescriptor__`` that is called during
attribute resolution by both ``super.__getattribute__``
and ``object.__getattribute``::
class MetaType(type):
def __getdescriptor__(cls, name):
try:
return cls.__dict__[name]
except KeyError:
raise AttributeError(name) from None
The ``__getdescriptor__`` method has as its arguments a class (which is an
instance of the meta type) and the name of the attribute that is looked up.
It should return the value of the attribute without invoking descriptors,
and should raise `AttributeError`_ when the name cannot be found.
The `type`_ class provides a default implementation for ``__getdescriptor__``,
that looks up the name in the class dictionary.
Example usage
.............
The code below implements a silly metaclass that redirects attribute lookup to
uppercase versions of names::
class UpperCaseAccess (type):
def __getdescriptor__(cls, name):
try:
return cls.__dict__[name.upper()]
except KeyError:
raise AttributeError(name) from None
class SillyObject (metaclass=UpperCaseAccess):
def m(self):
return 42
def M(self):
return "fourtytwo"
obj = SillyObject()
assert obj.m() == "fortytwo"
As mentioned earlier in this PEP a more realistic use case of this
functionallity is a ``__getdescriptor__`` method that dynamicly populates the
class ``__dict__`` based on attribute access, primarily when it is not
possible to reliably keep the class dict in sync with its source, for example
because the source used to populate ``__dict__`` is dynamic as well and does
not have triggers that can be used to detect changes to that source.
An example of that are the class bridges in PyObjC: the class bridge is a
Python object (class) that represents an Objective-C class and conceptually
has a Python method for every Objective-C method in the Objective-C class.
As with Python it is possible to add new methods to an Objective-C class, or
replace existing ones, and there are no callbacks that can be used to detect
this.
In C code
---------
A new slot ``tp_getdescriptor`` is added to the ``PyTypeObject`` struct, this
slot corresponds to the ``__getdescriptor__`` method on `type`_.
The slot has the following prototype::
PyObject* (*getdescriptorfunc)(PyTypeObject* cls, PyObject* name);
This method should lookup *name* in the namespace of *cls*, without looking at
superclasses, and should not invoke descriptors. The method returns ``NULL``
without setting an exception when the *name* cannot be found, and returns a
new reference otherwise (not a borrowed reference).
Use of this hook by the interpreter
-----------------------------------
The new method is required for metatypes and as such is defined on `type_`.
Both ``super.__getattribute__`` and
``object.__getattribute__``/`PyObject_GenericGetAttr`_
(through ``_PyType_Lookup``) use the this ``__getdescriptor__`` method when
walking the MRO.
Other changes to the implementation
-----------------------------------
The change for `PyObject_GenericGetAttr`_ will be done by changing the private
function ``_PyType_Lookup``. This currently returns a borrowed reference, but
must return a new reference when the ``__getdescriptor__`` method is present.
Because of this ``_PyType_Lookup`` will be renamed to ``_PyType_LookupName``,
this will cause compile-time errors for all out-of-tree users of this
private API.
The attribute lookup cache in ``Objects/typeobject.c`` is disabled for classes
that have a metaclass that overrides ``__getdescriptor__``, because using the
cache might not be valid for such classes.
Impact of this PEP on introspection
-----------------------------------
Use of the method introduced in this PEP can affect introspection of classes
with a metaclass that uses a custom ``__getdescriptor__`` method. This section
lists those changes.
The items listed below are only affected by custom ``__getdescriptor__``
methods, the default implementation for ``object`` won't cause problems
because that still only uses the class ``__dict__`` and won't cause visible
changes to the visible behaviour of the ``object.__getattribute__``.
* ``dir`` might not show all attributes
As with a custom ``__getattribute__`` method `dir()`_ might not see all
(instance) attributes when using the ``__getdescriptor__()`` method to
dynamicly resolve attributes.
The solution for that is quite simple: classes using ``__getdescriptor__``
should also implement `__dir__()`_ if they want full support for the builtin
`dir()`_ function.
* ``inspect.getattr_static`` might not show all attributes
The function ``inspect.getattr_static`` intentionally does not invoke
``__getattribute__`` and descriptors to avoid invoking user code during
introspection with this function. The ``__getdescriptor__`` method will also
be ignored and is another way in which the result of ``inspect.getattr_static``
can be different from that of ``builtin.getattr``.
* ``inspect.getmembers`` and ``inspect.get_class_attrs``
Both of these functions directly access the class __dict__ of classes along
the MRO, and hence can be affected by a custom ``__getdescriptor__`` method.
**TODO**: I haven't fully worked out what the impact of this is, and if there
are mitigations for those using either updates to these functions, or
additional methods that users should implement to be fully compatible with
these functions.
One possible mitigation is to have a custom ``__getattribute__`` for these
classes that fills ``__dict__`` before returning and and defers to the
default implementation for other attributes.
* Direct introspection of the class ``__dict__``
Any code that directly access the class ``__dict__`` for introspection
can be affected by a custom ``__getdescriptor__`` method.
Performance impact
------------------
**WARNING**: The benchmark results in this section are old, and will be updated
when I've ported the patch to the current trunk. I don't expect significant
changes to the results in this section.
[snipped]
Alternative proposals
---------------------
``__getattribute_super__``
..........................
An earlier version of this PEP used the following static method on classes::
def __getattribute_super__(cls, name, object, owner): pass
This method performed name lookup as well as invoking descriptors and was
necessarily limited to working only with ``super.__getattribute__``.
Reuse ``tp_getattro``
.....................
It would be nice to avoid adding a new slot, thus keeping the API simpler and
easier to understand. A comment on `Issue 18181`_ asked about reusing the
``tp_getattro`` slot, that is super could call the ``tp_getattro`` slot of all
methods along the MRO.
That won't work because ``tp_getattro`` will look in the instance
``__dict__`` before it tries to resolve attributes using classes in the MRO.
This would mean that using ``tp_getattro`` instead of peeking the class
dictionaries changes the semantics of the `super class`_.
Alternate placement of the new method
.....................................
This PEP proposes to add ``__getdescriptor__`` as a method on the metaclass.
An alternative would be to add it as a class method on the class itself
(simular to how ``__new__`` is a `staticmethod`_ of the class and not a method
of the metaclass).
The two are functionally equivalent, and there's something to be said about
not requiring the use of a meta class.
References
==========
* `Issue 18181`_ contains an out of date prototype implementation
Copyright
=========
This document has been placed in the public domain.
.. _`Issue 18181`: http://bugs.python.org/issue18181
.. _`super class`: http://docs.python.org/3/library/functions.html#super
.. _`super proxy`: http://docs.python.org/3/library/functions.html#super
.. _`super`: http://docs.python.org/3/library/functions.html#super
.. _`dir()`: http://docs.python.org/3/library/functions.html#dir
.. _`staticmethod`: http://docs.python.org/3/library/functions.html#staticmethod
.. _`__dir__()`: https://docs.python.org/3/reference/datamodel.html#object.__dir__
.. _`NotImplemented`: http://docs.python.org/3/library/constants.html#NotImplemented
.. _`PyObject_GenericGetAttr`: http://docs.python.org/3/c-api/object.html#PyObject_GenericGetAttr
.. _`type`: http://docs.python.org/3/library/functions.html#type
.. _`AttributeError`: http://docs.python.org/3/library/exceptions.html#AttributeError
.. _`PyObjC`: http://pyobjc.sourceforge.net/
.. _`classmethod`: http://docs.python.org/3/library/functions.html#classmethod
