At the moment, the array module of the standard library allows to
create arrays of different numeric types and to initialize them from
an iterable (eg, another array).
What's missing is the possiblity to specify the final size of the
array (number of items), especially for large arrays.
I'm thinking of suffix arrays (a text indexing data structure) for
large texts, eg the human genome and its reverse complement (about 6
billion characters from the alphabet ACGT).
The suffix array is a long int array of the same size (8 bytes per
number, so it occupies about 48 GB memory).
At the moment I am extending an array in chunks of several million
items at a time at a time, which is slow and not elegant.
The function below also initializes each item in the array to a given
value (0 by default).
Is there a reason why there the array.array constructor does not allow
to simply specify the number of items that should be allocated? (I do
not really care about the contents.)
Would this be a worthwhile addition to / modification of the array module?
My suggestions is to modify array generation in such a way that you
could pass an iterator (as now) as second argument, but if you pass a
single integer value, it should be treated as the number of items to
allocate.
Here is my current workaround (which is slow):
def filled_array(typecode, n, value=0, bsize=(1<<22)):
"""returns a new array with given typecode
(eg, "l" for long int, as in the array module)
with n entries, initialized to the given value (default 0)
"""
a = array.array(typecode, [value]*bsize)
x = array.array(typecode)
r = n
while r >= bsize:
x.extend(a)
r -= bsize
x.extend([value]*r)
return x
I just spent a few minutes staring at a bug caused by a missing comma
-- I got a mysterious argument count error because instead of foo('a',
'b') I had written foo('a' 'b').
This is a fairly common mistake, and IIRC at Google we even had a lint
rule against this (there was also a Python dialect used for some
specific purpose where this was explicitly forbidden).
Now, with modern compiler technology, we can (and in fact do) evaluate
compile-time string literal concatenation with the '+' operator, so
there's really no reason to support 'a' 'b' any more. (The reason was
always rather flimsy; I copied it from C but the reason why it's
needed there doesn't really apply to Python, as it is mostly useful
inside macros.)
Would it be reasonable to start deprecating this and eventually remove
it from the language?
--
--Guido van Rossum (python.org/~guido)
This idea is already casually mentioned, but sank deep into the threads
of the discussion. Raise it up.
Currently reprs of classes and functions look as:
>>> int
<class 'int'>
>>> int.from_bytes
<built-in method from_bytes of type object at 0x826cf60>
>>> open
<built-in function open>
>>> import collections
>>> collections.Counter
<class 'collections.Counter'>
>>> collections.Counter.fromkeys
<bound method Counter.fromkeys of <class 'collections.Counter'>>
>>> collections.namedtuple
<function namedtuple at 0xb6fc4adc>
What if change default reprs of classes and functions to just full
qualified name __module__ + '.' + __qualname__ (or just __qualname__ if
__module__ is builtins)? This will look more neatly. And such reprs are
evaluable.
Hello everybody,
recently, I started a discussion on how metaclasses could be
improved. This lead to a major discussion about PEP 422,
where I discussed some possible changes to PEP 422.
Nick proposed I write a competing PEP, so that the community
may decide.
Here it comes. I propose that a class may define a classmethod
called __init_subclass__ which initialized subclasses of this
class, as a lightweight version of metaclasses.
Compared to PEP 422, this proposal is much simpler to implement,
but I do believe that it also makes more sense. PEP 422
has a method called __autodecorate__ which acts on the class
currently being generated. This is normally not what one wants:
when writing a registering class, for example, one does not want
to register the registering class itself. Most evident it become if
one wants to introduce such a decorator as a mixin: certainly one
does not want to tamper with the mixin class, but with the class
it is mixed in!
A lot of the text in my new PEP has been stolen from PEP 422.
This is why I hesitated writing myself as the only author of my
new PEP. On the other hand, I guess the original authors don't
want to be bothered with my stuff, so I credited them in the
body of my new PEP.
Greetings
Martin
And here it comes, the draft for a new PEP:
PEP: 422a
Title: Simpler customisation of class creation
Version: $Revision$
Last-Modified: $Date$
Author: Martin Teichmann <lkb.teichmann(a)gmail.com>,
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 27-Feb-2015
Python-Version: 3.5
Post-History: 27-Feb-2015
Replaces: 422
Abstract
========
Currently, customising class creation requires the use of a custom metaclass.
This custom metaclass then persists for the entire lifecycle of the class,
creating the potential for spurious metaclass conflicts.
This PEP proposes to instead support a wide range of customisation
scenarios through a new ``namespace`` parameter in the class header, and
a new ``__init_subclass__`` hook in the class body.
The new mechanism should be easier to understand and use than
implementing a custom metaclass, and thus should provide a gentler
introduction to the full power Python's metaclass machinery.
Connection to other PEP
=======================
This is a competing proposal to PEP 422 by Nick Coughlan and Daniel Urban.
It shares both most of the PEP text and proposed code, but has major
differences in how to achieve its goals.
Background
==========
For an already created class ``cls``, the term "metaclass" has a clear
meaning: it is the value of ``type(cls)``.
*During* class creation, it has another meaning: it is also used to refer to
the metaclass hint that may be provided as part of the class definition.
While in many cases these two meanings end up referring to one and the same
object, there are two situations where that is not the case:
* If the metaclass hint refers to an instance of ``type``, then it is
considered as a candidate metaclass along with the metaclasses of all of
the parents of the class being defined. If a more appropriate metaclass is
found amongst the candidates, then it will be used instead of the one
given in the metaclass hint.
* Otherwise, an explicit metaclass hint is assumed to be a factory function
and is called directly to create the class object. In this case, the final
metaclass will be determined by the factory function definition. In the
typical case (where the factory functions just calls ``type``, or, in
Python 3.3 or later, ``types.new_class``) the actual metaclass is then
determined based on the parent classes.
It is notable that only the actual metaclass is inherited - a factory
function used as a metaclass hook sees only the class currently being
defined, and is not invoked for any subclasses.
In Python 3, the metaclass hint is provided using the ``metaclass=Meta``
keyword syntax in the class header. This allows the ``__prepare__`` method
on the metaclass to be used to create the ``locals()`` namespace used during
execution of the class body (for example, specifying the use of
``collections.OrderedDict`` instead of a regular ``dict``).
In Python 2, there was no ``__prepare__`` method (that API was added for
Python 3 by PEP 3115). Instead, a class body could set the ``__metaclass__``
attribute, and the class creation process would extract that value from the
class namespace to use as the metaclass hint. There is `published code`_ that
makes use of this feature.
Another new feature in Python 3 is the zero-argument form of the ``super()``
builtin, introduced by PEP 3135. This feature uses an implicit ``__class__``
reference to the class being defined to replace the "by name" references
required in Python 2. Just as code invoked during execution of a Python 2
metaclass could not call methods that referenced the class by name (as the
name had not yet been bound in the containing scope), similarly, Python 3
metaclasses cannot call methods that rely on the implicit ``__class__``
reference (as it is not populated until after the metaclass has returned
control to the class creation machinery).
Finally, when a class uses a custom metaclass, it can pose additional
challenges to the use of multiple inheritance, as a new class cannot
inherit from parent classes with unrelated metaclasses. This means that
it is impossible to add a metaclass to an already published class: such
an addition is a backwards incompatible change due to the risk of metaclass
conflicts.
Proposal
========
This PEP proposes that a new mechanism to customise class creation be
added to Python 3.5 that meets the following criteria:
1. Integrates nicely with class inheritance structures (including mixins and
multiple inheritance),
2. Integrates nicely with the implicit ``__class__`` reference and
zero-argument ``super()`` syntax introduced by PEP 3135,
3. Can be added to an existing base class without a significant risk of
introducing backwards compatibility problems, and
4. Restores the ability for class namespaces to have some influence on the
class creation process (above and beyond populating the namespace itself),
but potentially without the full flexibility of the Python 2 style
``__metaclass__`` hook.
Those goals can be achieved by adding two functionalities:
1. A ``__init_subclass__`` hook that initializes all subclasses of a
given class, and
2. A new keyword parameter ``namespace`` to the class creation statement,
that gives an initialization of the namespace.
As an example, the first proposal looks as follows::
class SpamBase:
# this is implicitly a @classmethod
def __init_subclass__(cls, ns, **kwargs):
# This is invoked after a subclass is created, but before
# explicit decorators are called.
# The usual super() mechanisms are used to correctly support
# multiple inheritance.
# ns is the classes namespace
# **kwargs are the keyword arguments to the subclasses'
# class creation statement
super().__init_subclass__(cls, ns, **kwargs)
class Spam(SpamBase):
pass
# the new hook is called on Spam
To simplify the cooperative multiple inheritance case, ``object`` will gain
a default implementation of the hook that does nothing::
class object:
def __init_subclass__(cls, ns):
pass
Note that this method has no keyword arguments, meaning that all
methods which are more specialized have to process all keyword
arguments.
This general proposal is not a new idea (it was first suggested for
inclusion in the language definition `more than 10 years ago`_, and a
similar mechanism has long been supported by `Zope's ExtensionClass`_),
but the situation has changed sufficiently in recent years that
the idea is worth reconsidering for inclusion.
The second part of the proposal is to have a ``namespace`` keyword
argument to the class declaration statement. If present, its value
will be called without arguments to initialize a subclasses
namespace, very much like a metaclass ``__prepare__`` method would
do.
In addition, the introduction of the metaclass ``__prepare__`` method
in PEP 3115 allows a further enhancement that was not possible in
Python 2: this PEP also proposes that ``type.__prepare__`` be updated
to accept a factory function as a ``namespace`` keyword-only argument.
If present, the value provided as the ``namespace`` argument will be
called without arguments to create the result of ``type.__prepare__``
instead of using a freshly created dictionary instance. For example,
the following will use an ordered dictionary as the class namespace::
class OrderedBase(namespace=collections.OrderedDict):
pass
class Ordered(OrderedBase):
# cls.__dict__ is still a read-only proxy to the class namespace,
# but the underlying storage is an OrderedDict instance
.. note::
This PEP, along with the existing ability to use __prepare__ to share a
single namespace amongst multiple class objects, highlights a possible
issue with the attribute lookup caching: when the underlying mapping is
updated by other means, the attribute lookup cache is not invalidated
correctly (this is a key part of the reason class ``__dict__`` attributes
produce a read-only view of the underlying storage).
Since the optimisation provided by that cache is highly desirable,
the use of a preexisting namespace as the class namespace may need to
be declared as officially unsupported (since the observed behaviour is
rather strange when the caches get out of sync).
Key Benefits
============
Easier use of custom namespaces for a class
-------------------------------------------
Currently, to use a different type (such as ``collections.OrderedDict``) for
a class namespace, or to use a pre-populated namespace, it is necessary to
write and use a custom metaclass. With this PEP, using a custom namespace
becomes as simple as specifying an appropriate factory function in the
class header.
Easier inheritance of definition time behaviour
-----------------------------------------------
Understanding Python's metaclasses requires a deep understanding of
the type system and the class construction process. This is legitimately
seen as challenging, due to the need to keep multiple moving parts (the code,
the metaclass hint, the actual metaclass, the class object, instances of the
class object) clearly distinct in your mind. Even when you know the rules,
it's still easy to make a mistake if you're not being extremely careful.
Understanding the proposed implicit class initialization hook only requires
ordinary method inheritance, which isn't quite as daunting a task. The new
hook provides a more gradual path towards understanding all of the phases
involved in the class definition process.
Reduced chance of metaclass conflicts
-------------------------------------
One of the big issues that makes library authors reluctant to use metaclasses
(even when they would be appropriate) is the risk of metaclass conflicts.
These occur whenever two unrelated metaclasses are used by the desired
parents of a class definition. This risk also makes it very difficult to
*add* a metaclass to a class that has previously been published without one.
By contrast, adding an ``__init_subclass__`` method to an existing type poses
a similar level of risk to adding an ``__init__`` method: technically, there
is a risk of breaking poorly implemented subclasses, but when that occurs,
it is recognised as a bug in the subclass rather than the library author
breaching backwards compatibility guarantees.
Integrates cleanly with \PEP 3135
---------------------------------
Given that the method is called on already existing classes, the new
hook will be able to freely invoke class methods that rely on the
implicit ``__class__`` reference introduced by PEP 3135, including
methods that use the zero argument form of ``super()``.
Replaces many use cases for dynamic setting of ``__metaclass__``
----------------------------------------------------------------
For use cases that don't involve completely replacing the defined
class, Python 2 code that dynamically set ``__metaclass__`` can now
dynamically set ``__init_subclass__`` instead. For more advanced use
cases, introduction of an explicit metaclass (possibly made available
as a required base class) will still be necessary in order to support
Python 3.
A path of introduction into Python
==================================
Most of the benefits of this PEP can already be implemented using
a simple metaclass. For the ``__init_subclass__`` hook this works
all the way down to python 2.7, while the namespace needs python 3.0
to work. Such a class has been `uploaded to PyPI`_.
The only drawback of such a metaclass are the mentioned problems with
metaclasses and multiple inheritance. Two classes using such a
metaclass can only be combined, if they use exactly the same such
metaclass. This fact calls for the inclusion of such a class into the
standard library, let's call it ``SubclassMeta``, with a base class
using it called ``SublassInit``. Once all users use this standard
library metaclass, classes from different packages can easily be
combined.
But still such classes cannot be easily combined with other classes
using other metaclasses. Authors of metaclasses should bear that in
mind and inherit from the standard metaclass if it seems useful
for users of the metaclass to add more functionality. Ultimately,
if the need for combining with other metaclasses is strong enough,
the proposed functionality may be introduced into python's ``type``.
Those arguments strongly hint to the following procedure to include
the proposed functionality into python:
1. The metaclass implementing this proposal is put onto PyPI, so that
it can be used and scrutinized.
2. Once the code is properly mature, it can be added to the python
standard library. There should be a new module called
``metaclass`` which collects tools for metaclass authors, as well
as a documentation of the best practices of how to write
metaclasses.
3. If the need of combining this metaclass with other metaclasses is
strong enough, it may be included into python itself.
New Ways of Using Classes
=========================
This proposal has many usecases like the following. In the examples,
we still inherit from the ``SubclassInit`` base class. This would
become unnecessary once this PEP is included in Python directly.
Subclass registration
---------------------
Especially when writing a plugin system, one likes to register new
subclasses of a plugin baseclass. This can be done as follows::
class PluginBase(SubclassInit):
subclasses = []
def __init_subclass__(cls, ns, **kwargs):
super().__init_subclass__(ns, **kwargs)
cls.subclasses.append(cls)
One should note that this also works nicely as a mixin class.
Trait descriptors
-----------------
There are many designs of python descriptors in the wild which, for
example, check boundaries of values. Often those "traits" need some support
of a metaclass to work. This is how this would look like with this
PEP::
class Trait:
def __get__(self, instance, owner):
return instance.__dict__[self.key]
def __set__(self, instance, value):
instance.__dict__[self.key] = value
class Int(Trait):
def __set__(self, instance, value):
# some boundary check code here
super().__set__(instance, value)
class HasTraits(SubclassInit):
def __init_subclass__(cls, ns, **kwargs):
super().__init_subclass__(ns, **kwargs)
for k, v in ns.items():
if isinstance(v, Trait):
v.key = k
The new ``namespace`` keyword in the class header enables a number of
interesting options for controlling the way a class is initialised,
including some aspects of the object models of both Javascript and Ruby.
Order preserving classes
------------------------
::
class OrderedClassBase(namespace=collections.OrderedDict):
pass
class OrderedClass(OrderedClassBase):
a = 1
b = 2
c = 3
Prepopulated namespaces
-----------------------
::
seed_data = dict(a=1, b=2, c=3)
class PrepopulatedClass(namespace=seed_data.copy):
pass
Cloning a prototype class
-------------------------
::
class NewClass(namespace=Prototype.__dict__.copy):
pass
Rejected Design Options
=======================
Calling the hook on the class itself
------------------------------------
Adding an ``__autodecorate__`` hook that would be called on the class
itself was the proposed idea of PEP 422. Most examples work the same
way or even better if the hook is called on the subclass. In general,
it is much easier to explicitly call the hook on the class in which it
is defined (to opt-in to such a behavior) than to opt-out, meaning
that one does not want the hook to be called on the class it is
defined in.
This becomes most evident if the class in question is designed as a
mixin: it is very unlikely that the code of the mixin is to be
executed for the mixin class itself, as it is not supposed to be a
complete class on its own.
The original proposal also made major changes in the class
initialization process, rendering it impossible to back-port the
proposal to older python versions.
Other variants of calling the hook
----------------------------------
Other names for the hook were presented, namely ``__decorate__`` or
``__autodecorate__``. This proposal opts for ``__init_subclass__`` as
it is very close to the ``__init__`` method, just for the subclass,
while it is not very close to decorators, as it does not return the
class.
Requiring an explicit decorator on ``__init_subclass__``
--------------------------------------------------------
One could require the explicit use of ``@classmethod`` on the
``__init_subclass__`` decorator. It was made implicit since there's no
sensible interpretation for leaving it out, and that case would need
to be detected anyway in order to give a useful error message.
This decision was reinforced after noticing that the user experience of
defining ``__prepare__`` and forgetting the ``@classmethod`` method
decorator is singularly incomprehensible (particularly since PEP 3115
documents it as an ordinary method, and the current documentation doesn't
explicitly say anything one way or the other).
Passing in the namespace directly rather than a factory function
----------------------------------------------------------------
At one point, PEP 422 proposed that the class namespace be passed
directly as a keyword argument, rather than passing a factory function.
However, this encourages an unsupported behaviour (that is, passing the
same namespace to multiple classes, or retaining direct write access
to a mapping used as a class namespace), so the API was switched to
the factory function version.
Possible Extensions
===================
Some extensions to this PEP are imaginable, which are postponed to a
later pep:
* A ``__new_subclass__`` method could be defined which acts like a
``__new__`` for classes. This would be very close to
``__autodecorate__`` in PEP 422.
* ``__subclasshook__`` could be made a classmethod in a class instead
of a method in the metaclass.
References
==========
.. _published code:
http://mail.python.org/pipermail/python-dev/2012-June/119878.html
.. _more than 10 years ago:
http://mail.python.org/pipermail/python-dev/2001-November/018651.html
.. _Zope's ExtensionClass:
http://docs.zope.org/zope_secrets/extensionclass.html
.. _uploaded to PyPI:
https://pypi.python.org/pypi/metaclass
Copyright
=========
This document has been placed in the public domain.
Hello all,
I was wondering if anyone had a suggestion on an accessible IDE for Python using Voiceover on the Mac. I use a Mac with Voiceover to interact with my computer, but can also use Linux or Windows if necessary. I would really like to find a tool that will allow me to use Python with some of the features of an IDE. I have tried:
• PyCharm
• iPy
• iPy Notebook
• Idel
• and Eclipse
So far Eclipse worked the best but still caused problems. Any suggestions and/or help would be great.
What you are think about turning deprecation warnings on by default in
the interactive interpreter?
Deprecation warnings are silent by default because they just annoys end
user that uses old Python program with new Python. End user usually is
not a programmer and can't do something with warnings. But in the
interactive interpreter the user is the author of executed code and can
avoid to use deprecated functions if warned.
Hey everyone,
I was writing a test the other day and I wanted to assert that some
function raises some exception, but with some exact value. Turns out you
cannot compare OSError("foobar") to OSError("foobar") as it doesn't have
any __eq__. (it's the same for all exceptions?)
I think exceptions are great candidates to have an __eq__ method - they
already have a __hash__ (that's computed from the contained values) and
they are already containers in a way (they all have the `args` attribute).
Comparing exceptions in a test assertion seems like a good usecase for this.
Thanks,
-- Ionel Cristian Mărieș, blog.ionelmc.ro
Coroutines are useless until primed by a next(my_coro) call. There are
decorators to solve this by returning a primed coroutine. Such
decorators add another problem: now the user may be unsure whether a
specific coroutine from a third-party API should be primed or not
prior to use.
How about having a built-in function named send() with the signature
send(coroutine, value) that does the right thing: sends a value to the
coroutine, priming it if necessary. So instead of writing this:
>>> next(my_coro)
>>> my_coro.send(a_value)
You could always write this:
>>> send(my_coro, a_value)
At a high level, the behavior of send() would be like this:
def send(coroutine, value):
if inspect.getgeneratorstate() == 'GEN_CREATED':
next(coroutine)
coroutine.send(value)
Consider the above pseudo-code. Proper handling of other generator
states should be added.
This would make coroutines easier to use, and would make their basic
usage more consistent with the usage of plain generators using a
function syntax -- eg. next(my_gen) -- instead of method call syntax.
What do you think? If this has been discussed before, please send me a
link (I searched but came up empty).
Best,
Luciano
--
Luciano Ramalho
Twitter: @ramalhoorg
Professor em: http://python.pro.br
Twitter: @pythonprobr
On 19 February 2015 at 22:22, Martin Teichmann
<martin.teichmann(a)gmail.com> wrote:
>> Yes, populating __class__ happens inside the interpreter only after the
>> class is fully constructed, so even though the *class* is fully initialised
>> at the end of __init__, a reference hasn't been inserted into the cell yet.
>> (The __class__ cell doesn't actually live on the class object - it's created
>> implicitly by the interpreter and then referenced from the individual
>> methods as a closure variable for each method that looks up "__class__" or
>> "super")
>>
>> Python 2 doesn't provide the implicitly populated __class___ cell at all
>> though, so you have to refer to the class object by its name - zero argument
>> super is a Python 3 only feature.
>
> In short: PEP 422 in the current form is not back-portable.
> With my modifications it is portable all the way down to python 2.7.
>
> PEP 422 being back-portable => people will immediately start using it.
> PEP 422 not back-portable => it won't be used for several years to come.
You haven't really explained why you think it's not backportable.
You've suggested a backport may require some workarounds in order to
handle cooperative multiple inheritance, but that's not even close to
being the same thing as not being able to backported at all.
That said, if you'd like to write a competing proposal for the
community's consideration, please go ahead (details on the process are
in PEP 1)
I'm certainly open to alternative ideas in this space, and PEP 422 is
deferred specifically because I don't currently have time to explore
those ideas further myself.
Regards,
Nick.
--
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
There's a lot of flexibility in the new layered IO system, but one
thing it doesn't allow is any means of adding "hooks" to the data
flow, or manipulation of an already-created io object.
For example, when a subprocess.Popen object uses a pipe for the
child's stdout, the data is captured instead of writing it to the
console. Sometimes it would be nice to capture it, but still write to
the console. That would be easy to do if we could wrap the underlying
RawIOBase object and intercept read() calls[1]. A subclass of
RawIOBase can do this trivially, but there's no way of replacing the
class on an existing stream.
The obvious approach would be to reassign the "raw" attribute of the
BufferedIOBase object, but that's readonly. Would it be possible to
make it read/write? Or provide another way of replacing the raw IO
object underlying an io object?
I'm sure there are buffer integrity issues to work out, but are there
any more fundamental problems with this approach?
Paul
[1] Actually, it's *not* that easy, because subprocess.Popen objects
are insanely hard to subclass - there are no hooks into the pipe
creation process, and no way to intercept the object before the
subprocess gets run (that happens in the __init__ method). But that's
a separate issue, and also the subject of a different thread here.
