What's the point in making implicit Python 3 check here:
try: # Python 3
from distutils.command.build_py import build_py_2to3 as build_py
except ImportError: # Python 2
from distutils.command.build_py import build_py
instead of explicit check like:
if sys.version_info >= 3:
from distutils.command.build_py import build_py_2to3 as build_py
I was hit by this today.
in test_hashlib.py there is this:
self.assertRaises(ValueError, hashlib.new, 'spam spam spam spam spam')
self.assertRaises(TypeError, hashlib.new, 1)
but in hashlib.py, there is this code:
pass # no extension module, this hash is unsupported.
raise ValueError('unsupported hash type %s' % name)
The code will raise ValueError when int(1) is passed in, but the unittests expect a TypeError.
So, which is correct?
On Fri, 20 Jul 2012 20:49:03 +0200 (CEST)
brett.cannon <python-checkins(a)python.org> wrote:
> diff --git a/Lib/importlib/test/__init__.py b/Lib/test/test_importlib/__init__.py
> rename from Lib/importlib/test/__init__.py
> rename to Lib/test/test_importlib/__init__.py
> --- a/Lib/importlib/test/__init__.py
> +++ b/Lib/test/test_importlib/__init__.py
> @@ -1,5 +1,6 @@
> import os
> import sys
> +from .. import support
I don't know if I'm only speaking for myself, but I really have trouble
parsing non-trivial relative imports, and I personally prefer when
people use absolute imports (e.g. "from test import support").
Software development and contracting: http://pro.pitrou.net
I would like to write yet another JIT compiler for CPython. Before
writing anything, I would like your opinion because I don't know well
other Python compilers. I also want to prepare a possible integration
into CPython since the beginning of the project, or at least stay very
close to the CPython project (and CPython developers!). I did not
understand exactly why Unladen Swallow and psyco projects failed, so
please tell me if you think that my project is going to fail too!
== Why? ==
CPython is still the reference implementation, new features are first
added to this implementation (ex: PyPy is not supporting Python 3 yet,
but there is a project to support Python 3). Some projects still rely
on low level properties of CPython, especially its C API (ex: numpy;
PyPy has a cpyext module to emulate the CPython C API).
A JIT is the most promising solution to speed up the main evaluation
loop: using a JIT, it is possible to compile a function for a specific
type on the fly and so enable deeper optimizations.
psyco is no more maintained. It had its own JIT which is complex to
maintain. For example, it is hard to port it to a new hardware.
LLVM is fast and the next version will be faster. LLVM has a
community, a documentation, a lot of tools and is active.
There are many Python compilers which are very fast, but most of them
only support a subset of Python or require to modify the code (ex:
specify the type of all parameters and variables). For example, you
cannot run Django with Shredskin.
IMO PyPy is complex and hard to maintain. PyPy has a design completly
different than CPython and is much faster and has a better memory
footprint. I don't expect to be as fast as PyPy, just faster than
== General idea ==
I don't want to replace CPython. This is an important point. All
others Python compilers try to write something completly new, which is
an huge task and is a problem to stay compatible with CPython. I would
like to reuse as much as possible code of CPython and don't try to
fight against the GIL or reference counting, but try to cooperate
I would like to use a JIT to generate specialized functions for a
combinaison of arguments types. Specialization enables more
optimizations. I would like to use LLVM because LLVM is an active
project, has many developers and users, is fast and the next version
will be faster! LLVM already supports common optimizations like
My idea is to emit the same code than ceval.c from the bytecode to be
fully compatible with CPython, and then write a JIT to optimize
functions for a specific type.
== Roadmap ==
-- Milestone 1: Proof of concept --
* Use the bytecode produced by CPython parser and compiler
* Only compile a single function
* Emit the same code than ceval.c using LLVM, but without tracing,
exceptions nor signal handling (they will be added later)
* Support compiling and calling the following functions: "def func(a,
b): return a+b"
The pymothoa project can be used as a base to implement quickly such
proof of concept.
-- Milestone 2: Specialized function for the int type --
* Use type annotation to generate specialized functions for the int type
* Use C int with a guard detecting integer overflow to fallback on Python int
-- Milestone 3: JIT --
* Depending on the type seen at runtime, recompile the function to
generate specialized functions
* Use guard to fallback to a generic implementation if the type is
not the expected type
* Drop maybe the code using function annotations
At this step, we can start to benchmark to check if the (JIT) compiler
is faster than CPython.
-- Later (unsorted ideas) --
* Support exceptions
* Full support of Python
- list comprehension
- avoid reference counting when possible
- avoid temporary objects when possible
- release the GIL when possible
- inlining: should be very interesting with list comprehension
- unroll loops?
- lazy creation of the frame?
* Use registers instead of a stack in the "evaluation loop"?
* Add code to allow tracing and profiling
* Add code to handle signals (pending calls)
* Write a compiler using the AST, with a fallback to the bytecode?
(would it be faster? easier or more complex to maintain?)
* Test LLVM optimizers
* Compile a whole module or even a whole program
* Reduce memory footprint
* Type annotation to help the optimizer? (with guards?)
* "const" annotation to help the optimizer? (with guards?)
* Support any build option of Python:
- support Python 2 (2.5, 2.6, 2.7) and 3 (3.1, 3.2, 3.3, 3.4)
- support narrow and wide mode: flag at runtime?
- support debug and release mode: flag at runtime?
- support 32 and 64 bits mode on Windows?
== Other Python VM and compilers ==
-- Fully Python compliant --
* `PyPy <http://pypy.org/>`_
* `Jython <http://www.jython.org/>`_ based on the JVM
* `IronPython <http://ironpython.net/>`_ based on the .NET VM
* `Unladen Swallow <http://code.google.com/p/unladen-swallow/>`_ fork
of CPython 2.6 using LLVM
- `Unladen Swallow Retrospective
- `PEP 3146 <http://python.org/dev/peps/pep-3146/>`_
* `psyco <http://psyco.sourceforge.net/>`_ (fully Python compliant?),
no more maintained
-- Subset of Python to C++ --
* `Nuitka <http://www.nuitka.net/pages/overview.html>`_
* `Python2C <http://strout.net/info/coding/python/ai/python2c.py>`_
* `Shedskin <http://code.google.com/p/shedskin/>`_
* `pythran <https://github.com/serge-sans-paille/pythran>`_ (no
class, set, dict, exception, file handling, ...)
-- Subset of Python --
* `pymothoa <http://code.google.com/p/pymothoa/>`_: use LLVM; don't
support classes nor exceptions.
* `unpython <http://code.google.com/p/unpython/>`_: Python to C
* `Perthon <http://perthon.sourceforge.net/>`_: Python to Perl
* `Copperhead <http://copperhead.github.com/>`_: Python to GPU (Nvidia)
-- Language very close to Python --
* `Cython <http://www.cython.org/>`_: "Cython is a programming
language based on Python, with extra syntax allowing for optional
static type declarations." Based on `Pyrex
== See also ==
* `Volunteer developed free-threaded cross platform virtual machines?
On Fri, Jul 20, 2012 at 4:48 AM, Antoine Pitrou <solipsis(a)pitrou.net> wrote:
> On Wed, 18 Jul 2012 02:26:14 +0200
> Victor Stinner <victor.stinner(a)gmail.com> wrote:
>> >> Monkey patching is a common practice in Python. test_os.py replaces
>> >> os.exec*() functions temporary for example.
>> > Perhaps for testing, but I don't think monkey-patching is common in
>> > production code. Perhaps you are thinking of Ruby :)
>> The gevent library does monkey-patch os.fork (and time.sleep and many
>> other functions), but gevent is maybe not ready for production? :-)
> Extensive monkey-patching of core OS functions would certainly make me
> weary of using such a third-party library, even if it claims to be
gevent is one of the well-behaved players in that game - it doesn't
monkeypatch anything unless you explicitly tell it to . You do need
to have some idea what you're doing if using the monkey patching
aspect (as not all third party modules will behave well in that case -
just as not all third party modules behave well on other Python
implementations), but you can also just use gevent as a normal library
without monkey patching anything (however, adhering strictly to that
rule leaves you in the same situation as Twisted, where modules that
use the standard blocking APIs are completely useless to you. Thus, if
your goal is "pure async" code without any need for monkeypatching,
Twisted is a much better bet than gevent, just because it's been
playing that game for longer and has more fully async aware components
That's why I call features like the gevent.monkey module "runtime
forks" - when they're implemented well, they effectively give you two
Python implementations without the complexity of actually having
parallel Python installations (compare the complexity of using
Stackless, a traditional fork, vs gevent.monkey, a runtime fork -
there's a reason the core greenlet component was lifted out of
Stackless and made available as a CPython extension module).
Applications can decide at startup which Python variant (standard
CPython, or CPython+gevent) they want to use without affecting other
Python applications installed on the system and without needing to
duplicating all installed Python modules into a second interpreter.
It's only when libraries monkeypatch standard interfaces *implicitly*
as a side effect of import that you start to get seriously broken
"action at a distance" behaviour. Or, as happened with distutils, you
get a third party project (setuptools) effectively freezing
implementation details of the stdlib version - when monkeypatching
starts delving too deep into implementation details, it's time to
switch to a traditional fork and start figuring out better ways to
solve the problem (which process is now grinding away on the
Monkeypatching is a powerful tool - used well, it lets you achieve
impressive things that would otherwise be substantially more difficult
to either create in the first place or maintain in the long run. As
with many powerful tools though, used badly it can lead to a lot of
pain. Static languages like Java decide the risk of action at a
distance from monkeypatching isn't worth the power it gives you to
work around issues in third party libraries without resorting to
creating a traditional fork, so they strictly enforce access controls
and have completely closed class definitions. Python permits
monkeypatching at a language level (because when you need it, you
really need it), but discourages it a social level (because there is
usually a less magical technique, such as creating and using a wrapper
or subclass, that will achieve the desired effect).
P.S. for a non-monkeypatching approach to reusing a synchronous
library when creating an asynchronous equivalent, take a look at
10gen's attempt to create a *maintainable* Tornado-compatible async
wrapper for PyMongo by using greenlets (). It's a clever idea,
but still a lot more work than just using PyMongo itself with gevent
if you're only interested in the "async" part rather than the
Nick Coghlan | ncoghlan(a)gmail.com | Brisbane, Australia
As part of pickle4, I found it interesting to add the possibility
of pickling bound functions (instance methods). This is done by
pickling f.__self__ and f.__func__ separately, and then adding
a BIND opcode to tie them together.
While this appears to work fine for python methods (non-builtin), some
issues arise with builtins. These are partly caused because
not all builtin function types support __func__, partly because
not all of them fill __module__ when they should and partly
because there are many (7) types a function can actually have:
ClassMethodDescriptorType = type(??)
BuiltinFunctionType = type(len)
FunctionType = type(f)
MethodType = type(A().f())
MethodDescriptorType = type(list.append)
WrapperDescriptorType = type(list.__add__)
MethodWrapperType = type(.__add__)
AllFunctionTypes = (ClassMethodDescriptorType, BuiltinFunctionType,
FunctionType, MethodType, MethodDescriptorType,
repr(AllFunctionTypes) = (
<class 'builtin_function_or_method'>, <class 'function'>,
<class 'method'>, <class 'method_descriptor'>,
<class 'wrapper_descriptor'>, <class 'method-wrapper'>)
I have created a patch at , which adds __func__ to some other
function types, as well as:
1) adds AllFunctionTypes etc. to Lib/types.py
2) inspect.isanyfunction(), inspect.isanyboundfunction(),
Note that I am not knowledgeable of cpython internals and therefore
the patch needs to be carefully reviewed.
Possible issues: Should classmethods be considered bound or unbound?
If cm is a classmethod, then should
cm.__func__.__self__ = cm.__self__ or cm.__func__.__self__ = None?
Currently does the latter:
>>> cm.__self__, hasattr(cm,'__self__'), hasattr(cm.__func__,
(<class '__main__.A'>, True, False)
This requires treating classmethods separately when pickling,
so I'm not sure if this is ideal.
Let me know if I should have opened an issue instead. I look
forward to hearing your opinions/suggestions on this matter.