On 5/27/2015 9:31 PM, Nick Coghlan wrote:

+1 from me, for basically the same reasons Guido gives: Python 2.7 is going to be with us for a long time, and this particular change shouldn't have any externally visible impacts at either an ABI or API level.

Immediately after a release, giving the patch plenty of time to be be tested, seems like a good time. -- Terry Jan Reedy

On 28 May 2015 at 18:54, Berker Peksağ wrote:

On Thu, May 28, 2015 at 3:17 AM, Parasa, Srinivas Vamsi wrote:

...

Attached is the computed goto patch (along with instructions to run) for Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin at http://bugs.python.org/issue4753). We built and tested this patch for Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon – Haswell EP CPU with 18 cores, hyper-threading off, turbo off).

Hi Vamsi,

Thank you for your work and your detailed email.

I'm -1 on the idea because:

* Performance improvements are not bug fixes

The "nothing except backwards compatible bug fixes in maintenance releases" rule was adopted for the *benefit of Python users*. When a new feature release can be reliably expected every 18-24 months, it makes sense to err heavily on the side of minimising risks to stability when it comes to making judgement calls on whether or not a change is appropriate to a maintenance release or not. Python 2.7 is an unusual case, as even though there *are* newer feature releases available, the barriers to migration are much higher than they would otherwise be. Each progressive 3.x release has brought those barriers down a bit, and 3.5 and the static type checking work being done through mypy et al will bring them down even further, but version migration work is still work where end users don't see any immediate practical benefit - they only see the benefit *after* they're able to drop Python 2 compatibility, and can start using Python 3 only features like matrix multiplication and the async/await syntax. *Adding* features to Python 2.7 is quite rightly still controversial, as they add complexity to the compatibility matrix for testing purposes. Code that runs correctly with the PEP 466 and 476 changes to SSL handling, may fail on earlier versions. Internal performance improvements, by contrast, don't hurt end users at all beyond the stability risks, and in this case, the request to make the change is being accompanied by the offer to assist with ongoing maintenance (including engaging an experienced core developer to help coach Intel contributors through the contribution process). So when folks ask "What changed?" in relation to this request, what changed is the fact that it isn't expected to be a one off contribution, but rather part of a broader effort focused on improving the performance of both Python 2 and Python 3, including contributions to ongoing maintenance activities.

* The patch doesn't make the migration process from Python 2 to Python 3 easier * In long term, it would be nice to work on making Python 3 better:

Indeed, but we also need help living up to the "Python 2.7 will be supported to 2020" commitment. Python 2.7 maintenance is *not* a particularly exciting task, and it's only going to get less interesting as Python 3 adoption climbs, so we're going to need paid contributors to start filling the gap as volunteers (quite reasonably) move on to the more inherently rewarding task of working to move Python 3 forward. That's going to be a negotiation process - companies don't typically contribute paid development time to open source projects out of the kindness of their hearts, they do it either because they're using the project themselves, because of deals they've made with individual contributors around how they spend their time, or because it helps them influence the direction of upstream development in ways that help them and their customers. (And sometimes it's a mix of all 3 of those factors) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

won't this need python compiled with gcc 5.1 to have any effect? Which compiler version was used for the benchmark? the issue that negated most computed goto improvements (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=39284) was only closed very recently (r212172, 9f4ec746affbde1)

On 28 May 2015 at 21:55, Maciej Fijalkowski wrote:

...

I'm -1 on the idea because:

* Performance improvements are not bug fixes * The patch doesn't make the migration process from Python 2 to Python 3 easier

And this is why people have been porting Python applications to Go.

For folks hitting the kinds of scalability problems that Go is designed to help with, a few percentage points here and there in CPython performance aren't going to make a big difference - they'll need the kinds of speed multipliers that PyPy can offer. Given that Go can't run Python C extensions any more than PyPy can, and involves a rewrite in a different programming language to boot, we'd do well to ponder what Go currently offers that PyPy doesn't. If we ignore the fashion-driven aspect of "Google wrote it, so it must be cool" (which we can't do anything about), and if we ignore the multi-vendor commercial support question (which tends to significantly lag community adoption for true community driven projects like PyPy), then one of the big keys in my view is the easy redistributability of Go binaries. For Linux based network services (and even Windows these days), Docker containers offer a potentially compelling way of bundling the PyPy runtime with Python applications, and Docker, Inc already maintain a set of PyPy base images at https://registry.hub.docker.com/_/pypy/ Docker's image layering model then means that applications sharing a PyPy runtime shouldn't need to download the interpreter runtime itself more than once. As a result, I personally suspect that better documenting and promoting the CPython->Docker+PyPy migration option is likely to offer a more effective alternative to CPython->Go migrations than the more modest performance improvements we can make to the CPython runtime itself. (I still think the latter are a good idea, though - there's no point leaving Python 2.7 slower than it needs to be given the offer of assistance in maintaining it) Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 28 May 2015 at 15:37, Donald Stufft wrote:

I think docker is a pretty crummy answer to Go’s static binaries. What I would love is for Python to get:

* The ability to import .so modules via zipzimport (ideally without a temporary directory, but that might require newer APIs from libc and such). * The ability to create a “static” Python that links everything it needs into the binary to do a zipimport of everything else (including the stdlib). * The ability to execute a zipfile that has been concat onto the end of the Python binary.

I think that if we get all of that, we could easily create a single file executable with real, native support from Python by simply compiling Python in that static mode and then appending a zip file containing the standard library and any other distributions we need to the end of it.

We’d probably want some more quality of life improvements around accessing resources from within that zip file as well, but that can be done as a library easier than the above three things can.

+1. The new embeddable Python distribution for Windows is a great step forward for this. It's not single-file, but it's easy to produce a single-directory self-contained application with it. I don't know if there's anything equivalent for Linux/OSX - maybe it's something we should look at for them as well (although the whole "static binaries" concept seems to be fairly frowned on in the Unix world, from what I've seen). Paul

Hello, On Fri, 29 May 2015 08:38:44 +1000 Nick Coghlan wrote: []

In that vein, it might be interesting to see what could be done with MicroPython in terms of providing a lightweight portable Python runtime without CPython's extensive integration with the underlying OS.

Thanks for mentioning that. That's definitely what I have on my mind, actually I wanted to do long ago a Lua-esque hack of being able to cat together an interpreter and a script, so resulting executable would just run the script. What stopped me is that it would be Lua-esque-ly useless, as how much useful one can do with a bare script without dependencies? And MicroPython definitely has some puzzle pieces for a generic solution, but so far not a complete picture: 1. There're frozen modules support, but they're that - modules, packages not supported for freezing so far. 2. Then frozen modules require recompiling, and that's not real-world scalable. 3. Standard library is already "distributed" (vs CPython's monolithic), but half of modules are dummy so far. That said, making a demo of self-contained webapp server in 350-400K is definitely on my TODO list (package support for frozen modules is the only blocker for that).

Cheers, Nick.

-- Best regards, Paul mailto:pmiscml@gmail.com

+1. The new embeddable Python distribution for Windows is a great step forward for this. It's not single-file, but it's easy to produce a single-directory self-contained application with it. I don't know if there's anything equivalent for Linux/OSX - maybe it's something we should look at for them as well (although the whole "static binaries" concept seems to be fairly frowned on in the Unix world, from what I've seen).

Just curious What is "the new embeddable Python distribution for Windows" ?

On 31 May 2015 at 11:41, Paul Moore wrote:

On 31 May 2015 at 10:14, Xavier Combelle wrote:

...

...

+1. The new embeddable Python distribution for Windows is a great step forward for this. It's not single-file, but it's easy to produce a single-directory self-contained application with it. I don't know if there's anything equivalent for Linux/OSX - maybe it's something we should look at for them as well (although the whole "static binaries" concept seems to be fairly frowned on in the Unix world, from what I've seen).

Just curious What is "the new embeddable Python distribution for Windows" ?

Python 3.5 ships a zipfile which contains a self-contained Python installation, intended for embedding. The idea is that you unzip it into your application directory, and use it from within your application (either via the embedding API, or using the included python.exe/pythonw.exe). It doesn't use the registry, or any global resources, so it's independent of any installed python that might be present.

By the way, IMO the new embeddable distribution is a pretty big deal on Windows. To make sure that it doesn't end up unnoticed, can I suggest we include a prominent "What's New" entry for it, and a section in "Python Setup and Usage" under "Using Python on Windows" for it? I'd hate to find that 3 or 4 versions from now, we're still trying to remind people that they can use the embeddable distribution, in the same way that executable zipfiles ended up an almost unknown feature for ages. Paul

On 5/31/2015 6:59 AM, Alexander Walters wrote:

A better course of action would be to deprecate the non-portable version. Other than setting the PATH envvar, why do we need to continue even touching the system on install? It is highly annoying for those of us that maintain several installs of python on a single windows system, and it really should stop.

Some people want the right-click context menu entries -- Run (also double click) and Edit with Idle, which should be Edit with Idle x.y.

The only use I can think of for ever touching the registry in the first place is to tell distutils installers where python is. I can tell you right now, that design choice is a bug. There are some mighty hacks you have to go through to correct that behavior when you happen to be using a virtualenv.

(We are calling it 'embedable', but the rest of the world would call it 'portable', as in, runable from a usb stick)

-- Terry Jan Reedy

"We are calling it 'embedable', but the rest of the world would call it 'portable', as in, runable from a usb stick" I called it embeddable because it's not intended for direct use and is not complete. There's no test suite, no documentation, no tkinter (pending high demand), no pip, no site-packages, and no folder structure. It really is meant to be a component in another application that provides the rest of the layout for its own needs. (I probably ought to blog about it so there's at least one detailed example of what it's for...) A nice side-effect is that you can make a regular per-user install portable by adding a pyvenv.cfg with "applocal = True", which disables regular path resolution (and also ignores PYTHONPATH, which is a feature or a bug, depending on your point of view). This only works on Windows right now, but could probably be ported from getpathp.c into getpath.c easily. Cheers, Steve Top-posted from my Windows Phone ________________________________ From: Alexander Waltersmailto:tritium-list@sdamon.com Sent: ‎5/‎31/‎2015 6:39 To: python-dev@python.orgmailto:python-dev@python.org Subject: Re: [Python-Dev] Computed Goto dispatch for Python 2 A better course of action would be to deprecate the non-portable version. Other than setting the PATH envvar, why do we need to continue even touching the system on install? It is highly annoying for those of us that maintain several installs of python on a single windows system, and it really should stop. The only use I can think of for ever touching the registry in the first place is to tell distutils installers where python is. I can tell you right now, that design choice is a bug. There are some mighty hacks you have to go through to correct that behavior when you happen to be using a virtualenv. (We are calling it 'embedable', but the rest of the world would call it 'portable', as in, runable from a usb stick) On 5/31/2015 06:47, Paul Moore wrote:

...

...

...

+1. The new embeddable Python distribution for Windows is a great step forward for this. It's not single-file, but it's easy to produce a single-directory self-contained application with it. I don't know if there's anything equivalent for Linux/OSX - maybe it's something we should look at for them as well (although the whole "static binaries" concept seems to be fairly frowned on in the Unix world, from what I've seen).

Just curious What is "the new embeddable Python distribution for Windows" ? Python 3.5 ships a zipfile which contains a self-contained Python installation, intended for embedding. The idea is that you unzip it into your application directory, and use it from within your application (either via the embedding API, or using the included

On 31 May 2015 at 10:14, Xavier Combelle wrote: python.exe/pythonw.exe). It doesn't use the registry, or any global resources, so it's independent of any installed python that might be present. By the way, IMO the new embeddable distribution is a pretty big deal on Windows. To make sure that it doesn't end up unnoticed, can I suggest we include a prominent "What's New" entry for it, and a

On 31 May 2015 at 11:41, Paul Moore wrote: section in "Python Setup and Usage" under "Using Python on Windows" for it?

I'd hate to find that 3 or 4 versions from now, we're still trying to remind people that they can use the embeddable distribution, in the same way that executable zipfiles ended up an almost unknown feature for ages.

Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium-list%40sdamon.com

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40microsoft.c...

On May 28, 2015 at 10:55:08 AM, Steve Dower (steve.dower@microsoft.com) wrote:

Donald Stufft wrote:

...

I think docker is a pretty crummy answer to Go’s static binaries. What I would love is for Python to get:

* The ability to import .so modules via zipzimport (ideally without a temporary directory, but that might require newer APIs from libc and such). * The ability to create a “static” Python that links everything it needs into the binary to do a zipimport of everything else (including the stdlib). * The ability to execute a zipfile that has been concat onto the end of the Python binary.

I think that if we get all of that, we could easily create a single file executable with real, native support from Python by simply compiling Python in that static mode and then appending a zip file containing the standard library and any other distributions we need to the end of it.

And it would look like a 20MB+ file just for a simple 1KB Python script...

For Windows at least, I'd prefer to have some app-style installer generation (e.g. http://pynsist.readthedocs.org/en/latest/) which, combined with the embeddable Python distro (new for 3.5.0b1 in case anyone missed it), can simply extract everything into an install directory and run it from there. None of the items on the list above are needed for or would help with this.

(Some other Windows-specific advantages of the latter - installers get special compatibility treatment when the OS does stuff to break them, modifying official Python binaries breaks the signatures, signed executables are fully scanned before running (slow if the file is big), IT departments know how to deal with installers and users know how to deal with installed binaries, and probably more.)

Alright everyone, back on topic now unless you want to rename the thread :)

Cheers, Steve

 

Well Python 3.4.3 binary is 4kb for me, so you'd have that + your 1KB Python script + whatever other pieces you need. It would be entirely possible to only include the parts of the standard library you actually need. There's no rule that if your single file executable doesn't use xmlrpc that you have to include xmlrpc just for purities sake. This isn't something that can't be done today using something like PyInstaller, it's just super janky and finnicky because it's being hacked in after the fact by PyInstaller and because it's not an officially supported thing a lot of projects simply don't support it. A CLI I worked on that uses PyInstaller is 5MB. It's certainly a trade off but it's not nearly as big of a trade off as you say. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On May 28, 2015 at 2:11:02 PM, Terry Reedy (tjreedy@udel.edu) wrote:

On 5/28/2015 10:55 AM, Steve Dower wrote:

...

And it would look like a 20MB+ file just for a simple 1KB Python script...

For Windows at least, I'd prefer to have some app-style installer generation (e.g. http://pynsist.readthedocs.org/en/latest/) which, combined with the embeddable Python distro (new for 3.5.0b1 in case anyone missed it), can simply extract everything into an install directory and run it from there. None of the items on the list above are needed for or would help with this.

What I wish, of course, is that Windows just came with Python3, the way that DOS came with BASIC, so people could publish and trade Python programs the way we once did with BASIC programs. Then a simple 1KB Python script would just take an extra 1KB on disk. To me, the removal of a simple, builtin programming language for everyone was the biggest Windows mistake.

Failing that, maybe PSF & edu-sig could somehow encourage universities that requires students to have a computer to recommend or even require that Python be loaded so students could at least assume that others students have Python already loaded. Python is the BASIC of the 21 century.

Somehow, trying to make it easier to have 50 duplicate copies of Python on a system seems the wrong direction to go.

-- Terry Jan Reedy

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

Honestly, I’m on an OS that *does* ship Python (OS X) and part of me hopes that they stop shipping it. It’s very rare that someone ships Python as part of their OS without modifying it in some way, and those modifications almost always cause pain to some set of users (and since I work on pip, they tend to come to us with the weirdo problems). Case in point: Python on OS X adds some preinstalled software, but they put this pre-installed software before site-packages in sys.path, so pip can’t upgrade those pre-installed software packages at all.  --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Thu, May 28, 2015 at 12:25 PM, Sturla Molden wrote:

Many Unix tools need Python, so Mac OS X (like Linux distros and FreeBSD) will always need a system Python. Yes, it would be great if could be called spython or something else than python. But the main problem is that it is used by end-users as well, not just the operating system.

I think it's great for it to be used by end users as a system library / utility. i.e. like you would a the system libc -- so if you can write a little python script that only uses the stdlib -- you can simply deliver that script. But if you want to go an install a bunch of extra non-standard packages (or, for heaven's sake, want a version with bug fixes!), they you really are better off installing a new python you can control. The system

Python should be left alone as it is.

absolutely! By the way, py2app will build an application bundle that depends on the system python, indeed, that's all it will do if you run it with the system python, as Apple has added some non-redistributable bits in there. But things get kin dof confusing if you want to rely on non-system packages... -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

Op 28 mei 2015 om 21:37 heeft Chris Barker het volgende geschreven:

On Thu, May 28, 2015 at 12:25 PM, Sturla Molden wrote:

...

The system Python should be left alone as it is.

absolutely!

By the way, py2app will build an application bundle that depends on the system python, indeed, that's all it will do if you run it with the system python, as Apple has added some non-redistributable bits in there.

That's not quite the reason. It's more that I don't want to guess whether or not it is valid to bundle binaries from a system location. Furthermore bundling files from a base install of the OS is pretty useless, especially when those binaries won't run on earlier releases anyway due to the compilation options used. Ronald

Single-file binaries are indeed important. (Though in most cases they don't have to be totally stand-alone -- they can depend on a system python and its stdlib. At least in typical datacenter setups.) Have people looked at PEX (a format developed by Twitter) or Pants (which seems to be an open-source tool that can build PEX files)? Łukasz has told me that at Facebook they have a similar system that they are now using to deploy Python 3 binaries. -- --Guido van Rossum (python.org/~guido)

On 28 May 2015 at 17:13, Barry Warsaw wrote:

On May 28, 2015, at 10:37 AM, Donald Stufft wrote:

...

I think docker is a pretty crummy answer to Go’s static binaries. What I would love is for Python to get:

* The ability to import .so modules via zipzimport (ideally without a temporary directory, but that might require newer APIs from libc and such).

+1 - Thomas Wouters mentioned at the language summit some work being done on glibc to add dlopen_from_memory() (sp?) which would allow for loading .so files directly from a zip. Not sure what the status is of that, but it would be a great addition.

+1 but it needs to be cross-platform - py2exe has something similar for Windows, which we should expect to use if the glibc solution is Unix-only.

...

* The ability to create a “static” Python that links everything it needs into the binary to do a zipimport of everything else (including the stdlib).

+1

+0 - I suppose it would be nice, but how important is this really? If all we do is end up with a single massive file instead of a directory, then I don't see we've gained much. Smallish C programs tend to hit the 10-100k executable size. Offhand I don't know how big a typical go "single file executable" is. If I could bundle up my Python script (something simple, let's say a "Hello world" for argument's sake) and get a 10-20k single-file executable, this would be worthwhile. If the resulting exe was 5MB (which is what today's solutions that bundle the Python DLL and full zipped stdlib tend to weigh in at) then I'm not interested.

...

*The ability to execute a zipfile that has been concat onto the end of the Python binary.

+1

+1. This would be immensely useful to tack on the front of a pyz archive. Even just the existing (39K on Windows) python.exe which relies on the Python installation would be great. Basically it could replace the setuptools/distlib "script wrapper" executables with something that doesn't need to run 2 processes just to run a script.

...

We’d probably want some more quality of life improvements around accessing resources from within that zip file as well, but that can be done as a library easier than the above three things can.

E.g. you really should be using the pkg_resources APIs for loading resources from your packages, otherwise you're gonna have problems with zip executables. We've talked before about adopting some of these APIs into Python's stdlib. pkgutil is a start, and the higher level APIs from pkg_resources should probably go there.

+1. We need the APIs available in the stdlib, then 3rd party libraries have a solution when users complain "your library isn't zipimport safe". (Of course "well, we have to support Python 2.7/3.4" remains a response for some time yet :-() Then we can start working on the culture change where library authors start expecting their code to be deployed in single-file "run-from-zip" applications. Paul

On 05/28/2015 06:13 PM, Barry Warsaw wrote:

Go seems to be popular where I work. It is replacing Python in a number of places, although Python (and especially Python 3) is still a very important part of our language toolbox.

There are several reasons why Go is gaining popularity. Single-file executables is definitely a reason; it makes deployment very easy, even if it increases the maintenance burden (e.g. without shared libraries, you have multiple copies of things so when a security fix is required for one of those things you have to recompile the world).

And the very same place where you are working is investing in getting shared libraries working for Go. Single binaries may be popular for distributing end user applications, but definitely not for distributing a core OS or a SDK. Sorry, you didn't yet arrive in distro land ... Matthias

Hello, On Thu, 28 May 2015 23:48:59 +0200 Matthias Klose wrote: []

And the very same place where you are working is investing in getting shared libraries working for Go. Single binaries may be popular for distributing end user applications, but definitely not for distributing a core OS or a SDK. Sorry, you didn't yet arrive in distro land ...

Of course it did. Like, Ubuntu 14.04LTS ships Go 1.2. No, it starts with the fact that when you don't have Go installed and type "go", it suggests to install gccgo, which just segfaults on running. Then you figure out that you need to install "golang", and that's 1.2, and a lot of things simply don't work with that version, like "go get" reports that a package not found, while it perfectly exists. So, let Go stay what it is - a corporate toy lingo for press-releases. That's until Google has thought that it generated enough buzz and it's time to shut it down like their numerous other projects. (Isn't Go old already and "everyone" uses Rust?) -- Best regards, Paul mailto:pmiscml@gmail.com

On May 28, 2015, at 11:48 PM, Matthias Klose wrote:

And the very same place where you are working is investing in getting shared libraries working for Go. Single binaries may be popular for distributing end user applications, but definitely not for distributing a core OS or a SDK.

Yep, I mentioned this in an earlier message (probably buried in the centithread avalanche). Both static and dynamic linking have valid use cases. Cheers, -Barry

On May 28, 2015 at 7:40:26 PM, Nick Coghlan (ncoghlan@gmail.com) wrote:

...

One thing I've seen more than once is that new development happens

in Python

...

until the problem is understood, then the code is ported to Go. Python's short path from idea to working code, along with its ability to quickly morph as requirements and understanding changes, its batteries included philosophy, and its "fits-your-brain" consistency are its biggest strengths!

Right, Go is displacing C/C++ in that regard (moreso than Python itself), and now that Rust has hit 1.0, I expect we'll see it becoming another contender for this task. Rust's big advantage over Go in that regard is being compatible with the C/C++ ecosystem, including Python's cffi.

I’m not sure if I’m reading this right or not, but just to be clear, I’ve seen a number of people express the sentiment that they are switching from Python to Go and that the deployment story is one of the reasons. It’s not just people switching from C/C++. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On May 29, 2015 at 2:58:28 AM, Nick Coghlan (ncoghlan@gmail.com) wrote:

On 29 May 2015 9:48 am, "Donald Stufft" wrote:

...

On May 28, 2015 at 7:40:26 PM, Nick Coghlan (ncoghlan@gmail.com) wrote:

...

...

One thing I've seen more than once is that new development happens

in Python

...

until the problem is understood, then the code is ported to Go. Python's short path from idea to working code, along with its ability to quickly morph as requirements and understanding changes, its batteries included philosophy, and its "fits-your-brain" consistency are its biggest strengths!

Right, Go is displacing C/C++ in that regard (moreso than Python itself), and now that Rust has hit 1.0, I expect we'll see it becoming another contender for this task. Rust's big advantage over Go in that regard is being compatible with the C/C++ ecosystem, including Python's cffi.

I’m not sure if I’m reading this right or not, but just to be clear, I’ve seen a number of people express the sentiment that they are switching from Python to Go and that the deployment story is one of the reasons. It’s not just people switching from C/C++.

C and C++ used to be the main "second version" languages used to create statically linked standalone binaries after an initial prototype in Python.

Folks that learned Python first understandably weren't keen on that idea, so they tended to either use Cython (or its predecessor, Pyrex), or else not bother doing it at all until first Go and now Rust came along (for reasons unknown to me, D appears to have never gained any popularity outside the ACM crowd).

If I seem blase about Go, that's the main reason why - the benefits it offers aren't novel from the point of view of C/C++ programmers, they're just now available without having to put up with arcane syntax, manual memory management, an unmaintainable threading model, relatively poor support for text manipulation, etc, etc.

I don't think Go is going to "kill" Python or anything, but I do think that not taking a look at other languages and why people are picking them over Python is important, otherwise we will end up dying (and would deserve to) because we'd be like the big company that didn't bother to keep up with the times and just assumed we'd be big forever. I talk to a lot of people about the distribution story of Python applications, what works and what doesn't. A very large majority of the people who have used both Go and Python in a serious capacity have indicated that they've at least considered writing new things in Go instead of Python due to the fact that distributing it is much easier and a not insignificant number of them have in fact started to switch to using Go in situations where they are trying to distribute things to disparate boxes. This might be something that people could have done before with C/C++ but with a nicer language behind it... but that's kind of the point? You don't need to be stuck with a terrible language to get a nice single file executable anymore, you can get that and use a good language at the same time which makes it a lot more compelling to a lot more people than having to be stuck with C. --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 29 May 2015 22:50, "Donald Stufft" wrote:

This might be something that people could have done before with C/C++ but

with

a nicer language behind it... but that's kind of the point? You don't need to be stuck with a terrible language to get a nice single file executable anymore, you can get that and use a good language at the same time which makes it a lot more compelling to a lot more people than having to be stuck with C.

Right, but the only things you can really write in Go are network services and console applications - once you start looking at curses & GUI applications on the end user side, you're back to the same kind of distribution complexity as C/C++ (where you have to choose between external dependency management or very large downloads), and once you start looking at the infrastructure side, Docker, Rocket & Kubernetes are bringing this kind of easy deployability to network services written in arbitrary languages. Hence my comment about MicroPython: the easiest way to make an interpreter runtime that's lighter than CPython is to have it *do less*. Communicating with embedded MicroPython instances via cffi could even potentially offer a way for both CPython and PyPy to split work across multiple cores without having to fundamentally redesign their main interpreters. Cheers, Nick.

On May 28, 2015, at 1:54 AM, Berker Peksağ wrote:

* Performance improvements are not bug fixes

Practicality beats purity here. Recognize that a huge number of Python users will remain in the Python2.7 world for some time. We have a responsibility to the bulk of our users (my estimate is that adoption rate for Python 3 is under 2%). The computed goto patch makes substantial performance improvements. It is callous to deny the improvement to 2.7 users.

* The patch doesn't make the migration process from Python 2 to Python 3 easier

Sorry, that is a red-herring (an orthogonal issue). If you care about 2-to-3 migration, then start opposing proposals for API changes that increase the semantic difference between 2 and 3. Raymond

On Thu, May 28, 2015 at 9:02 AM, Brett Cannon wrote:

But you could argue that "Special cases aren't special enough to break the rules" and that's what we are proposing here by claiming Python 2.7 is a special case and thus we should accept a patch that is not a one-liner change to gain some performance in a bugfix release.

One can read anything he wants into the Zen. I could respond with this retort: "Although practicality beats purity," but I won't. :-) Skip

On 28/05/2015 15:47, Skip Montanaro wrote:

On Thu, May 28, 2015 at 9:02 AM, Brett Cannon wrote:

...

But you could argue that "Special cases aren't special enough to break the rules" and that's what we are proposing here by claiming Python 2.7 is a special case and thus we should accept a patch that is not a one-liner change to gain some performance in a bugfix release.

One can read anything he wants into the Zen. I could respond with this retort: "Although practicality beats purity," but I won't. :-)

Skip

That's good, otherwise you'd just be repeating what Raymond said further up this subthread two hours and one minute before you didn't say it :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence

On Thu, May 28, 2015 at 9:08 AM Guido van Rossum wrote:

Wow. Such thread. :-)

This patch could save companies like Dropbox a lot of money. We run a ton of Python code in large datacenters, and while we are slow in moving to Python 3, we're good at updating to the latest 2.7.

Dropbox should be compiling its own interpreter with whatever patches it deems appropriate. The people it'll save resources for are companies not enlightened enough to do that: thousands of them, generally small or non-tech focused :) The patch is forward and backward compatible.I'm strongly in favor.

+1 I'm in favor as well. I mostly wanted to make sure that people were aware of profile-opt builds and that it was being compared. Sounds like both benefit, even used together. Win win. This is a 100% API compatible change. It just rearranges the interpreter loop on compilers enlightened enough to allow it. I was always bummed that it didn't make it into 2.7 itself. But given the world+dog is going to have 2.7 around and kicking for a long time, lets save the world some CPU cycles (read: carbon) for little effort. Very practical. Good for the world. People who need to save orders of magnitude more cycles shouldn't use an interpreter. ie: PyPy. Or consider the costs of moving to a compiled language. -gps

On 05/28/2015 05:58 PM, Victor Stinner wrote:

Why not continue to enhance Python 3 instead of wasting our time with Python 2? We have limited resources in term of developers to maintain Python.

Uh, guys, you might as well call off the debate. Benjamin already checked it in. https://hg.python.org/cpython/rev/17d3bbde60d2 //arry/

On Thu, May 28, 2015 at 6:10 PM Larry Hastings wrote:

On 05/28/2015 05:58 PM, Victor Stinner wrote:

Why not continue to enhance Python 3 instead of wasting our time with Python 2? We have limited resources in term of developers to maintain Python.

Uh, guys, you might as well call off the debate. Benjamin already checked it in.

https://hg.python.org/cpython/rev/17d3bbde60d2

Thanks Benjamin! :)

*/arry* _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/greg%40krypto.org

On 29 May 2015 11:01 am, "Victor Stinner" wrote:

Why not continue to enhance Python 3 instead of wasting our time with Python 2? We have limited resources in term of developers to maintain Python.

(I'm not talking about fixing *bugs* in Python 2 which is fine with me.)

I'm actually OK with volunteers deciding that even fixing bugs in 2.7 isn't inherently rewarding enough for them to be willing to do it for free on their own time. Stepping up to extrinsically reward activities that are beneficial for customers but aren't intrinsically interesting enough for people to be willing to do for free is one of the key reasons commercial open source redistributors get paid. That more explicitly commercial presence is a dynamic we haven't historically had to deal with in core development, so there are going to be some growing pains as we find an arrangement that everyone is comfortable with (or is at least willing to tolerate, but I'm optimistic we can do better than that). Cheers, Nick.

--

By the way, I just wrote sixer, a new tool to generate patches to port OpenStack to Python 3 :-) https://pypi.python.org/pypi/sixer

It's based on regex, so it's less reliable than 2to3, 2to6 or modernize, but it's just enough for my specific use case. On OpenStack, it's not possible to send one giant patch "hello, this is python 3". Code is modified by small and incremental changes.

Come on in the Python 3 world and... always look on the bright side of life ( https://www.youtube.com/watch?v=VOAtCOsNuVM )!

Victor _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On Fri, May 29, 2015 at 4:14 PM, Gregory P. Smith wrote:

On Fri, May 29, 2015 at 12:24 AM Nick Coghlan wrote:

...

On 29 May 2015 11:01 am, "Victor Stinner" wrote:

...

Why not continue to enhance Python 3 instead of wasting our time with Python 2? We have limited resources in term of developers to maintain Python.

(I'm not talking about fixing *bugs* in Python 2 which is fine with me.)

I'm actually OK with volunteers deciding that even fixing bugs in 2.7 isn't inherently rewarding enough for them to be willing to do it for free on their own time.

That is 100% okay.

What is not okay is for python-dev representatives to respond to users (in any list/forum/channel) reporting bugs in 2.7 or asking if a fix in 3 can be backported to 2.7 with things akin to "just use Python 3" or "sorry, 2.7 is critical fixes only. move to python 3 already." This is actively driving our largest users away. I bring this up because a user was bemoaning how useless they feel python core devs are because of this attitude recently. Leading to feelings of wishing to just abandon CPython if not Python all together.

I'm sure I have even made some of those responses myself (sorry!). My point here is: know it. recognize it. don't do it anymore. It harms the community.

A correct and accurate response to desires to make non-api-breaking changes in 2.7 is "Patches that do not change any APIs for 2.7 are welcome in the issue tracker." possibly including "I don't have the bandwidth to review 2.7 changes, find someone on python-dev to review and champion this for you if you need it." Finding someone may not always be easy. But at least is still the "patches welcome" attitude and suggests that the work can be done if someone is willing to do it. Lets make a concerted effort to not be hostile and against it by default.

Ex: Is someone with a python application that is a million of lines supposed to have everyone involved in that drop the productive work they are doing and spend that porting their existing application to python 3 because we have so far failed to provide the tools to make that migration easy? No. Empathize with our community. Feel their pain. (and everyone who is working on tools to aid the transition: keep doing that! Our users are gonna need it unless we don't want them as users anymore.)

We committed to supporting 2.7 until 2020 in 2014 per https://hg.python.org/peps/rev/76d43e52d978. That means backports of important bug or performance fixes should at least be allowed on the table, even if hairy, even if you won't work on them yourselves on a volunteer basis. This is the first long term support release of Python ever. This is what LTS means. LTS could also stand for Learn To Support...

At the same time, they can ask for it, but if people aren't motivated to do the work for it, it won't happen. We should be encouraging (and maybe even mentoring) these people who are desperately in need of the fixes to be backported, to backport the patches themselves. With that done, it can go through review and we can maybe get those fixes in faster if we can also get a larger group of reviews. The problem consists of a few parts: - We're all volunteers - Volunteers are going to work on what interests them - Python 2.7 maintenance doesn't seem to interest many of our volunteers currently Perhaps we should explain this to each of the people requesting backports to (ideally) encourage them.

On May 29, 2015, at 04:04 PM, Guido van Rossum wrote:

There are a fair number of people on this thread whose employer pays them to work on Python.

My guess is that as Python 2.7 gets longer in the tooth, and it becomes harder to motivate volunteers to shepherd contributed patches into Python 2, folks getting paid by employers who need Python 2 to continue to be maintained, will step up and go through the mentorship process so that they can more directly apply such patches. I.e. they can start to take over some of the active maintenance of the Python 2.7 branch. Cheers, -Barry

On Fri, May 29, 2015 at 6:04 PM, Guido van Rossum wrote:

On Fri, May 29, 2015 at 2:52 PM, Ian Cordasco wrote:

...

On Fri, May 29, 2015 at 4:14 PM, Gregory P. Smith wrote:

...

On Fri, May 29, 2015 at 12:24 AM Nick Coghlan wrote:

...

On 29 May 2015 11:01 am, "Victor Stinner" wrote:

...

Why not continue to enhance Python 3 instead of wasting our time with Python 2? We have limited resources in term of developers to maintain Python.

(I'm not talking about fixing *bugs* in Python 2 which is fine with me.)

I'm actually OK with volunteers deciding that even fixing bugs in 2.7 isn't inherently rewarding enough for them to be willing to do it for free on their own time.

That is 100% okay.

What is not okay is for python-dev representatives to respond to users (in any list/forum/channel) reporting bugs in 2.7 or asking if a fix in 3 can be backported to 2.7 with things akin to "just use Python 3" or "sorry, 2.7 is critical fixes only. move to python 3 already." This is actively driving our largest users away. I bring this up because a user was bemoaning how useless they feel python core devs are because of this attitude recently. Leading to feelings of wishing to just abandon CPython if not Python all together.

I'm sure I have even made some of those responses myself (sorry!). My point here is: know it. recognize it. don't do it anymore. It harms the community.

A correct and accurate response to desires to make non-api-breaking changes in 2.7 is "Patches that do not change any APIs for 2.7 are welcome in the issue tracker." possibly including "I don't have the bandwidth to review 2.7 changes, find someone on python-dev to review and champion this for you if you need it." Finding someone may not always be easy. But at least is still the "patches welcome" attitude and suggests that the work can be done if someone is willing to do it. Lets make a concerted effort to not be hostile and against it by default.

Ex: Is someone with a python application that is a million of lines supposed to have everyone involved in that drop the productive work they are doing and spend that porting their existing application to python 3 because we have so far failed to provide the tools to make that migration easy? No. Empathize with our community. Feel their pain. (and everyone who is working on tools to aid the transition: keep doing that! Our users are gonna need it unless we don't want them as users anymore.)

We committed to supporting 2.7 until 2020 in 2014 per https://hg.python.org/peps/rev/76d43e52d978. That means backports of important bug or performance fixes should at least be allowed on the table, even if hairy, even if you won't work on them yourselves on a volunteer basis. This is the first long term support release of Python ever. This is what LTS means. LTS could also stand for Learn To Support...

At the same time, they can ask for it, but if people aren't motivated to do the work for it, it won't happen. We should be encouraging (and maybe even mentoring) these people who are desperately in need of the fixes to be backported, to backport the patches themselves. With that done, it can go through review and we can maybe get those fixes in faster if we can also get a larger group of reviews.

The problem consists of a few parts:

- We're all volunteers

Speak for yourself. There are a fair number of people on this thread whose employer pays them to work on Python. And this thread originated when a patch was being contributed by people who were also paid by their employer to do all the dirty work (including benchmarks). And yet they were (initially) given the cold shoulder by some "high and mighty" Python 3 zealots. This attitude need to change.

...

- Volunteers are going to work on what interests them - Python 2.7 maintenance doesn't seem to interest many of our volunteers currently

Perhaps we should explain this to each of the people requesting backports to (ideally) encourage them.

Please let someone else do the explaining. I don't want to have to do the damage control after you "explain" something.

Good to know. I'll stop trying to make spare time to review patches then.

On 2015-05-29 23:14, Gregory P. Smith wrote:

On Fri, May 29, 2015 at 12:24 AM Nick Coghlan mailto:ncoghlan@gmail.com> wrote:

On 29 May 2015 11:01 am, "Victor Stinner" mailto:victor.stinner@gmail.com> wrote: > > Why not continue to enhance Python 3 instead of wasting our time with > Python 2? We have limited resources in term of developers to maintain > Python. > > (I'm not talking about fixing *bugs* in Python 2 which is fine with me.)

I'm actually OK with volunteers deciding that even fixing bugs in 2.7 isn't inherently rewarding enough for them to be willing to do it for free on their own time.

That is 100% okay.

What is not okay is for python-dev representatives to respond to users (in any list/forum/channel) reporting bugs in 2.7 or asking if a fix in 3 can be backported to 2.7 with things akin to "just use Python 3" or "sorry, 2.7 is critical fixes only. move to python 3 already." This is actively driving our largest users away. I bring this up because a user was bemoaning how useless they feel python core devs are because of this attitude recently. Leading to feelings of wishing to just abandon CPython if not Python all together.

I'm sure I have even made some of those responses myself (sorry!). My point here is: know it. recognize it. don't do it anymore. It harms the community.

A correct and accurate response to desires to make non-api-breaking changes in 2.7 is "Patches that do not change any APIs for 2.7 are welcome in the issue tracker." possibly including "I don't have the bandwidth to review 2.7 changes, find someone on python-dev to review and champion this for you if you need it." Finding someone may not always be easy. But at least is still the "patches welcome" attitude and suggests that the work can be done if someone is willing to do it. Lets make a concerted effort to not be hostile and against it by default.

Ex: Is someone with a python application that is a million of lines supposed to have everyone involved in that drop the productive work they are doing and spend that porting their existing application to python 3 because we have so far failed to provide the tools to make that migration easy? No. Empathize with our community. Feel their pain. (and everyone who is working on tools to aid the transition: keep doing that! Our users are gonna need it unless we don't want them as users anymore.)

We committed to supporting 2.7 until 2020 in 2014 per https://hg.python.org/peps/rev/76d43e52d978. That means backports of important bug or performance fixes should at least be allowed on the table, even if hairy, even if you won't work on them yourselves on a volunteer basis. This is the first long term support release of Python ever. This is what LTS means. LTS could /also/ stand for Learn To Support...

Over the last years I have changed my mind a bit, too. For Python 2.7 LTS I welcome performance improving patches as well as security improvements (SSL module) and build related fixes. For performance patches we have to consider our responsibility for the environment. Every improvement means more speed and less power consumption. Python runs of hundreds of thousands of machines in the cloud. Python 2.7 will be used for at least half a decade, probably longer. Servers can be replaced with faster machines later and less fossil fuel must be burned to produce power. Let's keep Python green! :) Thanks to Benjamin, the patch has already landed. Antoine's improved GIL may be another improvement for Python 2.7. Servers are getting more cores every year. The new GIL helps to scale multiple CPU bound threads on machines with more cores, e.g. http://www.dabeaz.com/python/NewGIL.pdf Christian

On 30 May 2015 09:57, "Antoine Pitrou" wrote:

On Sat, 30 May 2015 01:49:10 +0200 Christian Heimes wrote:

...

For performance patches we have to consider our responsibility for the environment. Every improvement means more speed and less power consumption. Python runs of hundreds of thousands of machines in the cloud. Python 2.7 will be used for at least half a decade, probably longer. Servers can be replaced with faster machines later and less fossil fuel must be burned to produce power.

Please keep your ideology out of this.

I'm a qualified engineer (in computer systems engineering), so caring about environmental sustainability is part of my professional ethical standards, not just a matter of personal preference: http://www.wfeo.org/ethics/ Given the power draw of large data centres, the environmental impact of performance improvements to the Python 2 series is a point well worth considering. Cheers, Nick.

Regards

Antoine.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On 30 May 2015 10:46, "Alexander Walters" wrote:

Python is a giant cache-miss generator. A little performance boost on the opt-code dispatch isn't going to change that much. If we really do care about improving python to do less environmental damage, then that is a discussion we should be having on it's own merits. It was really out of place, even in this tangenty thread.

I think the way core development gets funded is entirely on topic for the main core development mailing list, we just historically haven't discussed it openly, even though some of us have been advocating for various improvements to arrangements behind the scenes. I personally consider becoming more transparent about how we go about that process to be a good thing. Intel are looking to get involved in CPython core development *specifically* to work on performance improvements, so it's important to offer folks in the community good reasons for why we're OK with seeing at least some of that work applied to Python 2, rather than restricting their contributions to Python 3. The key is that the reason for not backporting performance enhancements *differs* from the reasons for not backporting new features. Rolling out new features has a broad ripple effect on the Python ecosystem as books, training material, etc, all need to be updated, and projects need to decide how to communicate their version dependencies appropriately if they decide to depend on one of the backported features. We pushed that kind of Python 2 change out to PyPI years ago, and aside from truly exceptional cases like the network security enhancements in PEPs 466 & 476 and the decision to bundle pip to make it easier to access PyPI, it isn't open for reconsideration as a general principle. Performance improvements, by contrast, historically haven't been backported solely due to the stability and maintainability implications for CPython itself - they don't have a change management ripple effect the way new language and standard library level features do. That lack of negative ripple effects that cause work for other people is why the proposal to contribute paid development time makes such a big difference to the acceptability of Python 2.7 performance patches, as it should be a pure gain for current Python 2.7 users, and the paid development contributions should address the maintainability concerns on the core development side (particularly since Intel are *paying* for their coaching in core contribution practices and politics, rather than expecting to receive that coaching from community volunteers for free). Backporting the computed goto patch is an easy place for them to start, since the change is already well tested in the Python 3 branch, but we don't expect this to be the end of the line for CPython 2 (or 3) performance enhancements. However, we also shouldn't downplay the significance of this as a notable policy change for the Python 2.7 maintenance branch, which means it is useful to offer folks as many reasons as we can to help them come to terms with the idea that Python 2 performance still matters, and that it is only the limitations on our development and support capacity that prevented us from further improving it previously. The commercially pragmatic reason is because Python 2 is where the largest current installed base is today, so applying some of the increased development capacity arising from sponsored contributions to Python 2.7 performance improvements is a good way to demonstrate to Python 2 developers that we still care about them *as Python 2 users*, rather than only being interested in them as potential future Python 3 users. This is the rationale that's likely to get our paid contributors (both current and future) on board with the idea, but it isn't necessarily going to be compelling to folks that are here as volunteers. The first "What's in it for the volunteers?" reason is the one I raised: giving the nod to an increased corporate developer presence in Python 2 maintenance should eventually let volunteers stop worrying about even Python 2.7 bug fix changes with a clear conscience, confident that as volunteer efforts drop away redistributors and other folks with an institutional interest will pick up the slack with paid development time. "Do the fun stuff for free, figure out a way to get paid for the boring-but-necessary stuff (or leave those tasks to someone else that's getting paid to handle them)" is a good sustainable approach to open source development, while trying to do it *all* for free is a fast path to burnout. Being ready, willing and able to handle the kind of situation created by the Python 2->3 community transition is a large part of what it means to offer commercial support for community driven open source projects, as it buys customers' time for either migration technologies to mature to a point where the cost of migration drops dramatically, for the newer version of a platform to move far enough ahead of the legacy version for there to be a clear and compelling business case for forward porting existing software, or (as is the case we're aiming to engineer for Python), both. The environmental argument is another one that may be appealing to folks that have no commercial interest in improving Python 2 performance. Regardless of which programming language we use to write our own software, we all still share the same planet, so reducing the amount of power we collectively use is something we can all benefit from. Even though none of us have the necessary data to even guess at the absolute magnitude of that reduction, we can at least be confident it's a non-trivial portion of the amount of power Python 2 applications are currently consuming. Regards, Nick.

On 30 May 2015 at 20:58, Antoine Pitrou wrote:

On Sat, 30 May 2015 18:55:20 +1000 Nick Coghlan wrote:

...

On 30 May 2015 10:46, "Alexander Walters" wrote:

...

Python is a giant cache-miss generator. A little performance boost on the opt-code dispatch isn't going to change that much. If we really do care about improving python to do less environmental damage, then that is a discussion we should be having on it's own merits. It was really out of place, even in this tangenty thread.

I think the way core development gets funded is entirely on topic for the main core development mailing list, we just historically haven't discussed it openly, even though some of us have been advocating for various improvements to arrangements behind the scenes. I personally consider becoming more transparent about how we go about that process to be a good thing.

The way this so-called discussion is taking place feels much less like an actual discussion than an aggressive push for a change in maintenance policy. Guido has already taunted Ian Cordasco out of contributing.

Ian was unfortunately responding from incomplete information. While "we're all volunteers here" was true for a very long time, with Guido being the main exception since the PythonLabs days, a number of folks (both existing core contributors and members of other organisations) have been working hard to change that, since it's an unsustainable state of affairs given the criticality of CPython as a piece of Internet infrastructure. Given the extensive complaints about the lack of corporate contribution to upstream CPython maintenance, the hostile reaction to a concrete proposal for such ongoing contributions has been both incredibly surprising *and* disappointing, especially when it was deliberately aimed at tasks that most volunteers find to be a unrewarding chore rather than an entertaining use of their free time. The offer came with one string attached: that the Python 2.7 branch be opened up for performance improvements in addition to bug fixes. Since maintainability was the main concern with not backporting performance improvements in the first place, this seemed like a straight up win to me (and presumably to other folks aware of the offer), so it never even occurred to us that folks might not accept "because this proposal is backed by a credible offer of ongoing contributions to CPython maintenance and support" as a complete answer to the question of "Why accept this proposal to backport performance enhancements, and not previous proposals?". Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 30 May 2015 at 21:37, Antoine Pitrou wrote:

On Sat, 30 May 2015 21:20:56 +1000 Nick Coghlan wrote:

...

Given the extensive complaints about the lack of corporate contribution to upstream CPython maintenance, the hostile reaction to a concrete proposal for such ongoing contributions has been both incredibly surprising *and* disappointing

IMHO, they were not more hostile than against individuals' contributions of the same kind. Any patch proposal is bound to controversy, that's a normal aspect of the process, and one that contributors should usually be willing to go through.

Also, when there are in rules in place, most people want to see them upholded, because that tends to promote fairness much more than when exceptions are granted all over the place. So people's reactions have really been understandable, if debatable.

Agreed, but it's also understandable when folks forget that things that they're taking for granted aren't necessarily common knowledge. In this case: * the fact that this proposal was a suggested starting point for ongoing contributions, not a one-and-done effort * the fact that the rationale for the prohibition on performance enhancements was *different* from the reason for disallowing new features (and hence requiring a PEP for *any* new Python 2.7 feature) For folks that already knew both those facts, this *wasn't* a controversial suggestion. We unfortunately failed to account for the fact that not everyone was aware of that context, and that *is* a highly regrettable mistake. Hence my information dumps thoughout the thread, attempting to provide that context without committing folks to things they haven't committed to, and without disclosing potentially confidential third party information.

...

The offer came with one string attached: that the Python 2.7 branch be opened up for performance improvements in addition to bug fixes. Since maintainability was the main concern with not backporting performance improvements in the first place, this seemed like a straight up win to me (and presumably to other folks aware of the offer), so it never even occurred to us that folks might not accept "because this proposal is backed by a credible offer of ongoing contributions to CPython maintenance and support" as a complete answer to the question of "Why accept this proposal to backport performance enhancements, and not previous proposals?".

You're making contribution some kind of contractual engagement. That's not an obvious improvement, because it has some large impacts on the power structure (for one, volunteers can't reasonably compete with contractual engagements).

We've long had a requirement that certain kinds of proposal come with at least nominal support commitments from the folks proposing them (e.g. adding modules to the standard library, supporting new platforms). Institutions with a clear financial interest in a particular problem area can certainly make such commitments more credibly, so I agree with your concerns about the potential impact on the power dynamics of core development. That's one of the main benefits I see in attempting to guide sponsored contributions towards Python 2.7, at least initially - that's in LTS mode, so working on it is fairly uninteresting anyway, and it keeps discussion of *new* features (and hence the overall direction of language evolution) a community focused activity. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 31 May 2015 at 08:37, Greg Ewing wrote:

Nick Coghlan wrote:

...

We've long had a requirement that certain kinds of proposal come with at least nominal support commitments from the folks proposing them (e.g. adding modules to the standard library, supporting new platforms). Institutions with a clear financial interest in a particular problem area can certainly make such commitments more credibly,

Are such commitments from commercial entities really any more reliable in the long term than anyone else's? Such entities can be expected to drop them as soon as they perceive them as no longer being in their financial interests.

Yes, if the credibility stems from the market situation and the financial incentives leading an organisation to make the offer, rather than from the personal interest of one or two key folks at that organisation. Structural incentives are harder to shift than personal interests, so this is a case where institutional inertia actually works for the community rather than against us. It's not an ironclad guarantee (since businesses fail, divisions get shut down, companies decide to exit markets, etc), but if we understand the business case backing an investment decision (whether that investment is in the form of funding, developer time, or both), that's genuinely more reliable than commitments from individuals (since we don't have the kind of ability to manage and distribute risk that larger organisations do). Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On May 30, 2015 1:56 AM, "Nick Coghlan" wrote:

Being ready, willing and able to handle the kind of situation created by the Python 2->3 community transition is a large part of what it means to offer commercial support for community driven open source projects, as it buys customers' time for either migration technologies to mature to a point where the cost of migration drops dramatically, for the newer version of a platform to move far enough ahead of the legacy version for there to be a clear and compelling business case for forward porting existing software, or (as is the case we're aiming to engineer for Python), both.

Earlier, you said that it had been a surprise that people were against this change. I'd just point out that the reason is bound up in what you say here. Porting performance features from python 3 to python 2 has the disadvantage of cutting into a compelling business case for users to move forward to python 3.[1] so doing this has a cost to python 3 adoption. But, the question is whether there is a benefit that outweighs that cost. I think seeing more steady, reliable contributors to python core is a very large payment. Sure, for now that payment is aimed at extending the legs on the legacy version of python but at some point in the future python 2's legs will be well and truly exhausted. When that happens both the developers who have gained the skill of contributing to cpython and the companies who have invested money in training people to be cpython contributors will have to decide whether to give up on all of that or continue to utilize those skills and investments by bettering python 3. I'd hope that we can prove ourselves a welcoming enough community that they'd choose to stay. -Toshio [1] In fact, performance differences are a rather safe way to build compelling business cases for forwards porting. Safe because it is a difference (unlike api and feature differences) that will not negatively affect your ability to incrementally move your code to python 3.

On 31 May 2015 at 09:20, Larry Hastings wrote:

On 05/30/2015 07:26 AM, Toshio Kuratomi wrote:

Porting performance features from python 3 to python 2 has the disadvantage of cutting into a compelling business case for users to move forward to python 3.[1] so doing this has a cost to python 3 adoption. But, the question is whether there is a benefit that outweighs that cost. [...]

Backporting performance enhancements from 3 to 2 does seem to be counterproductive from the perspective of the Core Dev community. But certainly in this case, when Intel drops a major bundle of working code in our collective lap, it absolutely feels like the right thing to me to check it in and support it. And happily the Python Core Dev community generally does the right thing.

There's another benefit that I didn't think to mention earlier, which is that getting folks from Python 2 -> Python 3 isn't actually my major version adoption concern at the moment: I'm more interested in how I can persuade them to stop using Python *2.6*, which is still a higher proportion of PyPI downloads with an identifiable client version than Python 3 [1], and the relative proportions between them are likely to be even worse once we start venturing inside corporate firewalls where direct downloads from PyPI aren't permitted. While I suspect barriers to migration at the distro level carry a fair bit of the blame there (and we're working on those), performance improvements in the 2.7 branch help provide an additional carrot to assist in that process, complementing the stick of trying to educate the community at large that it's unrealistic and exploitative [2] for folks to expect free community support for versions of Python that are so old that not even the core development team support them any more (i.e. Python 2.6 and earlier). My one consolation is that the Python community are far from alone in struggling to win that fight against institutional inertia once folks have widely adopted a version of a product that "works for them". My theory is that folks will pay to be able to keep using these older systems because our industry doesn't have very good tools for quantifying the cost of the technical debt incurred by attempting to maintain the status quo in the face of an evolving ecosystem. As infrastructure change management practices improve (e.g. through ideas like Holistic Software's hybrid dynamic management [3]), and not only the platform level tools but also the related business models evolve to better support those approaches, I'm hoping we'll see things change for the better not just in terms of Python in particular, but in terms of institutional infrastructure as a whole. Cheers, Nick. [1] https://caremad.io/2015/04/a-year-of-pypi-downloads/ [2] http://www.curiousefficiency.org/posts/2015/04/stop-supporting-python26.html [3] http://www.holistic-software.com/hybrid-dynamic-model -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 01.06.2015 12:44, Armin Rigo wrote:

Hi Larry,

On 31 May 2015 at 01:20, Larry Hastings wrote:

...

p.s. Supporting this patch also helps cut into PyPy's reported performance lead--that is, if they ever upgrade speed.pypy.org from comparing against Python *2.7.2*.

Right, we should do this upgrade when 2.7.11 is out.

There is some irony in your comment which seems to imply "PyPy is cheating by comparing with an old Python 2.7.2": it is inside a thread which started because "we didn't backport performance improvements to 2.7.x so far".

Just to convince myself, I just ran a performance comparison. I ran the same benchmark suite as speed.pypy.org, with 2.7.2 against 2.7.10, both freshly compiled with no "configure" options at all. The differences are usually in the noise, but range from +5% to... -60%. If anything, this seems to show that CPython should take more care about performance regressions. If someone is interested:

* "raytrace-simple" is 1.19 times slower * "bm_mako" is 1.29 times slower * "spitfire_cstringio" is 1.60 times slower * a number of other benchmarks are around 1.08.

The "7.0x faster" number on speed.pypy.org would be significantly *higher* if we upgraded the baseline to 2.7.10 now.

If someone were to volunteer to set up and run speed.python.org, I think we could add some additional focus on performance regressions. Right now, we don't have any way of reliably and reproducibly testing Python performance. Hint: The PSF would most likely fund such adventures :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 01 2015)

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

Hi There was a PSF-sponsored effort to improve the situation with the https://bitbucket.org/pypy/codespeed2/src being written (thank you PSF). It's not better enough than codespeed that I would like, but gives some opportunities. That said, we have a benchmark machine for benchmarking cpython and I never deployed nightly benchmarks of cpython for a variety of reasons. * would be cool to get a small VM to set up the web front * people told me that py3k is only interesting, so I did not set it up for py3k because benchmarks are mostly missing I'm willing to set up a nightly speed.python.org using nightly build on python 2 and possible python 3 if there is an interest. I need support from someone maintaining python buildbot to setup builds and a VM to set up stuff, otherwise I'm good to go DISCLAIMER: I did facilitate in codespeed rewrite that was not as successful as I would have hoped. I did not receive any money from the PSF on that though. Cheers, fijal On Mon, Jun 1, 2015 at 1:14 PM, M.-A. Lemburg wrote:

On 01.06.2015 12:44, Armin Rigo wrote:

...

Hi Larry,

On 31 May 2015 at 01:20, Larry Hastings wrote:

...

p.s. Supporting this patch also helps cut into PyPy's reported performance lead--that is, if they ever upgrade speed.pypy.org from comparing against Python *2.7.2*.

Right, we should do this upgrade when 2.7.11 is out.

There is some irony in your comment which seems to imply "PyPy is cheating by comparing with an old Python 2.7.2": it is inside a thread which started because "we didn't backport performance improvements to 2.7.x so far".

Just to convince myself, I just ran a performance comparison. I ran the same benchmark suite as speed.pypy.org, with 2.7.2 against 2.7.10, both freshly compiled with no "configure" options at all. The differences are usually in the noise, but range from +5% to... -60%. If anything, this seems to show that CPython should take more care about performance regressions. If someone is interested:

* "raytrace-simple" is 1.19 times slower * "bm_mako" is 1.29 times slower * "spitfire_cstringio" is 1.60 times slower * a number of other benchmarks are around 1.08.

The "7.0x faster" number on speed.pypy.org would be significantly *higher* if we upgraded the baseline to 2.7.10 now.

If someone were to volunteer to set up and run speed.python.org, I think we could add some additional focus on performance regressions. Right now, we don't have any way of reliably and reproducibly testing Python performance.

Hint: The PSF would most likely fund such adventures :-)

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Jun 01 2015)

...

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

---

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

On 02.06.2015 21:07, Maciej Fijalkowski wrote:

Hi

There was a PSF-sponsored effort to improve the situation with the https://bitbucket.org/pypy/codespeed2/src being written (thank you PSF). It's not better enough than codespeed that I would like, but gives some opportunities.

That said, we have a benchmark machine for benchmarking cpython and I never deployed nightly benchmarks of cpython for a variety of reasons.

* would be cool to get a small VM to set up the web front

* people told me that py3k is only interesting, so I did not set it up for py3k because benchmarks are mostly missing

I'm willing to set up a nightly speed.python.org using nightly build on python 2 and possible python 3 if there is an interest. I need support from someone maintaining python buildbot to setup builds and a VM to set up stuff, otherwise I'm good to go

DISCLAIMER: I did facilitate in codespeed rewrite that was not as successful as I would have hoped. I did not receive any money from the PSF on that though.

I think we should look into getting speed.python.org up and running for both Python 2 and 3 branches: https://speed.python.org/ What would it take to make that happen ?

Cheers, fijal

On Mon, Jun 1, 2015 at 1:14 PM, M.-A. Lemburg wrote:

...

On 01.06.2015 12:44, Armin Rigo wrote:

...

Hi Larry,

On 31 May 2015 at 01:20, Larry Hastings wrote:

...

p.s. Supporting this patch also helps cut into PyPy's reported performance lead--that is, if they ever upgrade speed.pypy.org from comparing against Python *2.7.2*.

Right, we should do this upgrade when 2.7.11 is out.

There is some irony in your comment which seems to imply "PyPy is cheating by comparing with an old Python 2.7.2": it is inside a thread which started because "we didn't backport performance improvements to 2.7.x so far".

Just to convince myself, I just ran a performance comparison. I ran the same benchmark suite as speed.pypy.org, with 2.7.2 against 2.7.10, both freshly compiled with no "configure" options at all. The differences are usually in the noise, but range from +5% to... -60%. If anything, this seems to show that CPython should take more care about performance regressions. If someone is interested:

* "raytrace-simple" is 1.19 times slower * "bm_mako" is 1.29 times slower * "spitfire_cstringio" is 1.60 times slower * a number of other benchmarks are around 1.08.

The "7.0x faster" number on speed.pypy.org would be significantly *higher* if we upgraded the baseline to 2.7.10 now.

If someone were to volunteer to set up and run speed.python.org, I think we could add some additional focus on performance regressions. Right now, we don't have any way of reliably and reproducibly testing Python performance.

Hint: The PSF would most likely fund such adventures :-)

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Jun 01 2015)

...

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

---

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

---

Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com

-- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 03 2015)

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

::::: Try our mxODBC.Connect Python Database Interface for free ! :::::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Wed, 03 Jun 2015 12:04:10 +0200, Maciej Fijalkowski wrote:

On Wed, Jun 3, 2015 at 11:38 AM, M.-A. Lemburg wrote:

...

On 02.06.2015 21:07, Maciej Fijalkowski wrote:

...

Hi

There was a PSF-sponsored effort to improve the situation with the https://bitbucket.org/pypy/codespeed2/src being written (thank you PSF). It's not better enough than codespeed that I would like, but gives some opportunities.

That said, we have a benchmark machine for benchmarking cpython and I never deployed nightly benchmarks of cpython for a variety of reasons.

* would be cool to get a small VM to set up the web front

* people told me that py3k is only interesting, so I did not set it up for py3k because benchmarks are mostly missing

I'm willing to set up a nightly speed.python.org using nightly build on python 2 and possible python 3 if there is an interest. I need support from someone maintaining python buildbot to setup builds and a VM to set up stuff, otherwise I'm good to go

DISCLAIMER: I did facilitate in codespeed rewrite that was not as successful as I would have hoped. I did not receive any money from the PSF on that though.

I think we should look into getting speed.python.org up and running for both Python 2 and 3 branches:

https://speed.python.org/

What would it take to make that happen ?

I guess ideal would be some cooperation from some of the cpython devs, so say someone can setup cpython buildbot

What does "set up cpython buildbot" mean in this context? --David

If someone were to volunteer to set up and run speed.python.org, I think we could add some additional focus on performance regressions. Right now, we don't have any way of reliably and reproducibly testing Python performance.

I'm very interested in speed.python.org and feel regret that the project is standing still. I have a mind to contribute something ... thanks, Tetsuya On Mon, Jun 1, 2015 at 8:14 PM, M.-A. Lemburg wrote:

...

Hi Larry,

On 31 May 2015 at 01:20, Larry Hastings wrote:

...

p.s. Supporting this patch also helps cut into PyPy's reported

On 01.06.2015 12:44, Armin Rigo wrote: performance

...

...

lead--that is, if they ever upgrade speed.pypy.org from comparing against Python *2.7.2*.

Right, we should do this upgrade when 2.7.11 is out.

There is some irony in your comment which seems to imply "PyPy is cheating by comparing with an old Python 2.7.2": it is inside a thread which started because "we didn't backport performance improvements to 2.7.x so far".

Just to convince myself, I just ran a performance comparison. I ran the same benchmark suite as speed.pypy.org, with 2.7.2 against 2.7.10, both freshly compiled with no "configure" options at all. The differences are usually in the noise, but range from +5% to... -60%. If anything, this seems to show that CPython should take more care about performance regressions. If someone is interested:

* "raytrace-simple" is 1.19 times slower * "bm_mako" is 1.29 times slower * "spitfire_cstringio" is 1.60 times slower * a number of other benchmarks are around 1.08.

The "7.0x faster" number on speed.pypy.org would be significantly *higher* if we upgraded the baseline to 2.7.10 now.

If someone were to volunteer to set up and run speed.python.org, I think we could add some additional focus on performance regressions. Right now, we don't have any way of reliably and reproducibly testing Python performance.

Hint: The PSF would most likely fund such adventures :-)

-- Marc-Andre Lemburg eGenix.com

Professional Python Services directly from the Source (#1, Jun 01 2015)

...

...

...

Python Projects, Coaching and Consulting ... http://www.egenix.com/ mxODBC Plone/Zope Database Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

---

::::: Try our mxODBC.Connect Python Database Interface for free ! ::::::

eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/tetsuya.morimoto%40gmail....

On Thu, 04 Jun 2015 12:55:55 +0200, "M.-A. Lemburg" wrote:

On 04.06.2015 04:08, Tetsuya Morimoto wrote:

...

...

If someone were to volunteer to set up and run speed.python.org, I think we could add some additional focus on performance regressions. Right now, we don't have any way of reliably and reproducibly testing Python performance.

I'm very interested in speed.python.org and feel regret that the project is standing still. I have a mind to contribute something ...

On 03.06.2015 18:59, Maciej Fijalkowski wrote:> On Wed, Jun 3, 2015 at 3:49 PM, R. David Murray

...

...

...

...

I think we should look into getting speed.python.org up and running for both Python 2 and 3 branches:

https://speed.python.org/

What would it take to make that happen ?

I guess ideal would be some cooperation from some of the cpython devs, so say someone can setup cpython buildbot

What does "set up cpython buildbot" mean in this context?

The way it works is dual - there is a program running the benchmarks (the runner) which is in the pypy case run by the pypy buildbot and the web side that reports stuff. So someone who has access to cpython buildbot would be useful.

(I don't seem to have gotten a copy of Maciej's message, at least not yet.) OK, so what you are saying is that speed.python.org will run a buildbot slave so that when a change is committed to cPython, a speed run will be triggered? Is "the runner" a normal buildbot slave, or something custom? In the normal case the master controls what the slave runs...but regardless, you'll need to let us know how the slave invocation needs to be configured on the master.

Ok, so there's interest and we have at least a few people who are willing to help.

Now we need someone to take the lead on this and form a small project group to get everything implemented. Who would be up to such a task ?

The speed project already has a mailing list, so you could use that for organizing the details.

If it's a low volume list I'm willing to sign up, but regardless I'm willing to help with the buildbot setup on the CPython side. (As soon as my credential-update request gets through infrastructure, at least :) --David

On Thu, Jun 4, 2015 at 10:51 AM, Maciej Fijalkowski wrote:

On Thu, Jun 4, 2015 at 4:32 PM, R. David Murray wrote:

...

OK, so what you are saying is that speed.python.org will run a buildbot slave so that when a change is committed to cPython, a speed run will be triggered? Is "the runner" a normal buildbot slave, or something custom? In the normal case the master controls what the slave runs...but regardless, you'll need to let us know how the slave invocation needs to be configured on the master.

Ideally nightly (benchmarks take a while). The setup for pypy looks like this:

https://bitbucket.org/pypy/buildbot/src/5fa1f1a4990f842dfbee416c4c2e2f6f75d4...

so fairly easy. This already generates a json file that you can plot. We can setup an upload automatically too.

I've been looking at what it will take to set up the buildmaster for this, and it looks like it's just a matter of checking out the benchmarks, building Python, testing it, and running the benchmarks. There is the question of which benchmark repo to use: https://bitbucket.org/pypy/benchmarks or https://hg.python.org/benchmarks; ideally, we should use hg.python.org/benchmarks for CPython benchmarks, but it looks like pypy/benchmarks has the necessary runner, so I suppose we'll be using it for now. Is there interest from both sides to merge those repositories? The big question for the buildmaster is what options to pass to the benchmark runner. I suppose most of them should match the CPythonBenchmark BuildFactory from the PyPy buildbot master configuration, but otherwise I'm not sure. The other big question is where the benchmarks will be run. The speed.python.org page makes it sound like there's a box intended for that purpose (the one running the speed.python.org page?); can anyone with access to it contact me to get the build slave set up? -- Zach

On Jun 22, 2015, at 6:58 PM, Zachary Ware wrote:

On Thu, Jun 4, 2015 at 10:51 AM, Maciej Fijalkowski wrote:

...

On Thu, Jun 4, 2015 at 4:32 PM, R. David Murray wrote:

...

OK, so what you are saying is that speed.python.org will run a buildbot slave so that when a change is committed to cPython, a speed run will be triggered? Is "the runner" a normal buildbot slave, or something custom? In the normal case the master controls what the slave runs...but regardless, you'll need to let us know how the slave invocation needs to be configured on the master.

Ideally nightly (benchmarks take a while). The setup for pypy looks like this:

https://bitbucket.org/pypy/buildbot/src/5fa1f1a4990f842dfbee416c4c2e2f6f75d4...

so fairly easy. This already generates a json file that you can plot. We can setup an upload automatically too.

I've been looking at what it will take to set up the buildmaster for this, and it looks like it's just a matter of checking out the benchmarks, building Python, testing it, and running the benchmarks. There is the question of which benchmark repo to use: https://bitbucket.org/pypy/benchmarks or https://hg.python.org/benchmarks; ideally, we should use hg.python.org/benchmarks for CPython benchmarks, but it looks like pypy/benchmarks has the necessary runner, so I suppose we'll be using it for now. Is there interest from both sides to merge those repositories?

PyPy’s is a more extensive suite but it does not fully support Python 3 yet. It makes sense to merge them. -- Philip Jenvey

For now, I'm following the mailing-lists from a spy-glass: I don't read most of the e-mails. However, this thread seems to be "infected": I can smell from here your emotions behind your words. Why to push a lot of emotions inside a technical discussion ? What's the nerves have been hit with this discussion ? If you know me a little bit, you know I'm always interested in by efficiency improvements, especially around Python. However, I see two parts of this discussion: 1. Python 3 must continue to be the first class citizen for the features, bugs-killing and performance improvements, as Barry explained. Programming in Python isn't only a language, it's also a spirit and a community with forces and weaknesses. The main "issue" for the Python 3 adoption by the community is that Python community is mainly composed by Late Majority and Laggards [1], contrary to some fancy programming language like Ruby, Go, Rust, <insert your fancy language here> where you have a majority of Early Adopters. For example, the migration from Ruby 1.8 to 1.9 has taken time because they changed some critical parts, but finally, now, almost nobody uses Ruby 1.8 on production. FYI, Ruby 1.9 has been released only one year after Python 3.0, and Ruby community has finished their migration a long time ago, where you continue to support Python 2.7. Maybe the change was less important between Ruby 1.8 and 1.9 that between Python 2 and Python 3, however I personally think the majority of Early Adopters in Ruby community has helped a lot for that. Nevertheless, at least to my eyes, it's a proof that, despite the fact time to time somebody announce that Python is dying and that nobody will use that on production for the new projects, in fact, Python is a clearly a mainstream programming language, Python 3 migration time is the best proof, you don't have that with the fancy languages. But, it also means that to accelerate Python 3 adoption, we need more incentives: Have a clean way to migrate, almost important libraries ported and the fact that Python 3 is more newcomers friendly [2] aren't enough, new features and performances are a better incentive, at least to me. Without AsyncIO, I'll continue to code for Python 2. 2. From a strategical point of view, even if it should be reduce the adoption speed of Python 3, it should be a good "move" to support that for Python 2, to reduce the risk of fork of Python: It's better for the Python community to use Python 2 than not Python at all. See the NodeJS community: even if the reasons seem to be more political than technical, fork a language isn't a potential myth. If we force too much Python 2 users to migrate to Python 3, they should reject completely the language, everybody will lose in this story. Moreover, if we start to have a critical mass of Laggards with Python 2 who have enough money/time to maintain a patch like that, and we reject that, we should lose the discussion link and mutual enrichment: everybody is concerned by performance improvements. Personally, only final results matter, I don't care about the personal motivations: economical, ecological, or basely to publish a blog post about the fact that the Python community has a bigger one that some others ;-) And don't forget: Almost nobody cares about our internal discussions and our drama, they only interested by the source code we produce, even the Python developers who use CPython. Even if we have different motivations, I'm sure that everybody on this mailing-list, or at least in this thread, "believe" in Python: You don't take personal time during a week-end if Python isn't something important to you, because during the time you take to write e-mails/source code, you don't watch series or take care of your family. [1] http://en.wikipedia.org/wiki/Early_adopter#History [2] It's in French (Google translate is your friend), however an interesting point of view of a Python trainer who has switched to Python 3: http://sametmax.com/python-3-est-fait-pour-les-nouveaux-venus/ (The website is down for now) -- Ludovic Gasc (GMLudo) http://www.gmludo.eu/ 2015-05-30 17:42 GMT+02:00 Barry Warsaw :

On May 30, 2015, at 06:55 PM, Nick Coghlan wrote:

...

Intel are looking to get involved in CPython core development *specifically* to work on performance improvements, so it's important to offer folks in the community good reasons for why we're OK with seeing at least some of that work applied to Python 2, rather than restricting their contributions to Python 3.

I think that's fine, for all the reasons you, Toshio, and others mention. For better or worse, Python 2.7 *is* our LTS release so I think we can make life easier for the folks stuck on it <wink>.

However, I want us to be very careful not to accept performance improvements in Python 2.7 that haven't also been applied to Python 3, unless of course they aren't relevant. Python 3 also has a need for performance improvements, perhaps more so for various reasons, so let's make sure we're pushing that forward too.

In many cases where you have a long lived stable release and active development releases, it's generally the policy that fixes show up in the dev release first. At least, this is the case with Ubuntu and SRUs, and it makes a lot of sense.

Cheers, -Barry _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/gmludo%40gmail.com

2015-05-31 0:26 GMT+02:00 Nick Coghlan :

On 31 May 2015 04:20, "Ludovic Gasc" wrote:

...

For now, I'm following the mailing-lists from a spy-glass: I don't read

most of the e-mails.

...

However, this thread seems to be "infected": I can smell from here your emotions behind your words.

Why to push a lot of emotions inside a technical discussion ? What's the nerves have been hit with this discussion ?

I think you answered your own question fairly well

Thanks.

- there's a longstanding, but rarely articulated, culture clash between the folks that are primarily interested in the innovators and early adopters side of things, and those of us that are most interested in bridging the gap to the early majority, late majority and laggards.

Add in the perfectly reasonable wariness a lot of folks have regarding the potential for commercial interests to unfairly exploit open source contributors without an adequate return contribution of development effort, gratis software, gratis services,

Based on my professional experience, more a client pays for your skills, more you have chance that he will respect you, because he knows your value. The contrary is, that, less a client pays, more he will try to manipulate you to do more things that it was planned in the contract. Now, for an open source software, you don't have money cost, but, you still have the knowledge cost. If you replace money by knowledge in my two previous sentences, theses sentences are also true. However, things aren't binary: Apart the contribution level [1] of each member, the "good" and "bad" ideas for the future of Python can arrive from everybody. The only thing I'm sure: I'm incompetent to predict the future, I've no idea how each member of our community will react, I can list only some possible scenarios. But with Internet, you know as me that with only few persons you can change a lot of things, look Edward Snowden for example. About Python 3 migration, I think that one of our best control stick is newcomers, and by extension, Python trainers/teachers. If newcomers learn first Python 3, when they will start to work professionally, they should help to rationalize the Python 3 migration inside existing dev teams, especially because they don't have an interest conflict based on the fact that they haven't written plenty of code with Python 2. 2020 is around the corner, 5 years shouldn't be enough to change the community mind, I don't know. [1] Don't forget that contributions aren't only the source code ;-)

or interesting employment opportunities, and you're going to see the occasional flare-ups as we find those rough edges where differences in motivation & background lead to differences of opinion & behaviour.

Cheers, Nick.

On Sat, May 30, 2015 at 8:35 PM, Antoine Pitrou wrote:

On Sat, 30 May 2015 10:34:15 +1000 Nick Coghlan wrote:

...

On 30 May 2015 09:57, "Antoine Pitrou" wrote:

...

On Sat, 30 May 2015 01:49:10 +0200 Christian Heimes wrote:

...

For performance patches we have to consider our responsibility for the environment. Every improvement means more speed and less power consumption. Python runs of hundreds of thousands of machines in the cloud. Python 2.7 will be used for at least half a decade, probably longer. Servers can be replaced with faster machines later and less fossil fuel must be burned to produce power.

Please keep your ideology out of this.

I'm a qualified engineer (in computer systems engineering), so caring about environmental sustainability is part of my professional ethical standards, not just a matter of personal preference: http://www.wfeo.org/ethics/

There is no reason to assume that a smallish performance improvement in a single Python 2.7 release will make any difference in "environmental sustainability" of the world's computing infrastructure, while the problem is measured in orders of magnitude. The onus is on to you to prove the contrary. Otherwise, bringing it up is mere ideology.

The magnitude of the environmental benefit of Python performance improvement is uncertain, but we know what direction it's going to be. If there's going to be a massive maintenance nightmare, or if the change comes at a cost of functionality or debuggability, then sure, the onus is on the person begging for performance improvements; but if there's no such cost (or if the cost is being carried by the same person/people who proposed the change), then surely it's worth something? Suppose someone came up with a magic patch that makes the CPython core run 25% faster. No downsides, just 25% faster across the board. I wouldn't pay money for it on the sole basis of expecting to make that back in reduced electricity bills, but I certainly wouldn't be sorry to watch the load averages drop. Why is this controversial? ChrisA

On Sat, May 30, 2015 at 9:00 PM, Antoine Pitrou wrote:

On Sat, 30 May 2015 20:52:21 +1000 Chris Angelico wrote:

...

Suppose someone came up with a magic patch that makes the CPython core run 25% faster. No downsides, just 25% faster across the board. I wouldn't pay money for it on the sole basis of expecting to make that back in reduced electricity bills, but I certainly wouldn't be sorry to watch the load averages drop. Why is this controversial?

That was not my point. What I'm opposing is the idea that "environmental sustainability" (or what people's ideological conception of it is) should become part of our criteria when making maintenance decisions.

Obviously if a patch makes CPython faster without any downsides, there is no need to argue about environmental sustainability to make the patch desirable. The performance improvement itself is a sufficient reason.

Okay. But what objection do you have to reduced electricity usage? I'm still not understanding how this is a problem. It might not be a priority for everyone, but surely it's a nice bonus? ChrisA

Antoine Pitrou writes:

On Sat, 30 May 2015 01:49:10 +0200 Christian Heimes wrote:

...

For performance patches we have to consider our responsibility for the environment. Every improvement means more speed and less power consumption. Python runs of hundreds of thousands of machines in the cloud. Python 2.7 will be used for at least half a decade, probably longer. Servers can be replaced with faster machines later and less fossil fuel must be burned to produce power.

Please keep your ideology out of this.

Bad idea, unless you have benchmarks and engineering studies proving that that effect doesn't exist and never will. In a community of volunteers, ideology is typically a great motivator. If it weren't for ideology (specifically, RMS's), many of us wouldn't be here, and quite likely nothing like the current Linux and BSD ecosystems would be available yet, and maybe not at all. Which points to a better idea: Harness ideology to encourage contributions that help us all. Hey, Christian, maybe you know some sustainability advocates who'd like to help fund that work? Or do the programming?

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 2015-05-30 14:03, Antoine Pitrou wrote:

No, it's up to the proponent to prove that the effect exists, with a magnitude large enough to make any interesting difference. That's part of the process when suggesting a change.

If it doesn't, or if it's entirely cosmetical, it may be an important part of Christian's lifestyle (as are many individual practices, including religious, militant, dietetic...), but it certainly shouldn't brought up here. We don't want everyone trying to inject their beliefs in the maintenance process.

Antoine, now your are putting it over the top. You make it sound like I'm some crazy environmentalist or eco-warrior. Well, I'm not. I merely keep the environment in mind. Yes, I have a modern, power saving washing machine and LED lights at home. Mostly because they save money in the long run (Germany's electricity prices are high). That was also the point behind my comment. Increased performance result in better efficiency which lead to better utilization of hardware and less power consumption. Companies are interested in better efficiency, because they have to pay less for hardware, power and cooling. The obvious benefits for our environment are a side effect. A smaller CO2 foot print is not my main concern. But I wanted to bring it up anyway. For some it is an small additional motivator for performance improvements. For others it could be a marketing instrument. In Germany ads are full of crazy 'green' slogans. Christian -----BEGIN PGP SIGNATURE----- iQEcBAEBCgAGBQJVabPTAAoJEIZoUkkhLbaJTF0H+wb8ciikP762qc8u586H2AjV 2xV9AAamI1Z6RwlvKRM7YHVk48coYIKk9WQ6DZODNlVSIhnijexII1dai91gbQvy jEVkLK2P6/C7I4gz7Fp0/SoCwkpGCev2CiSJUhIoE4oIw+Mm4BRASpf5hn4n+pRI yqXixYf7h+QWHgN0FRU3GU8RxNYRe65zB/3YeDUhKLQdkf8Gq4NVX7rlTx1gvZrq DbaGjKtkT8uec6hnvZcXwWVODYW10VHTonhlV3ff0sReXw94sXOeQwQ3n+7uwKAb sqvy11k0r6JejNGFxJqfMyXH557LP5ucc2g9+J8M2Sw4SOs7L6E+caaX89FY754= =soyL -----END PGP SIGNATURE-----

Antoine Pitrou writes:

...

In a community of volunteers, ideology is typically a great motivator.

If and only everyone agrees on it.

That, my friend, is *your* ideology speaking. Some people work on open source to scratch technical itches -- the program doesn't do what they want, they're able to improve it, the license allows them to improve it, so they do, done. Others use that same freedom to change the software to improve the world in other ways. We don't need to agree on *why* we do the work we do. We only need to agree on an evaluation and arbitration process for determining *which* work gets released as part of "original Python". More on that below.

Otherwise, it is typically a great divisor.

Only because some people make a point of insisting on implementing theirs[1] -- and others insist on objecting to any mention of it. I think both extremes are divisive -- but nothing new there, extremes usually are divisive. Now, Christian did say "must" when he suggested considering the environment, and that's obviously not right. To the extent that folks are volunteers and not bound by the professional ethics that Nick professes, there's no *must* about it. I don't think Christian really meant to try to impose that on everybody in the project, though. It was more a wish on his part as I understand it, one he knows will at best be fulfilled gradually and voluntarily as people come to be aware of the issue and agree with him that some things need to be done to address it. But if people like Christian choose to work on patches because they are "environmentally friendly", or vote +1 on them, even if that means a clarification or even reinterpretation of maintenance policy, why should we care whether they say what their motivation is? On the other hand, if it *is* a change in maintenance policy to commit the Intel patch, IMO you have right on your side to speak up about that (as you do elsewhere). (OTOH, it seems to me that most posters in this thread so far agree that it's a mere clarification of *policy*, although it's a clear reallocation of *effort* that probably wouldn't come voluntarily from the core committers.) You're also right to point out that the nature of the community will change as people paid to work on commercially desirable tasks become committers. Definitely the natures of Linux and GUI framework development changed (as indeed X11 did when it passed from a commercial consortium to a more open organization) as commercial interests started supplying more and more effort, as well as hiring core developers. Whether that prospective change is a good thing for Python is a matter for debate, and (speaking only for myself, and this may not be the appropriate channel anyway) I'm interested in hearing your discussion on that matter.

Even abidance to RMS' writings and actions would probably not be unanimous here...

I assure there's absolutely no "probably" about it. You evidently missed the (intended though obscure) irony of *me* praising RMS's ideology (see return address). Footnotes: [1] You could argue that "insisting on implementing" is implied by "ideology", but then I expect that Christian would deny a desire to *impose* his values on the project.

Hi Matthias and Gregory, The results shown were run on Python 2.7.10 built using gcc. The goal of our team is to make long-term open source contributions with emphasis on performance optimization and support for the larger community and hence icc wasn't used. We've experimented with gcc profile-guided optimization (PGO) and LTO a month ago. PGO being an independent/orthogonal optimization, it shows improvement for both the stock version (i.e. current switch based dispatch) and the computed-goto version. We ran PGO optimized Python on the workloads available at language benchmarks game (http://benchmarksgame.alioth.debian.org/u64/python.php) and found that PGO benefits computed-goto version more than the stock version. I haven't run PGO optimized Python with the "grand unified python benchmarks" (GUPB) suite ...please give me a day or two and will get back to you with PGO (and LTO) numbers as well. (LTO hasn't shown much benefit so far on the language benchmarks game workloads). Also, in our analysis using CPU performance counters, we found that python workloads (in general) have higher CPU front-end issues (mainly I-cache misses) and PGO is very helpful in mitigating those issues. We're also investigating and working on ways to further reduce those front-end issues and speedup Python workloads. Thanks, Vamsi -----Original Message----- From: Matthias Klose [mailto:doko@ubuntu.com] Sent: Thursday, May 28, 2015 5:01 AM To: Parasa, Srinivas Vamsi; 'python-dev@python.org' Subject: Re: [Python-Dev] Computed Goto dispatch for Python 2 On 05/28/2015 02:17 AM, Parasa, Srinivas Vamsi wrote:

Hi All,

This is Vamsi from Server Scripting Languages Optimization team at Intel Corporation.

Would like to submit a request to enable the computed goto based dispatch in Python 2.x (which happens to be enabled by default in Python 3 given its performance benefits on a wide range of workloads). We talked about this patch with Guido and he encouraged us to submit a request on Python-dev (email conversation with Guido shown at the bottom of this email).

Attached is the computed goto patch (along with instructions to run) for Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin at http://bugs.python.org/issue4753). We built and tested this patch for Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - Haswell EP CPU with 18 cores, hyper-threading off, turbo off).

Below is a summary of the performance we saw on the "grand unified python benchmarks" suite (available at https://hg.python.org/benchmarks/). We made 3 rigorous runs of the following benchmarks. In each rigorous run, a benchmark is run 100 times with and without the computed goto patch. Below we show the average performance boost for the 3 rigorous runs.

Python 2.7.10 (original) vs Computed Goto performance Benchmark

-1 As Gregory pointed out, there are other options to build the interpreter, and we are missing data how these compare with your patch. I assume, you tested with the Intel compiler, so it would be good to see results for other compilers as well (GCC, clang). Please could you provide the data for LTO and profile guided optimized builds (maybe combined too)? I'm happy to work with you on setting up these builds, but currently don't have the machine resources to do so myself. If the benefits show up for these configurations too, then I'm +/-0 on this patch. Matthias

On 28 May 2015 at 23:37, Nick Coghlan wrote:

On 28 May 2015 at 22:00, Matthias Klose wrote:

...

On 05/28/2015 02:17 AM, Parasa, Srinivas Vamsi wrote:

...

Hi All,

This is Vamsi from Server Scripting Languages Optimization team at Intel Corporation.

Would like to submit a request to enable the computed goto based dispatch in Python 2.x (which happens to be enabled by default in Python 3 given its performance benefits on a wide range of workloads). We talked about this patch with Guido and he encouraged us to submit a request on Python-dev (email conversation with Guido shown at the bottom of this email).

Attached is the computed goto patch (along with instructions to run) for Python 2.7.10 (based on the patch submitted by Jeffrey Yasskin at http://bugs.python.org/issue4753). We built and tested this patch for Python 2.7.10 on a Linux machine (Ubuntu 14.04 LTS server, Intel Xeon - Haswell EP CPU with 18 cores, hyper-threading off, turbo off).

Below is a summary of the performance we saw on the "grand unified python benchmarks" suite (available at https://hg.python.org/benchmarks/). We made 3 rigorous runs of the following benchmarks. In each rigorous run, a benchmark is run 100 times with and without the computed goto patch. Below we show the average performance boost for the 3 rigorous runs.

Python 2.7.10 (original) vs Computed Goto performance Benchmark

-1

As Gregory pointed out, there are other options to build the interpreter, and we are missing data how these compare with your patch.

That's shifting the goal posts: suggesting that we should optimise Python 2 differently from the way we optimised Python 3 isn't a reasonable request to make in the context of a backporting proposal.

Sorry, I misread your email. I thought you were talking about the part of Greg's email where he suggested different optimisation techniques, but you were actually referring to the part where he requested more detail on the compiler used and the number for gcc and clang. *That* request I agree with. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia