Hi,
On Twitter, Raymond Hettinger wrote:
"The decision making process on Python-dev is an anti-pattern,
governed by anecdotal data and ambiguity over what problem is solved."
https://twitter.com/raymondh/status/887069454693158912
About "anecdotal data", I would like to discuss the Python startup time.
== Python 3.7 compared to 2.7 ==
First of all, on speed.python.org, we have:
* Python 2.7: 6.4 ms with site, 3.0 ms without site (-S)
* master (3.7): 14.5 ms with site, 8.4 ms without site (-S)
Python 3.7 startup time is 2.3x slower with site (default mode), or
2.8x slower without site (-S command line option).
(I will skip Python 3.4, 3.5 and 3.6 which are much worse than Python 3.7...)
So if an user complained about Python 2.7 startup time: be prepared
for a 2x - 3x more angry user when "forced" to upgrade to Python 3!
== Mercurial vs Git, Python vs C, startup time ==
Startup time matters a lot for Mercurial since Mercurial is compared
to Git. Git and Mercurial have similar features, but Git is written in
C whereas Mercurial is written in Python. Quick benchmark on the
speed.python.org server:
* hg version: 44.6 ms +- 0.2 ms
* git --version: 974 us +- 7 us
Mercurial startup time is already 45.8x slower than Git whereas tested
Mercurial runs on Python 2.7.12. Now try to sell Python 3 to Mercurial
developers, with a startup time 2x - 3x slower...
I tested Mecurial 3.7.3 and Git 2.7.4 on Ubuntu 16.04.1 using "python3
-m perf command -- ...".
== CPython core developers don't care? no, they do care ==
Christian Heimes, Naoki INADA, Serhiy Storchaka, Yury Selivanov, me
(Victor Stinner) and other core developers made multiple changes last
years to reduce the number of imports at startup, optimize impotlib,
etc.
IHMO all these core developers are well aware of the competition of
programming languages, and honesty Python startup time isn't "good".
So let's compare it to other programming languages similar to Python.
== PHP, Ruby, Perl ==
I measured the startup time of other programming languages which are
similar to Python, still on the speed.python.org server using "python3
-m perf command -- ...":
* perl -e ' ': 1.18 ms +- 0.01 ms
* php -r ' ': 8.57 ms +- 0.05 ms
* ruby -e ' ': 32.8 ms +- 0.1 ms
Wow, Perl is quite good! PHP seems as good as Python 2 (but Python 3
is worse). Ruby startup time seems less optimized than other
languages.
Tested versions:
* perl 5, version 22, subversion 1 (v5.22.1)
* PHP 7.0.18-0ubuntu0.16.04.1 (cli) ( NTS )
* ruby 2.3.1p112 (2016-04-26) [x86_64-linux-gnu]
== Quick Google search ==
I also searched for "python startup time" and "python slow startup
time" on Google and found many articles. Some examples:
"Reducing the Python startup time"
http://www.draketo.de/book/export/html/498
=> "The python startup time always nagged me (17-30ms) and I just
searched again for a way to reduce it, when I found this: The
Python-Launcher caches GTK imports and forks new processes to reduce
the startup time of python GUI programs."
https://nelsonslog.wordpress.com/2013/04/08/python-startup-time/
=> "Wow, Python startup time is worse than I thought."
"How to speed up python starting up and/or reduce file search while
loading libraries?"
https://stackoverflow.com/questions/15474160/how-to-speed-up-python-start...
=> "The first time I log to the system and start one command it takes
6 seconds just to show a few line of help. If I immediately issue the
same command again it takes 0.1s. After a couple of minutes it gets
back to 6s. (proof of short-lived cache)"
"How does one optimise the startup of a Python script/program?"
https://www.quora.com/How-does-one-optimise-the-startup-of-a-Python-scrip...
=> "I wrote a Python program that would be used very often (imagine
'cd' or 'ls') for very short runtimes, how would I make it start up as
fast as possible?"
"Python Interpreter Startup time"
https://bytes.com/topic/python/answers/34469-pyhton-interpreter-startup-time
"Python is very slow to start on Windows 7"
https://stackoverflow.com/questions/29997274/python-is-very-slow-to-start...
=> "Python takes 17 times longer to load on my Windows 7 machine than
Ubuntu 14.04 running on a VM"
=> "returns in 0.614s on Windows and 0.036s on Linux"
"How to make a fast command line tool in Python" (old article Python 2.5.2)
https://files.bemusement.org/talks/OSDC2008-FastPython/
=> "(...) some techniques Bazaar uses to start quickly, such as lazy imports."
--
So please continue efforts for make Python startup even faster to beat
all other programming languages, and finally convince Mercurial to
upgrade ;-)
Victor
Python 3.3 is fast approaching its end-of-life date, 2017-09-29. Per our release policy, that date is five years after the initial release of 3.3, 3.3.0 final on 2012-09-29. Note that 3.3 has been in security-fix only mode since the 2014-03-08 release of 3.3.5. It has been a while since we produced a 3.3.x security-fix release and, due to his commitments elsewhere, Georg has agreed for me to lead 3.3 to its well-deserved retirement.
To that end, I would like to schedule its next, and hopefully final, security-fix release to coincide with the already announced 3.4.7 security-fix release. In particular, we'll plan to tag and release 3.3.7rc1 on Monday 2017-07-24 (UTC) and tag and release 3.3.7 final on Monday 2017-08-07. In the coming days, I'll be reviewing the outstanding 3.3 security issues and merging appropriate 3.3 PRs. Some of them have been sitting as patches for a long time so, if you have any such security issues that you think belong in 3.3, it would be very helpful if you would review such patches and turn them into 3.3 PRs.
As a reminder, here are the guidelines from the devguide as to what is appropriate for a security-fix only branch:
"The only changes made to a security branch are those fixing issues exploitable by attackers such as crashes, privilege escalation and, optionally, other issues such as denial of service attacks. Any other changes are not considered a security risk and thus not backported to a security branch. You should also consider fixing hard-failing tests in open security branches since it is important to be able to run the tests successfully before releasing."
Note that documentation changes, other than any that might be related to a security fix, are also out of scope.
Assuming no new security issues arise prior to the EOL date, 3.3.7 will likely be the final release of 3.3. And you really shouldn't be using 3.3 at all at this point; while downstream distributors are, of course, free to provide support of 3.3 to their customers, in a little over two months when EOL is reached python-dev will no longer accept any issues or make any changes available for 3.3. If you are still using 3.3, you really owe it to your applications, to your users, and to yourself to upgrade to a more recent release of Python 3, preferably 3.6! Many, many fixes, new features, and substantial performance improvements await you.
https://www.python.org/dev/peps/pep-0398/https://devguide.python.org/devcycle/#security-branches
--
Ned Deily
nad(a)python.org -- []
Hello all,
Over in Ubuntu, we've gotten reports about some performance regressions in
Python 2.7 when moving from Trusty (14.04 LTS) to Xenial (16.04 LTS).
Trusty's version is based on 2.7.6 while Xenial's version is based on 2.7.12
with bits of .13 cherry picked.
We've not been able to identify any change in Python itself (or the
Debian/Ubuntu deltas) which could account for this, so the investigation has
led to various gcc compiler options and version differences. In particular
disabling LTO (link-time optimization) seems to have a positive impact, but
doesn't completely regain the loss.
Louis (Cc'd here) has done a ton of work to measure and analyze the problem,
but we've more or less hit a roadblock, so we're taking the issue public to
see if anybody on this mailing list has further ideas. A detailed analysis is
available in this Google doc:
https://docs.google.com/document/d/1zrV3OIRSo99fd2Ty4YdGk_scmTRDmVauBprKL...
The document should be public for comments and editing.
If you have any thoughts, or other lines of investigation you think are
worthwhile pursuing, please add your comments to the document.
Cheers,
-Barry
Hello,
I think that the expression "for...else" or "while...else" is completely
counter-intuitive. Wouldn't it be possible to make it clearer? Maybe
something like
break in for i in range(n):
...
if cond:
break
else:
...
I'm not an English native speaker so I don't know whether "break in" is
acceptable English in this context or can only mean "to get into a
building by force".
Kiuhnm
hello,
we're seeing strange problems when trying to do reproducible builds of some python 3.6 modules.
Namely, from one build to another, there will be something like the following difference in the
compiled object:
00004e40 da 07 5f 5f 61 6c 6c 5f 5f da 0a 5f 5f 61 75 74 |..__all__..__aut|
-00004e50 68 6f 72 5f 5f da 07 64 65 63 69 6d 61 6c 72 0c |hor__..decimalr.|
+00004e50 68 6f 72 5f 5f 5a 07 64 65 63 69 6d 61 6c 72 0c |hor__Z.decimalr.|
00004e60 00 00 00 72 43 00 00 00 72 08 00 00 00 72 41 00 |...rC...r....rA.|
This specific one is in the top-level co_names segment and the 0x5a vs 0xda byte is
TYPE_SHORT_ASCII_INTERNED, with FLAG_REF set or unset.
I'm also seeing off-by-one differences in reference ids, i.e., the number appearing after TYPE_REF.
Not in all cases, but it seems that when a "part" is affected, all references in that "part" are
changed (for some value of "part"; all the knowledge of pycs I have was gained from about an hour of
reading marshal.c). So that seems to imply that there's a reference that is sometimes included and
sometimes not?
This is most often found in __init__.py. Often this affects optimized pycs, but we can see it in
un-optimized as well.
The issue is rare -- 99% of all pycs are stable -- but when it occurs, it's easy to replicate it in
the same place. This also happens on different machines, so that seems to rule out hardware memory
errors :)
The pycs in question are generated by normal "setup.py build" -> "setup.py install". It happens on
Python 3.6 but not on Python 2.7. I'm not sure about Python 3.5 because we don't currently use it.
It doesn't seem to depend on hash seed - the instability is observed even with PYTHONHASHSEED set to
zero. What seems to fix it, however, is running the build on disorderfs, which ensures that the
filesystem entries are in the same order.
Any ideas why something like this would happen and why would it be correlated with filesystem ordering?
thanks
m.
Thanks to Kushal Das we now have one of the most requested features since
the transition: a link in PRs back to bugs.python.org (in a more
discoverable way since we have had them since Bedevere launched :) . When a
pull request comes in with an issue number in the title (or one gets
added), a link to bugs.python.org will be appended to the PR's body (the
message you fill out when creating a PR). There's no logic to remove the
link if the issue number is removed from the title, changed, or for
multiple issue numbers since basically those cases are all rare and it was
easier to launch without that kind of support.
P.S.: Berker Peksag is working on providing commit emails with diffs in
them which is the other most requested feature since the transition.
https://www.python.org/dev/peps/pep-0007/ says two things:
> Python versions greater than or equal to 3.6 use C89 with several select C99 features:
> [...]
> C++-style line comments
and also:
> Never use C++ style // one-line comments.
Which is it?
-- Devin
On behalf of the Python development community and the Python 3.4 and
Python 3.5 release teams, I'm relieved to announce the availability of
Python 3.4.7rc1 and Python 3.5.4rc1.
Python 3.4 is now in "security fixes only" mode. This is the final
stage of support for Python 3.4. Python 3.4 now only receives security
fixes, not bug fixes, and Python 3.4 releases are source code only--no
more official binary installers will be produced.
Python 3.5.4 will be the final 3.5 release in "bug fix" mode. After
3.5.4 is released, Python 3.5 will also move into "security fixes mode".
Both these releases are "release candidates". They should not be
considered the final releases, although the final releases should
contain only minor differences. Python users are encouraged to test
with these releases and report any problems they encounter.
You can find Python 3.4.7rc1 here:
https://www.python.org/downloads/release/python-347rc1/
And you can find Python 3.5.4rc1 here:
https://www.python.org/downloads/release/python-354rc1/
Python 3.4.7 final and Python 3.5.4 final are both scheduled for release
on August 6th, 2017.
Happy Pythoning,
//arry/
Hi folks,
(Not entirely sure this is the right place for this question, but hopefully
it's of interest to several folks.)
A few days ago I posted a note in response to Victor Stinner's articles on
his CPython contributions, noting that I wrote a program that ran in 11.7
seconds on Python 2.7, but only takes 5.1 seconds on Python 3.5 (on my 2.5
GHz macOS i7), more than 2x as fast. Obviously this is a Good Thing, but
I'm curious as to why there's so much difference.
The program is a pentomino puzzle solver, and it works via code generation,
generating a ton of nested "if" statements, so I believe it's exercising
the Python bytecode interpreter heavily. Obviously there have been some big
optimizations to make this happen, but I'm curious what the main
improvements are that are causing this much difference.
There's a writeup about my program here, with benchmarks at the bottom:
http://benhoyt.com/writings/python-pentomino/
This is the generated Python code that's being exercised:
https://github.com/benhoyt/python-pentomino/blob/master/generated_solve.py
For reference, on Python 3.6 it runs in 4.6 seconds (same on Python 3.7
alpha). This smallish increase from Python 3.5 to Python 3.6 was more
expected to me due to the bytecode changing to wordcode in 3.6.
I tried using cProfile on both Python versions, but that didn't say much,
because the functions being called aren't taking the majority of the time.
How does one benchmark at a lower level, or otherwise explain what's going
on here?
Thanks,
Ben
