Hi,
On Twitter, Raymond Hettinger wrote:
"The decision making process on Python-dev is an anti-pattern,
governed by anecdotal data and ambiguity over what problem is solved."
https://twitter.com/raymondh/status/887069454693158912
About "anecdotal data", I would like to discuss the Python startup time.
== Python 3.7 compared to 2.7 ==
First of all, on speed.python.org, we have:
* Python 2.7: 6.4 ms with site, 3.0 ms without site (-S)
* master (3.7): 14.5 ms with site, 8.4 ms without site (-S)
Python 3.7 startup time is 2.3x slower with site (default mode), or
2.8x slower without site (-S command line option).
(I will skip Python 3.4, 3.5 and 3.6 which are much worse than Python 3.7...)
So if an user complained about Python 2.7 startup time: be prepared
for a 2x - 3x more angry user when "forced" to upgrade to Python 3!
== Mercurial vs Git, Python vs C, startup time ==
Startup time matters a lot for Mercurial since Mercurial is compared
to Git. Git and Mercurial have similar features, but Git is written in
C whereas Mercurial is written in Python. Quick benchmark on the
speed.python.org server:
* hg version: 44.6 ms +- 0.2 ms
* git --version: 974 us +- 7 us
Mercurial startup time is already 45.8x slower than Git whereas tested
Mercurial runs on Python 2.7.12. Now try to sell Python 3 to Mercurial
developers, with a startup time 2x - 3x slower...
I tested Mecurial 3.7.3 and Git 2.7.4 on Ubuntu 16.04.1 using "python3
-m perf command -- ...".
== CPython core developers don't care? no, they do care ==
Christian Heimes, Naoki INADA, Serhiy Storchaka, Yury Selivanov, me
(Victor Stinner) and other core developers made multiple changes last
years to reduce the number of imports at startup, optimize impotlib,
etc.
IHMO all these core developers are well aware of the competition of
programming languages, and honesty Python startup time isn't "good".
So let's compare it to other programming languages similar to Python.
== PHP, Ruby, Perl ==
I measured the startup time of other programming languages which are
similar to Python, still on the speed.python.org server using "python3
-m perf command -- ...":
* perl -e ' ': 1.18 ms +- 0.01 ms
* php -r ' ': 8.57 ms +- 0.05 ms
* ruby -e ' ': 32.8 ms +- 0.1 ms
Wow, Perl is quite good! PHP seems as good as Python 2 (but Python 3
is worse). Ruby startup time seems less optimized than other
languages.
Tested versions:
* perl 5, version 22, subversion 1 (v5.22.1)
* PHP 7.0.18-0ubuntu0.16.04.1 (cli) ( NTS )
* ruby 2.3.1p112 (2016-04-26) [x86_64-linux-gnu]
== Quick Google search ==
I also searched for "python startup time" and "python slow startup
time" on Google and found many articles. Some examples:
"Reducing the Python startup time"
http://www.draketo.de/book/export/html/498
=> "The python startup time always nagged me (17-30ms) and I just
searched again for a way to reduce it, when I found this: The
Python-Launcher caches GTK imports and forks new processes to reduce
the startup time of python GUI programs."
https://nelsonslog.wordpress.com/2013/04/08/python-startup-time/
=> "Wow, Python startup time is worse than I thought."
"How to speed up python starting up and/or reduce file search while
loading libraries?"
https://stackoverflow.com/questions/15474160/how-to-speed-up-python-start...
=> "The first time I log to the system and start one command it takes
6 seconds just to show a few line of help. If I immediately issue the
same command again it takes 0.1s. After a couple of minutes it gets
back to 6s. (proof of short-lived cache)"
"How does one optimise the startup of a Python script/program?"
https://www.quora.com/How-does-one-optimise-the-startup-of-a-Python-scrip...
=> "I wrote a Python program that would be used very often (imagine
'cd' or 'ls') for very short runtimes, how would I make it start up as
fast as possible?"
"Python Interpreter Startup time"
https://bytes.com/topic/python/answers/34469-pyhton-interpreter-startup-time
"Python is very slow to start on Windows 7"
https://stackoverflow.com/questions/29997274/python-is-very-slow-to-start...
=> "Python takes 17 times longer to load on my Windows 7 machine than
Ubuntu 14.04 running on a VM"
=> "returns in 0.614s on Windows and 0.036s on Linux"
"How to make a fast command line tool in Python" (old article Python 2.5.2)
https://files.bemusement.org/talks/OSDC2008-FastPython/
=> "(...) some techniques Bazaar uses to start quickly, such as lazy imports."
--
So please continue efforts for make Python startup even faster to beat
all other programming languages, and finally convince Mercurial to
upgrade ;-)
Victor
Hi,
Could anyone put this five year-old bug about parsing iso8601 format date-times
on the front burner?
http://bugs.python.org/issue15873
In the comments there's a lot of hand-wringing about different variations that
bogged it down, but right now I only need it to handle the output of
datetime.isoformat():
>>> dt.isoformat()
'2017-10-20T08:20:08.986166+00:00'
Perhaps if we could get that minimum first step in, it could be iterated on and
made more lenient in the future.
Thank you,
-Mike
I couldn’t resist one more PEP from the Core sprint. I won’t reveal where or how this one came to me.
-Barry
PEP: 559
Title: Built-in noop()
Author: Barry Warsaw <barry(a)python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 2017-09-08
Python-Version: 3.7
Post-History: 2017-09-09
Abstract
========
This PEP proposes adding a new built-in function called ``noop()`` which does
nothing but return ``None``.
Rationale
=========
It is trivial to implement a no-op function in Python. It's so easy in fact
that many people do it many times over and over again. It would be useful in
many cases to have a common built-in function that does nothing.
One use case would be for PEP 553, where you could set the breakpoint
environment variable to the following in order to effectively disable it::
$ setenv PYTHONBREAKPOINT=noop
Implementation
==============
The Python equivalent of the ``noop()`` function is exactly::
def noop(*args, **kws):
return None
The C built-in implementation is available as a pull request.
Rejected alternatives
=====================
``noop()`` returns something
----------------------------
YAGNI.
This is rejected because it complicates the semantics. For example, if you
always return both ``*args`` and ``**kws``, what do you return when none of
those are given? Returning a tuple of ``((), {})`` is kind of ugly, but
provides consistency. But you might also want to just return ``None`` since
that's also conceptually what the function was passed.
Or, what if you pass in exactly one positional argument, e.g. ``noop(7)``. Do
you return ``7`` or ``((7,), {})``? And so on.
The author claims that you won't ever need the return value of ``noop()`` so
it will always return ``None``.
Coghlin's Dialogs (edited for formatting):
My counterargument to this would be ``map(noop, iterable)``,
``sorted(iterable, key=noop)``, etc. (``filter``, ``max``, and
``min`` all accept callables that accept a single argument, as do
many of the itertools operations).
Making ``noop()`` a useful default function in those cases just
needs the definition to be::
def noop(*args, **kwds):
return args[0] if args else None
The counterargument to the counterargument is that using ``None``
as the default in all these cases is going to be faster, since it
lets the algorithm skip the callback entirely, rather than calling
it and having it do nothing useful.
Copyright
=========
This document has been placed in the public domain.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
Hi,
I rejected my own PEP 511 "API for code transformers" that I wrote in
January 2016:
https://github.com/python/peps/commit/9d8fd950014a80324791d7dae3c130b1b64...
Rejection Notice:
"""
This PEP was rejected by its author.
This PEP was seen as blessing new Python-like programming languages
which are close but incompatible with the regular Python language. It
was decided to not promote syntaxes incompatible with Python.
This PEP was also seen as a nice tool to experiment new Python features,
but it is already possible to experiment them without the PEP, only with
importlib hooks. If a feature becomes useful, it should be directly part
of Python, instead of depending on an third party Python module.
Finally, this PEP was driven was the FAT Python optimization project
which was abandonned in 2016, since it was not possible to show any
significant speedup, but also because of the lack of time to implement
the most advanced and complex optimizations.
"""
Victor
Hi,
While discussions on this PEP are not over on python-ideas, I proposed
this PEP directly on python-dev since I consider that my PEP already
summarizes current and past proposed alternatives.
python-ideas threads:
* Add time.time_ns(): system clock with nanosecond resolution
* Why not picoseconds?
The PEP 564 will be shortly online at:
https://www.python.org/dev/peps/pep-0564/
Victor
PEP: 564
Title: Add new time functions with nanosecond resolution
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner(a)gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 16-October-2017
Python-Version: 3.7
Abstract
========
Add five new functions to the ``time`` module: ``time_ns()``,
``perf_counter_ns()``, ``monotonic_ns()``, ``clock_gettime_ns()`` and
``clock_settime_ns()``. They are similar to the function without the
``_ns`` suffix, but have nanosecond resolution: use a number of
nanoseconds as a Python int.
The best ``time.time_ns()`` resolution measured in Python is 3 times
better then ``time.time()`` resolution on Linux and Windows.
Rationale
=========
Float type limited to 104 days
------------------------------
The clocks resolution of desktop and latop computers is getting closer
to nanosecond resolution. More and more clocks have a frequency in MHz,
up to GHz for the CPU TSC clock.
The Python ``time.time()`` function returns the current time as a
floatting point number which is usually a 64-bit binary floatting number
(in the IEEE 754 format).
The problem is that the float type starts to lose nanoseconds after 104
days. Conversion from nanoseconds (``int``) to seconds (``float``) and
then back to nanoseconds (``int``) to check if conversions lose
precision::
# no precision loss
>>> x = 2 ** 52 + 1; int(float(x * 1e-9) * 1e9) - x
0
# precision loss! (1 nanosecond)
>>> x = 2 ** 53 + 1; int(float(x * 1e-9) * 1e9) - x
-1
>>> print(datetime.timedelta(seconds=2 ** 53 / 1e9))
104 days, 5:59:59.254741
``time.time()`` returns seconds elapsed since the UNIX epoch: January
1st, 1970. This function loses precision since May 1970 (47 years ago)::
>>> import datetime
>>> unix_epoch = datetime.datetime(1970, 1, 1)
>>> print(unix_epoch + datetime.timedelta(seconds=2**53 / 1e9))
1970-04-15 05:59:59.254741
Previous rejected PEP
---------------------
Five years ago, the PEP 410 proposed a large and complex change in all
Python functions returning time to support nanosecond resolution using
the ``decimal.Decimal`` type.
The PEP was rejected for different reasons:
* The idea of adding a new optional parameter to change the result type
was rejected. It's an uncommon (and bad?) programming practice in
Python.
* It was not clear if hardware clocks really had a resolution of 1
nanosecond, especially at the Python level.
* The ``decimal.Decimal`` type is uncommon in Python and so requires
to adapt code to handle it.
CPython enhancements of the last 5 years
----------------------------------------
Since the PEP 410 was rejected:
* The ``os.stat_result`` structure got 3 new fields for timestamps as
nanoseconds (Python ``int``): ``st_atime_ns``, ``st_ctime_ns``
and ``st_mtime_ns``.
* The PEP 418 was accepted, Python 3.3 got 3 new clocks:
``time.monotonic()``, ``time.perf_counter()`` and
``time.process_time()``.
* The CPython private "pytime" C API handling time now uses a new
``_PyTime_t`` type: simple 64-bit signed integer (C ``int64_t``).
The ``_PyTime_t`` unit is an implementation detail and not part of the
API. The unit is currently ``1 nanosecond``.
Existing Python APIs using nanoseconds as int
---------------------------------------------
The ``os.stat_result`` structure has 3 fields for timestamps as
nanoseconds (``int``): ``st_atime_ns``, ``st_ctime_ns`` and
``st_mtime_ns``.
The ``ns`` parameter of the ``os.utime()`` function accepts a
``(atime_ns: int, mtime_ns: int)`` tuple: nanoseconds.
Changes
=======
New functions
-------------
This PEP adds five new functions to the ``time`` module:
* ``time.clock_gettime_ns(clock_id)``
* ``time.clock_settime_ns(clock_id, time: int)``
* ``time.perf_counter_ns()``
* ``time.monotonic_ns()``
* ``time.time_ns()``
These functions are similar to the version without the ``_ns`` suffix,
but use nanoseconds as Python ``int``.
For example, ``time.monotonic_ns() == int(time.monotonic() * 1e9)`` if
``monotonic()`` value is small enough to not lose precision.
Unchanged functions
-------------------
This PEP only proposed to add new functions getting or setting clocks
with nanosecond resolution. Clocks are likely to lose precision,
especially when their reference is the UNIX epoch.
Python has other functions handling time (get time, timeout, etc.), but
no nanosecond variant is proposed for them since they are less likely to
lose precision.
Example of unchanged functions:
* ``os`` module: ``sched_rr_get_interval()``, ``times()``, ``wait3()``
and ``wait4()``
* ``resource`` module: ``ru_utime`` and ``ru_stime`` fields of
``getrusage()``
* ``signal`` module: ``getitimer()``, ``setitimer()``
* ``time`` module: ``clock_getres()``
Since the ``time.clock()`` function was deprecated in Python 3.3, no
``time.clock_ns()`` is added.
Alternatives and discussion
===========================
Sub-nanosecond resolution
-------------------------
``time.time_ns()`` API is not "future-proof": if clocks resolutions
increase, new Python functions may be needed.
In practive, the resolution of 1 nanosecond is currently enough for all
structures used by all operating systems functions.
Hardware clock with a resolution better than 1 nanosecond already
exists. For example, the frequency of a CPU TSC clock is the CPU base
frequency: the resolution is around 0.3 ns for a CPU running at 3
GHz. Users who have access to such hardware and really need
sub-nanosecond resolution can easyly extend Python for their needs.
Such rare use case don't justify to design the Python standard library
to support sub-nanosecond resolution.
For the CPython implementation, nanosecond resolution is convenient: the
standard and well supported ``int64_t`` type can be used to store time.
It supports a time delta between -292 years and 292 years. Using the
UNIX epoch as reference, this type supports time since year 1677 to year
2262::
>>> 1970 - 2 ** 63 / (10 ** 9 * 3600 * 24 * 365.25)
1677.728976954687
>>> 1970 + 2 ** 63 / (10 ** 9 * 3600 * 24 * 365.25)
2262.271023045313
Different types
---------------
It was proposed to modify ``time.time()`` to use float type with better
precision. The PEP 410 proposed to use ``decimal.Decimal``, but it was
rejected. Apart ``decimal.Decimal``, no portable ``float`` type with
better precision is currently available in Python. Changing the builtin
Python ``float`` type is out of the scope of this PEP.
Other ideas of new types were proposed to support larger or arbitrary
precision: fractions, structures or 2-tuple using integers,
fixed-precision floating point number, etc.
See also the PEP 410 for a previous long discussion on other types.
Adding a new type requires more effort to support it, than reusing
``int``. The standard library, third party code and applications would
have to be modified to support it.
The Python ``int`` type is well known, well supported, ease to
manipulate, and supports all arithmetic operations like:
``dt = t2 - t1``.
Moreover, using nanoseconds as integer is not new in Python, it's
already used for ``os.stat_result`` and
``os.utime(ns=(atime_ns, mtime_ns))``.
.. note::
If the Python ``float`` type becomes larger (ex: decimal128 or
float128), the ``time.time()`` precision will increase as well.
Different API
-------------
The ``time.time(ns=False)`` API was proposed to avoid adding new
functions. It's an uncommon (and bad?) programming practice in Python to
change the result type depending on a parameter.
Different options were proposed to allow the user to choose the time
resolution. If each Python module uses a different resolution, it can
become difficult to handle different resolutions, instead of just
seconds (``time.time()`` returning ``float``) and nanoseconds
(``time.time_ns()`` returning ``int``). Moreover, as written above,
there is no need for resolution better than 1 nanosecond in practive in
the Python standard library.
Annex: Clocks Resolution in Python
==================================
Script ot measure the smallest difference between two ``time.time()`` and
``time.time_ns()`` reads ignoring differences of zero::
import math
import time
LOOPS = 10 ** 6
print("time.time_ns(): %s" % time.time_ns())
print("time.time(): %s" % time.time())
min_dt = [abs(time.time_ns() - time.time_ns())
for _ in range(LOOPS)]
min_dt = min(filter(bool, min_dt))
print("min time_ns() delta: %s ns" % min_dt)
min_dt = [abs(time.time() - time.time())
for _ in range(LOOPS)]
min_dt = min(filter(bool, min_dt))
print("min time() delta: %s ns" % math.ceil(min_dt * 1e9))
Results of time(), perf_counter() and monotonic().
Linux (kernel 4.12 on Fedora 26):
* time_ns(): **84 ns**
* time(): **239 ns**
* perf_counter_ns(): 84 ns
* perf_counter(): 82 ns
* monotonic_ns(): 84 ns
* monotonic(): 81 ns
Windows 8.1:
* time_ns(): **318000 ns**
* time(): **894070 ns**
* perf_counter_ns(): 100 ns
* perf_counter(): 100 ns
* monotonic_ns(): 15000000 ns
* monotonic(): 15000000 ns
The difference on ``time.time()`` is significant: **84 ns (2.8x better)
vs 239 ns on Linux and 318 us (2.8x better) vs 894 us on Windows**. The
difference (presion loss) will be larger next years since every day adds
864,00,000,000,000 nanoseconds to the system clock.
The difference on ``time.perf_counter()`` and ``time.monotonic clock()``
is not visible in this quick script since the script runs less than 1
minute, and the uptime of the computer used to run the script was
smaller than 1 week. A significant difference should be seen with an
uptime of 104 days or greater.
.. note::
Internally, Python starts ``monotonic()`` and ``perf_counter()``
clocks at zero on some platforms which indirectly reduce the
precision loss.
Copyright
=========
This document has been placed in the public domain.
Hi,
We are using Mailman 3 for the new buildbot-status mailing list and it
works well:
https://mail.python.org/mm3/archives/list/buildbot-status@python.org/
I prefer to read archives with this UI, it's simpler to follow
threads, and it's possible to reply on the web UI!
To be honest, we got some issues when the new security-announce
mailing list was quickly migrated from Mailman 2 to Mailman 3, but
issues were quicky fixed as well.
Would it be possible to migrate python-dev to Mailman 3? Do you see
any blocker issue?
I sent to email to the Python postmaster as well.
Victor
According to the specification of format string syntax [1] (I meant
str.format(), not f-strings), both argument name and attribute name must
be Python identifiers.
But the current implementation is more lenient and allow arbitrary
sequences of characters while they don't contain '.', '[', ']', '{',
'}', ':', '!'.
>>> '{#}'.format_map({'#': 42})
'42'
>>> import types
>>> '{0.#}'.format(types.SimpleNamespace(**{'#': 42}))
'42'
This can be confusing due to similarity with the format string syntaxes
in str.format() and f-strings.
>> name = 'abc'
>>> f'{name.upper()}'
'ABC'
>>> '{name.upper()}'.format(name='abc')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'str' object has no attribute 'upper()'
If accept only identifiers, we could produce more specific error message.
Is there a bug in the documentation or in the implementation?
[1] https://docs.python.org/3/library/string.html#format-string-syntax
The copy() methods of list, dict, bytearray, set, frozenset,
WeakValueDictionary, WeakKeyDictionary return an instance of the base
type containing the content of the original collection.
The copy() methods of deque, defaultdict, OrderedDict, Counter,
ChainMap, UserDict, UserList, WeakSet, ElementTree.Element return an
instance of the same type as the original collection.
The copy() method of mappingproxy returns a copy of the underlying
mapping (using its copy() method).
os.environ.copy() returns a dict.
Shouldn't it be more consistent?
==============================================================
SQLITE3 Function
def get_db():
def dict_factory(cursor, row):
d = {}
for idx, col in enumerate(cursor.description):
d[col[0]] = row[idx]
return d
db = getattr(g, '_database', None)
if db is None:
db = g._database = sqlite3.connect(DATABASE)
db.row_factory = dict_factory
return db
================================================================
I try this form but isnt ok . Can I help me ?
import cx_Oracle
con = cx_Oracle.connect('xxxx/xxxx@xxxxxxx/xxxxxxx')
cur = con.cursor()
cur.execute("select * from test")
desc = [d[0] for d in cur.description]
result = [dict(zip(dec,line))for line in cur]
print (result)
cur.close()
--
Carlos Eugênio
