Hi,
My question is simple: do we officially support Solaris and/or OpenIndiana?
Jesus Cea runs an OpenIndiana buildbot slave:
http://buildbot.python.org/all/buildslaves/cea-indiana-x86
"Open Indiana 32 bits"
The platform module of Python says "Solaris-2.11", I don't know the
exact OpenIndiana version.
A lot of unit tests fail on this buildbot with MemoryError. I guess
that it's related to Solaris which doesn't allow overcommit
(allocating more memory than available memory on the system), or more
simply because the slave has not enough memory.
There is now an issue which seems specific to OpenIndiana:
http://bugs.python.org/issue27847
It might impact Solaris as well, but the Solaris buildbot is offline
since "684 builds".
Five years ago, I reported a bug because the curses module of Python 3
doesn't build on Solaris nor OpenIndiana anymore. It seems like the
bug was not fixed, and the issue is still open:
http://bugs.python.org/issue13552
So my question is if we officially support Solaris and/or OpenIndiana.
If yes, how can we fix issues when we only have buildbot slave which
has memory errors, and no SSH access to this server?
Solaris doesn't seem to be officially supported in Python, so I
suggest to drop the OpenIndiana buildbot (which is failing since at
least 2 years) and close all Solaris issues as "WONTFIX".
Victor
Classes that doesn't define the __format__ method for custom PEP 3101
formatting inherits it from parents.
Originally the object.__format__ method was designed as [1]:
def __format__(self, format_spec):
return format(str(self), format_spec)
An instance is converted to string and resulting string is formatted
according to format specifier.
Later this design was reconsidered [2], and now object.__format__ is
equivalent to:
def __format__(self, format_spec):
assert format_spec == ''
return format(str(self), '')
Non-empty format specifier is rejected.
But why call format() on resulting string? Why not return resulting
string as is? object.__format__ could be simpler (not just
implementation, but for understanding):
def __format__(self, format_spec):
assert format_spec == ''
return str(self)
This can change the behaviour in corner case. str(self) can return not
exact string, but string subclass with overloaded __format__. But I
think we can ignore such subtle difference.
[1] https://www.python.org/dev/peps/pep-3101/
[2] http://bugs.python.org/issue7994
Hi all,
It's an old feature of the weakref API that you can define an
arbitrary callback to be invoked when the referenced object dies, and
that when this callback is invoked, it gets handed the weakref wrapper
object -- BUT, only after it's been cleared, so that the callback
can't access the originally referenced object. (I.e., this callback
will never raise: def callback(ref): assert ref() is None.)
AFAICT the original motivation for this seems was that if the weakref
callback could get at the object, then the weakref callback would
effectively be another finalizer like __del__, and finalizers and
reference cycles don't mix, so weakref callbacks can't be finalizers.
There's a long document from the 2.4 days about all the terrible
things that could happen if arbitrary code like callbacks could get
unfettered access to cyclic isolates at weakref cleanup time [1].
But that was 2.4. In the mean time, of course, PEP 442 fixed it so
that finalizers and weakrefs mix just fine. In fact, weakref callbacks
are now run *before* __del__ methods [2], so clearly it's now okay for
arbitrary code to touch the objects during that phase of the GC -- at
least in principle.
So what I'm wondering is, would anything terrible happen if we started
passing still-live weakrefs into weakref callbacks, and then clearing
them afterwards? (i.e. making step 1 of the PEP 442 cleanup order be
"run callbacks and then clear weakrefs", instead of the current "clear
weakrefs and then run callbacks"). I skimmed through the PEP 442
discussion, and AFAICT the rationale for keeping the old weakref
behavior was just that no-one could be bothered to mess with it [3].
[The motivation for my question is partly curiosity, and partly that
in the discussion about how to handle GC for async objects, it
occurred to me that it might be very nice if arbitrary classes that
needed access to the event loop during cleanup could do something like
def __init__(self, ...):
loop = asyncio.get_event_loop()
loop.gc_register(self)
# automatically called by the loop when I am GC'ed; async equivalent
of __del__
async def aclose(self):
...
Right now something *sort* of like this is possible but it requires a
much more cumbersome API, where every class would have to implement
logic to fetch a cleanup callback from the loop, store it, and then
call it from its __del__ method -- like how PEP 525 does it. Delaying
weakref clearing would make this simpler API possible.]
-n
[1] https://github.com/python/cpython/blob/master/Modules/gc_weakref.txt
[2] https://www.python.org/dev/peps/pep-0442/#id7
[3] https://mail.python.org/pipermail/python-dev/2013-May/126592.html
--
Nathaniel J. Smith -- https://vorpus.org
Hi,
Last months, I worked a lot on benchmarks. I ran benchmarks, analyzed
results in depth (up to the hardware and kernel drivers!), I wrote new
tools and enhanced existing tools.
* I wrote a new perf module which runs benchmarks in a reliable way
and contains a LOT of features: collect metadata, JSON file format,
commands to compare, render an histogram, etc.
* I rewrote the Python benchmark suite: the old benchmarks Mercurial
repository moved to a new performance GitHub project which uses my
perf module and contains more benchmarks.
* I also made minor enhancements to timeit in Python 3.7 -- some dev
don't want major changes to not "break the backward compatibility".
For timeit, I suggest to use my perf tool which includes a reliable
timeit command and has much more features like --duplicate (repeat the
statements to reduce the cost of the outer loop) and --compare-to
(compare two versions of Python), but also all builtin perf features
(JSON output, statistics, histogram, etc.).
I added benchmarks from PyPy and Pyston benchmark suites to
performance: performance 0.3.1 contains 51 benchmark scripts which run
a total of 121 benchmarks. Example of tested Python modules:
* SQLAlchemy
* Dulwich (full Git implementation in Python)
* Mercurial (currently only the startup time)
* html5lib
* pyaes (AES crypto cipher in pure Python)
* sympy
* Tornado (HTTP client and server)
* Django (sadly, only the template engine right now, Pyston contains
HTTP benchmarks)
* pathlib
* spambayes
More benchmarks will be added later. It would be nice to add
benchmarks on numpy for example, numpy is important for a large part
of our community.
All these (new or updated) tools can now be used to take smarter
decisions on optimizations. Please don't push any optimization anymore
without providing reliable benchmark results!
My first major action was to close the latest attempt to
micro-optimize int+int in Python/ceval.c,
http://bugs.python.org/issue21955 : I closed the issue as rejected,
because there is no significant speedup on benchmarks other than two
(tiny) microbenchmarks. To make sure that no one looses its time on
trying to micro-optimize int+int, I even added a comment to
Python/ceval.c :-)
https://hg.python.org/cpython/rev/61fcb12a9873
"Please don't try to micro-optimize int+int"
The perf and performance are now well tested: Travis CI runs tests on
the new commits and pull requests, and the "tox" command can be used
locally to test different Python versions, pep8, doc, ... in a single
command.
Next steps:
* Run performance 0.3.1 on speed.python.org: the benchmark runner is
currently stopped (and still uses the old benchmarks project). The
website part may be updated to allow to download full JSON files which
includes *all* information (all timings, metadata and more).
* I plan to run performance on CPython 2.7, CPython 3.7, PyPy and PyPy
3. Maybe also CPython 3.5 and CPython 3.6 if they don't take too much
resources.
* Later, we can consider adding more implementations of Python:
Jython, IronPython, MicroPython, Pyston, Pyjion, etc. All benchmarks
should be run on the same hardware to be comparable.
* Later, we might also allow other projects to upload their own
benchmark results, but we should find a solution to groups benchmark
results per benchmark runner (ex: at least by the hostname, perf JSON
contains the hostname) to not compare two results from two different
hardware
* We should continue to add more benchmarks to the performance
benchmark suite, especially benchmarks more representative of real
applications (we have enough microbenchmarks!)
Links:
* perf: http://perf.readthedocs.io/
* performance: https://github.com/python/performance
* Python Speed mailing list: https://mail.python.org/mailman/listinfo/speed
* https://speed.python.org/ (currently outdated, and don't use performance yet)
See https://pypi.python.org/pypi/performance which contains even more
links to Python benchmarks (PyPy, Pyston, Numba, Pythran, etc.)
Victor
I need to do a little 2.6 spelunking. I don't see a 2.6 branch in the
output of "hg branches". Is "hg clone v2.6.9" the proper incantation to get
the latest version (or perhaps "v2.6")?
Thx,
Skip
(Added python-dev in CC list, because there are enough +1 already).
On Mon, Oct 17, 2016 at 3:06 PM, Chris Angelico <rosuav(a)gmail.com> wrote:
> On Mon, Oct 17, 2016 at 5:02 PM, INADA Naoki <songofacandy(a)gmail.com> wrote:
>> $ ./python.exe -V
>> Python 3.6.0b2+
>>
>> $ ./python.exe -VV
>> Python 3.6.0b2+ (3.6:0b29adb5c804+, Oct 17 2016, 15:00:12)
>> [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)]
>
> LGTM.
>
> What's the view on backporting this to 2.7.x? We're still a good few
> years away from its death, and it'd be helpful if recent 2.7s could
> give this info too.
>
> ChrisA
I want to add it at least Python 3.6. Because one reason I want to
propose this is
I can't see exact Python version (commit id) for "nightly" or
"3.6-dev" on Travis-CI test.
But Python 3.6 is beta stage already. If we apply rule strictly, it
should be added
only in default branch (Python 3.7).
So, what version can I add this?
a. Only Python 3.7+
b. (beta) Python 3.6+
c. (maintenance) Python 2.7 and Python 3.5+
--
INADA Naoki <songofacandy(a)gmail.com>
The third, and next-to-last, beta snapshot planned for the 3.6 release cycle is coming up in a few days. With fewer than 7 weeks remaining until the 3.6.0 release, it is very important that we all focus on stability and correctness. Please try to make sure that all remaining non-doc issues associated with new features are addressed in b3. Any remaining 3.6 feature-related issues still open after b3 should be marked in the issue tracker as "release blocker" and *must* be addressed one way or another by b4, 11-21. All other non-critical bug fixes should also be checked in by b4. Only release critical and doc fixes will be allowed once we exit the beta phase. Please plan accordingly.
Please contact me if you have any questions about the 3.6.0 schedule or about whether a change is appropriate at this point in the 3.6.0 cycle.
To recap, the remaining milestones for 3.6.0:
2016-10-31, 1200 UTC: 3.6.0 beta 3 (feature fixes, bug fixes, doc fixes)
2016-11-21: 3.6.0 beta 4 (important bug fixes and doc fixes)
2016-12-05 3.6.0 release candidate 1 (3.6.0 code freeze, release critical bug fixes, doc fixes)
2016-12-16 3.6.0 release (3.6.0rc1 plus any necessary emergency fixes)
Thank you all again for your efforts so far on 3.6!
--Ned
https://www.python.org/dev/peps/pep-0494/
--
Ned Deily
nad(a)python.org -- []
The docs for this class state:
"Future instances are created by Executor.submit() and should not be
created directly except for testing."
https://docs.python.org/3/library/concurrent.futures.html#future-objects
We have a need for a thread-safe future type in our extension but this
statement makes us hesitate to use it. We don't need the executor
functionality.
We can write our own future class easily enough, we're just wondering what
the justification was for the limitations mentioned in the docs.
Thanks,
Mark Spruiell
ZeroC, Inc.
Hi.
While there were no reply to my previous post on Python-ideas ML,
Now I'm sure about bytes.frombuffer() is worth enough.
Let's describe why I think it's important.
Background
=========
>From Python 3.4, bytearray is good solution for I/O buffer, thanks to
#19087 [1].
Actually, asyncio uses bytearray as I/O buffer often.
When bytearray is used for read buffer, we can parse received data on bytearray
directly, and consume it. For example, read until '\r\n' is easier
than io.BytesIO().
Sample code:
def read_line(buf: bytearray) -> bytes:
try:
n = buf.index(b'\r\n')
except ValueError:
return b''
line = bytes(buf)[:n] # bytearray -> bytes -> bytes
del buf[:n+2]
return line
buf = bytearray(b'foo\r\nbar\r\nbaz\r\n')
while True:
line = read_line(buf)
if not line:
break
print(line)
As you can see, redundant temporary bytes is used.
This is not ideal for performance and memory efficiency.
Since code like this is typically in lower level code (e.g. asyncio),
performance and
efficiency is important.
[1] https://bugs.python.org/issue19087
(Off topic: bytearray is nice for write buffer too. written =
s.send(buf); del buf[:written];)
Memoryview problem
=================
To avoid redundant copy of `line = bytes(buf)[:n]`, current solution
is using memoryview.
First code I wrote is: `line = bytes(memoryview(buf)[:n])`.
On CPython, it works fine. But `del buff[:n+2]` in next line may fail
on other Python
implementations. Changing bytearray size is inhibited while
memoryview is alive.
So right code is:
with memoryview(buf) as m:
line = bytes(m[:n])
The problem of memoryview approach is:
* Overhead: creating temporary memoryview, __enter__, and __exit__. (see below)
* It isn't "one obvious way": Developers including me may forget to
use context manager.
And since it works on CPython, it's hard to point it out.
Quick benchmark:
(temporary bytes)
$ python3 -m perf timeit -s 'buf =
bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- 'bytes(buf)[:3]'
....................
Median +- std dev: 652 ns +- 19 ns
(temporary memoryview without "with"
$ python3 -m perf timeit -s 'buf =
bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- 'bytes(memoryview(buf)[:3])'
....................
Median +- std dev: 886 ns +- 26 ns
(temporary memoryview with "with")
$ python3 -m perf timeit -s 'buf = bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- '
with memoryview(buf) as m:
bytes(m[:3])
'
....................
Median +- std dev: 1.11 us +- 0.03 us
Proposed solution
===============
Adding one more constructor to bytes:
# when length=-1 (default), use until end of *byteslike*.
bytes.frombuffer(byteslike, length=-1, offset=0)
With ths API
with memoryview(buf) as m:
line = bytes(m[:n])
becomes
line = bytes.frombuffer(buf, n)
Thanks,
--
INADA Naoki <songofacandy(a)gmail.com>
