I thought Piston had been shuttered by Dropbox, so I wouldn't worry about
convincing them to change speed.pyston.org.
On Mon, May 29, 2017, 13:57 Victor Stinner, <victor.stinner(a)gmail.com>
wrote:
> 2017-05-29 22:45 GMT+02:00 Antoine Pitrou <solipsis(a)pitrou.net>:
> > I don't know. It means that benchmark results published on the Web
> > are generally not comparable with each other unless they happen to be
> > generated with the exact same version. It reduces the usefulness of
> > the benchmarks suite quite a bit IMHO.
>
> I only know a 3 websites to compare Python performances:
>
> * speed.python.org
> * speed.pypy.org
> * speed.pyston.org
>
> My goal is to convaince PyPy developers to use performance. I'm not
> sure that pyston.org is revelant: it seems like their forked benchmark
> suite is modified, so I don't expect that results on pypy.org and
> pyston.org are comparable. I would also prefer that Pyston uses the
> same benchmark suite.
>
> About speed.python.org, what was decided is to *drop* all previous
> results if we modify benchmarks. That's what I already did 3 times:
>
> * 2017-03-31: old results removed, new CPython results to use Git
> commits instead of Mercurial.
> * 2017-01: old results computed without PGO removed (unstable because
> of code placement), new CPython results using PGO
> * 2016-11-04: old results computed with benchmarks removed, new
> CPython results (using LTO but not PGO) computed with the new
> performance benchmark suite.
>
> To be honest, in the meanwhile, I chose to run the master branch of
> perf and performance to develop perf and performance. In practice, I
> never noticed any significant performance change on any performance
> the last 12 months when I updated dependencies. Sadly, it seems like
> no significant optimization was merged in our dependencies.
>
> > Let's ask the question a different way: was there any necessity to
> > update those dependencies? If yes, then fair enough. Otherwise, the
> > compatibility breakage is gratuitous.
>
> When I started to work on benchmarks last year, I noticed that we used
> a Mercurial version which was 5 years old, and a Django version which
> was something like 3 years old. I would like to benchmark the
> Mercurial and Django versions deployed on production.
>
> Why do you want to update performance if you want a pinned version of
> Django? Just always use the same performance version, no?
>
> For speed.python.org, maybe we can decide that we always use a fixed
> version of performance, and that we must remove all data each time we
> change the performance version. For my needs, maybe we could spawn a
> "beta" subdomain running master branches? Again, I expect no
> significant difference between the main website and the beta website.
> But we can do it if you want.
>
> Victor
> _______________________________________________
> Speed mailing list
> Speed(a)python.org
> https://mail.python.org/mailman/listinfo/speed
>
2017-05-29 22:57 GMT+02:00 Victor Stinner <victor.stinner(a)gmail.com>:
> When I started to work on benchmarks last year, I noticed that we used
> a Mercurial version which was 5 years old, and a Django version which
> was something like 3 years old. I would like to benchmark the
> Mercurial and Django versions deployed on production.
Hum, I sent my email before checking.
At revision 9923b81a1d34, benchmarks used:
lib/2to3
lib3/2to3
lib3/Chameleon-2.9.2
lib3/mako-0.3.6
lib3/Mako-0.7.3
lib/bazaar
lib/Chameleon-2.22
lib/Chameleon-2.9.2
lib/django
lib/Django-1.5
lib/Django-1.9
lib/google_appengine
lib/html5lib
lib/lockfile
lib/mako-0.3.6
lib/Mako-0.7.3
lib/mercurial # version 1.2.1
lib/pathlib
lib/psyco
lib/rietveld
lib/spambayes
lib/spitfire
lib/tornado-3.1.1
Django 1.9 was released at 2015-12-01, Mercurial 1.2.1 was released at
2009-03-20.
So sorry, Django was kind of up to date, but Mercurial wasn't. Many
libraries were also deprecated.
Victor
On Mon, 29 May 2017 22:29:44 +0200
Victor Stinner <victor.stinner(a)gmail.com>
wrote:
>
> > This is quite common in network and/or distributed programming where you
> > want to allow logging many events for diagnosis of unexpected runtime
> > issues (because many unexpected conditions can appear), but with
> > those logs disabled by default for performance and readability reasons.
>
> Note: I would suggest to use a preprocessor or something like to
> *remove* the calls if performance is critical. It is the solution was
> chose in a previous company working on embedded devices :-)
Then you lose the ability to enable debug logs on user /
customer systems. Unless the preprocessor is run each time the user
changes the logging configuration, which adds another layer of
fragility.
> But I'm not sure that it's really doable. I don't think that it's
> matter so much to provide backward compatibility. It's very easy to
> get an old version of performance if you want to compare a new version
> with old results.
I don't know. It means that benchmark results published on the Web
are generally not comparable with each other unless they happen to be
generated with the exact same version. It reduces the usefulness of
the benchmarks suite quite a bit IMHO.
Let's ask the question a different way: was there any necessity to
update those dependencies? If yes, then fair enough. Otherwise, the
compatibility breakage is gratuitous.
Regards
Antoine.
On Mon, 29 May 2017 18:49:37 +0200
Victor Stinner <victor.stinner(a)gmail.com>
wrote:
> * The ``float`` benchmark now uses ``__slots__`` on the ``Point`` class.
So the benchmark numbers are not comparable with previously generated
ones?
> * Remove the following microbenchmarks. They have been moved to the
> `pymicrobench <https://github.com/haypo/pymicrobench>`_ project because
> they are too short, not representative of real applications and are too
> unstable.
>
[...]
> - ``logging_silent``: values are faster than 1 ns on PyPy with 2^27 loops!
> (and around 0.7 us on CPython)
The performance of silent logging calls is actually important for
all applications which have debug() calls in their critical paths.
This is quite common in network and/or distributed programming where you
want to allow logging many events for diagnosis of unexpected runtime
issues (because many unexpected conditions can appear), but with
those logs disabled by default for performance and readability reasons.
This is no more a micro-benchmark than is, say, pickling or JSON
encoding; and much less so than solving the N-body problem in pure
Python without Numpy...
> * Update requirements
>
> - Django: 1.11 => 1.11.1
> - SQLAlchemy: 1.1.9 => 1.1.10
> - certifi: 2017.1.23 => 2017.4.17
> - perf: 1.2 => 1.3
> - mercurial: 4.1.2 => 4.2
> - tornado: 4.4.3 => 4.5.1
Are those requirements for the benchmark runner or for the benchmarks
themselves? If the latter, won't updating the requirements make
benchmark numbers non-comparable with those generated by previous
versions? This is something that the previous benchmarks suite tried
to above by using pinned versions of 3rd party libraries.
Regards
Antoine.
Hi,
I just released performance 0.5.5 and perf 1.3. The main change is the
removal of microbenchmarks from performance, you can now find them in
a different project:
https://github.com/haypo/pymicrobench
The perf warmup calibration is still highly unstable, and so PyPy
still don't use performance yet. I began to analyze warmups manually
but I was too lazy to finish the work:
http://haypo-notes.readthedocs.io/pypy_warmups.html
=== performance 0.5.5 ===
* On the 2.x branch on CPython, ``compile`` now pass ``--enable-unicode=ucs4``
to the ``configure`` script on all platforms, except on Windows which uses
UTF-16 because of its 16-bit wchar_t.
* The ``float`` benchmark now uses ``__slots__`` on the ``Point`` class.
* Remove the following microbenchmarks. They have been moved to the
`pymicrobench <https://github.com/haypo/pymicrobench>`_ project because
they are too short, not representative of real applications and are too
unstable.
- ``pybench`` microbenchmark suite
- ``call_simple``
- ``call_method``
- ``call_method_unknown``
- ``call_method_slots``
- ``logging_silent``: values are faster than 1 ns on PyPy with 2^27 loops!
(and around 0.7 us on CPython)
* Update requirements
- Django: 1.11 => 1.11.1
- SQLAlchemy: 1.1.9 => 1.1.10
- certifi: 2017.1.23 => 2017.4.17
- perf: 1.2 => 1.3
- mercurial: 4.1.2 => 4.2
- tornado: 4.4.3 => 4.5.1
=== perf 1.3 ===
* Add ``get_loops()`` and ``get_inner_loops()`` methods to Run and Benchmark
classes
* Documentation: add export_csv.py and plot.py examples
* Rewrite warmup calibration for PyPy:
- Use Q1, Q3 and stdev, rather than mean and checking if the first value
is an outlier
- Always use a sample of 10 values, rather than using a sample of a variable
size starting with 3 values
* Use lazy import for most imports of the largest modules to reduce the
number of imported module on 'import perf'.
* Fix handling of broken pipe error to prevent logging the error:
"Exception ignored in: ... BrokenPipeError: ..."
* ``collect_metadata`` gets more metadata on FreeBSD:
- use ``os.getloadavg()`` if ``/proc/loadavg`` is not available (ex: FreeBSD)
- use ``psutil.boot_time()`` if ``/proc/stat`` is not available (ex: FreeBSD)
to get ``boot_time`` and ``uptime`` metadata
* The Runner constructor now raises an exception if more than one instance is
created.
Victor
Thanks for the update, I'm processing unread e-mails ;-)
--
Ludovic Gasc (GMLudo)
Lead Developer Architect at ALLOcloud
https://be.linkedin.com/in/ludovicgasc
2017-05-10 18:16 GMT+02:00 Victor Stinner <victor.stinner(a)gmail.com>:
> FYI since my previous email in March, I removed the following
> microbenchmarks from pyperformance to move them to pymicrobench:
>
> * call_simple, call_method, call_method_slots, call_method_unknown
> * logging_silent
>
> pyperformance development version:
> http://pyperformance.readthedocs.io/changelog.html#version-0-5-5
>
> https://github.com/haypo/pymicrobench
>
> Victor
>
Interesting, you might ping python-dev with this repository.
--
Ludovic Gasc (GMLudo)
Lead Developer Architect at ALLOcloud
https://be.linkedin.com/in/ludovicgasc
2017-03-17 3:29 GMT+01:00 Victor Stinner <victor.stinner(a)gmail.com>:
> Hi,
>
> I started to create a collection of microbenchmarks for CPython from
> scripts found on the bug tracker:
> https://github.com/haypo/pymicrobench
>
> I'm not sure that this collection is used yet, but some of you may
> want to take a look :-)
>
> I know that some people have random microbenchmarks in a local
> directory. Maybe you want to share them?
>
> I don't really care to sort them or group them. My plan is first to
> populate the repository, and later see what to do with it :-)
>
> Victor
> _______________________________________________
> Speed mailing list
> Speed(a)python.org
> https://mail.python.org/mailman/listinfo/speed
>
