[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH v6 14/15] scripts/simplebench: improve ascii table: add diffe
From: |
Max Reitz |
Subject: |
Re: [PATCH v6 14/15] scripts/simplebench: improve ascii table: add difference line |
Date: |
Fri, 25 Sep 2020 12:24:44 +0200 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 |
On 18.09.20 20:19, Vladimir Sementsov-Ogievskiy wrote:
> Performance improvements / degradations are usually discussed in
> percentage. Let's make the script calculate it for us.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> scripts/simplebench/simplebench.py | 46 +++++++++++++++++++++++++++---
> 1 file changed, 42 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/simplebench/simplebench.py
> b/scripts/simplebench/simplebench.py
> index 56d3a91ea2..0ff05a38b8 100644
> --- a/scripts/simplebench/simplebench.py
> +++ b/scripts/simplebench/simplebench.py
[...]
> + for j in range(0, i):
> + env_j = results['envs'][j]
> + res_j = case_results[env_j['id']]
> +
> + if 'average' not in res_j:
> + # Failed result
> + cell += ' --'
> + continue
> +
> + col_j = chr(ord('A') + j)
> + avg_j = res_j['average']
> + delta = (res['average'] - avg_j) / avg_j * 100
I was wondering why you’d subtract, when percentage differences usually
mean a quotient. Then I realized that this would usually be written as:
(res['average'] / avg_j - 1) * 100
> + delta_delta = (res['delta'] + res_j['delta']) / avg_j * 100
Why not use the new format_percent for both cases?
> + cell += f' {col_j}{round(delta):+}±{round(delta_delta)}%'
I don’t know what I should think about ±delta_delta. If I saw “Compared
to run A, this is +42.1%±2.0%”, I would think that you calculated the
difference between each run result, and then based on that array
calculated average and standard deviation.
Furthermore, I don’t even know what the delta_delta is supposed to tell
you. It isn’t even a delta_delta, it’s an average_delta.
The delta_delta would be (res['delta'] / res_j['delta'] - 1) * 100.0.
And that might be presented perhaps like “+42.1% Δ± +2.0%” (if delta
were the SD, “Δx̅=+42.1% Δσ=+2.0%” would also work; although, again, I do
interpret ± as the SD anyway).
Max
signature.asc
Description: OpenPGP digital signature
- [PATCH v6 10/15] iotests: qemu_io_silent: support --image-opts, (continued)
- [PATCH v6 10/15] iotests: qemu_io_silent: support --image-opts, Vladimir Sementsov-Ogievskiy, 2020/09/18
- [PATCH v6 13/15] scripts/simplebench: improve view of ascii table, Vladimir Sementsov-Ogievskiy, 2020/09/18
- [PATCH v6 15/15] scripts/simplebench: add bench_prealloc.py, Vladimir Sementsov-Ogievskiy, 2020/09/18
- [PATCH v6 12/15] scripts/simplebench: support iops, Vladimir Sementsov-Ogievskiy, 2020/09/18
- [PATCH v6 14/15] scripts/simplebench: improve ascii table: add difference line, Vladimir Sementsov-Ogievskiy, 2020/09/18
- Re: [PATCH v6 14/15] scripts/simplebench: improve ascii table: add difference line,
Max Reitz <=