[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[lmi-commits] [lmi] valyuta/005 3d44f42 13/17: Save a sorted list of reg
From: |
Greg Chicares |
Subject: |
[lmi-commits] [lmi] valyuta/005 3d44f42 13/17: Save a sorted list of regressions |
Date: |
Sat, 16 Jan 2021 21:06:18 -0500 (EST) |
branch: valyuta/005
commit 3d44f42267b4c97e7a0bf6918afcc0af729041a0
Author: Gregory W. Chicares <gchicares@sbcglobal.net>
Commit: Gregory W. Chicares <gchicares@sbcglobal.net>
Save a sorted list of regressions
Added "$(test_dir)/regressions.tsv" to the 'system_test' recipe. This
TSV file is readable as either a text file or a spreadsheet. Its
contents are sorted in decreasing order by absolute error (primary)
and relative error (secondary), making it easy to see which files have
the severest regressions.
---
workhorse.make | 18 +++++++++++++-----
1 file changed, 13 insertions(+), 5 deletions(-)
diff --git a/workhorse.make b/workhorse.make
index 794df3c..9b1b5bf 100644
--- a/workhorse.make
+++ b/workhorse.make
@@ -1441,12 +1441,15 @@ test_result_suffixes := test test0 test1
monthly_trace.* mec.tsv mec.xml gpt.tsv
# names to accumulate history in $(test_dir), which permits comparison
# across several versions. An undatestamped copy of the md5sums is
# made for storage in a version-control system (which naturally keeps
-# historical versions in its own way).
+# historical versions in its own way). An undatestamped copy of the
+# analysis, stripped of descriptive text and sorted in descending
+# order, is saved as 'regressions.tsv'.
-system_test_analysis := $(test_dir)/analysis-$(yyyymmddhhmm)
-system_test_diffs := $(test_dir)/diffs-$(yyyymmddhhmm)
-system_test_md5sums := $(test_dir)/md5sums-$(yyyymmddhhmm)
-system_test_md5sums2 := $(test_dir)/md5sums
+system_test_analysis := $(test_dir)/analysis-$(yyyymmddhhmm)
+system_test_regressions := $(test_dir)/regressions.tsv
+system_test_diffs := $(test_dir)/diffs-$(yyyymmddhhmm)
+system_test_md5sums := $(test_dir)/md5sums-$(yyyymmddhhmm)
+system_test_md5sums2 := $(test_dir)/md5sums
%.cns: test_emission := emit_quietly,emit_test_data
%.ill: test_emission := emit_quietly,emit_test_data
@@ -1506,6 +1509,11 @@ system_test: $(datadir)/configurable_settings.xml
$(touchstone_md5sums) install
-e '/rel err.*e-0*1[5-9]/d' \
-e '/abs.*0\.00.*rel/d' \
-e '/abs diff: 0 /d'
+ @-< $(system_test_analysis) $(SED) \
+ -e 's/ Summary: max abs diff: /\t/' \
+ -e 's/ max rel err: /\t/' \
+ | $(SORT) --key=2gr --key=3gr \
+ > $(system_test_regressions)
@$(DIFF) --brief $(system_test_md5sums) $(touchstone_md5sums) \
&& $(ECHO) "All `<$(touchstone_md5sums) $(WC) -l` files match." \
|| $(MAKE) --file=$(this_makefile) system_test_discrepancies
- [lmi-commits] [lmi] branch valyuta/005 created (now 4401725), Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 fd74b61 03/17: Correct an obvious mistake, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 7d6e748 02/17: Improve currency class and unit tests, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 4902da4 01/17: Reinvent currency ab ovo, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 ffa2ce4 09/17: Redesign unit test, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 90d7483 10/17: Improve currency class, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 3d44f42 13/17: Save a sorted list of regressions,
Greg Chicares <=
- [lmi-commits] [lmi] valyuta/005 c8864d3 04/17: Add commented-out relops, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 f3219f8 12/17: Use CURRENCY_UNIT_IS_CENTS appropriately, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 aad51dd 07/17: Inline more, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 b852a68 05/17: Add features, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 d69f05c 06/17: Ratify some regression-testing differences, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 2aae91a 14/17: Improve incrementally, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 0735250 15/17: Improve incrementally, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 dfdbf3c 16/17: Use currency type for class members as appropriate, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 221c094 08/17: Prefer constexpr, Greg Chicares, 2021/01/16
- [lmi-commits] [lmi] valyuta/005 4401725 17/17: Merge branch 'master' into valyuta/005, Greg Chicares, 2021/01/16