Benchmark diffs not being output on threshold regression

Motivation

* `check_benchmark_thresholds.sh` was updated to check for exit code 2
from `swift package benchmark thresholds check` to distinguish
threshold regression from build errors.
* It seems SwiftPM's `CommandPlugin` infrastructure always exits with code 1 when
`performCommand` throws, regardless of the error's raw value. So the
exit code is never 2, causing all regressions to fall through to the
`else` branch and be misreported as build errors, with no diff output.

Modifications

* Remove the `rc == 2` check and instead attempt `thresholds update`
for any non-zero `rc` from `thresholds check`.
* Use the result of `thresholds update` to distinguish regression
(success) from build error (failure).
* Remove `--exit-code` from `git diff` to prevent `set -uo pipefail`
from aborting the script before output is fully flushed.

Result

* Benchmark threshold regressions correctly output the `=== BEGIN DIFF
===` section again.
* Actual build errors are still correctly detected and reported.
This commit is contained in:
Rick Newton-Rogers committed 2026-02-20 15:30:18 +00:00
1 parent 535c2f8260
commit af3eee91b9
1 file changed
+17 -19
+17 -19
View File
@@ -37,25 +37,23 @@ rc="$?"
# Benchmarks are unchanged, nothing to recalculate
if [[ "$rc" == 0 ]]; then
exit 0
# Benchmark thresholds changed – 2 is defined in
# package-benchmark/BenchmarkShared/Command+Helpers.swift/ExitCode.thresholdRegression
elif [[ "$rc" == 2 ]]; then
log "Recalculating thresholds..."
swift package --package-path "$benchmark_package_path" "${swift_package_arguments[@]}" benchmark thresholds update --format metricP90AbsoluteThresholds --path "${benchmark_package_path}/Thresholds/${swift_version}/"
update_rc="$?"
if [[ "$update_rc" != 0 ]]; then
error "Benchmark failed to run due to build error."
exit $update_rc
fi
echo "=== BEGIN DIFF ===" # use echo, not log for clean output to be scraped
git diff --exit-code HEAD
# all other errors
else
error "Benchmark failed to run due to build error."
exit $rc
fi
# Non-zero exit from 'thresholds check' means thresholds regressed or a build
# error occurred. Try 'thresholds update' to distinguish: if it also fails, it
# was a build error; if it succeeds, thresholds were updated.
log "Recalculating thresholds..."
swift package --package-path "$benchmark_package_path" "${swift_package_arguments[@]}" benchmark thresholds update --format metricP90AbsoluteThresholds --path "${benchmark_package_path}/Thresholds/${swift_version}/"
update_rc="$?"
if [[ "$update_rc" != 0 ]]; then
error "Benchmark failed to run due to build error."
exit $update_rc
fi
echo "=== BEGIN DIFF ===" # use echo, not log for clean output to be scraped
git diff HEAD
exit 1