mirror of
https://github.com/apple/swift-nio
synced 2026-11-22 04:30:41 +00:00
Benchmark diffs not being output on threshold regression
Motivation * `check_benchmark_thresholds.sh` was updated to check for exit code 2 from `swift package benchmark thresholds check` to distinguish threshold regression from build errors. * It seems SwiftPM's `CommandPlugin` infrastructure always exits with code 1 when `performCommand` throws, regardless of the error's raw value. So the exit code is never 2, causing all regressions to fall through to the `else` branch and be misreported as build errors, with no diff output. Modifications * Remove the `rc == 2` check and instead attempt `thresholds update` for any non-zero `rc` from `thresholds check`. * Use the result of `thresholds update` to distinguish regression (success) from build error (failure). * Remove `--exit-code` from `git diff` to prevent `set -uo pipefail` from aborting the script before output is fully flushed. Result * Benchmark threshold regressions correctly output the `=== BEGIN DIFF ===` section again. * Actual build errors are still correctly detected and reported.
This commit is contained in:
1 parent
535c2f8260
commit
af3eee91b9
1 file changed
+17
-19
@@ -37,25 +37,23 @@ rc="$?"
|
||||
# Benchmarks are unchanged, nothing to recalculate
|
||||
if [[ "$rc" == 0 ]]; then
|
||||
exit 0
|
||||
# Benchmark thresholds changed – 2 is defined in
|
||||
# package-benchmark/BenchmarkShared/Command+Helpers.swift/ExitCode.thresholdRegression
|
||||
elif [[ "$rc" == 2 ]]; then
|
||||
log "Recalculating thresholds..."
|
||||
|
||||
swift package --package-path "$benchmark_package_path" "${swift_package_arguments[@]}" benchmark thresholds update --format metricP90AbsoluteThresholds --path "${benchmark_package_path}/Thresholds/${swift_version}/"
|
||||
update_rc="$?"
|
||||
|
||||
if [[ "$update_rc" != 0 ]]; then
|
||||
error "Benchmark failed to run due to build error."
|
||||
exit $update_rc
|
||||
fi
|
||||
|
||||
echo "=== BEGIN DIFF ===" # use echo, not log for clean output to be scraped
|
||||
git diff --exit-code HEAD
|
||||
# all other errors
|
||||
else
|
||||
error "Benchmark failed to run due to build error."
|
||||
exit $rc
|
||||
fi
|
||||
|
||||
# Non-zero exit from 'thresholds check' means thresholds regressed or a build
|
||||
# error occurred. Try 'thresholds update' to distinguish: if it also fails, it
|
||||
# was a build error; if it succeeds, thresholds were updated.
|
||||
log "Recalculating thresholds..."
|
||||
|
||||
swift package --package-path "$benchmark_package_path" "${swift_package_arguments[@]}" benchmark thresholds update --format metricP90AbsoluteThresholds --path "${benchmark_package_path}/Thresholds/${swift_version}/"
|
||||
update_rc="$?"
|
||||
|
||||
if [[ "$update_rc" != 0 ]]; then
|
||||
error "Benchmark failed to run due to build error."
|
||||
exit $update_rc
|
||||
fi
|
||||
|
||||
echo "=== BEGIN DIFF ===" # use echo, not log for clean output to be scraped
|
||||
git diff HEAD
|
||||
exit 1
|
||||
|
||||
|
||||
Reference in new issue
Block a user