Static SDK installation script improvements
### Motivation:
Make static SDK installation scripts more modular and make
failures easier to diagnose.
### Modifications:
- Allow specification of binaries from outside the installation script
- Remove script dependency installation from the script
- Separate the Swift dependency installation into its own script
- Fatal error explicitly if the installed Swift can't be found
- Make curl operations for downloading artifacts explicitly fail
### Result:
Scripts which are easier to maintain and operate.
An example CI run with this in operation
https://github.com/apple/swift-nio/actions/runs/14329482180/job/40162003318?pr=3186
### Motivation
A few repositories have custom workflows for checking CMake, we would
like to unify this for maintenance and adoption simplicity.
Further to this at the moment they only check that the scripts are
up-to-date, we'd also like to have them build using CMake.
### Modifications
Add the re-usable workflow and supporting scripts
### Result:
Repositories using CMake can switch to the centralised workflow.
An example of it in action
https://github.com/apple/swift-crypto/actions/runs/13837092514/job/38782284575
### Motivation:
The static SDK workflow currently assumes a local setup script which
isn't true for external adopters.
### Modifications:
* The static SDK workflow now curls for the setup script.
* Run Static SDK checks on PR commits
### Result:
External adopters should be unblocked.
A re-usable static SDK compilation workflow
### Motivation:
We would like to offer the ability to opt-in to CI coverage for
compilation against the Linux static SDK.
### Modifications:
Introduce a workflow which will install a Swift snapshot and Linux SDK
and test the repository builds against it.
The script which sets up the VM will install snapshots and SDKs for
either branch or released-version snapshots. They can be specified as:
- a specific version string
- the 'latest' version, the script attempts to find the most recent
version enumerated on swift.org with both SDK and toolchain available
- the most recent snapshot for the specified branch name enumerated on
swift.org with both SDK and toolchain available
At the moment the script should work on ubuntu and amazonlinux OSes,
this can be expanded in the future.
The current matrix covers testing building on Ubuntu Jammy for the main
branch and latest released Swift version (6.0.3) and enables it on our
periodic CI runs.
See
https://github.com/apple/swift-nio/actions/runs/13722773053/job/38381907947?pr=3019
for a recent example of this workflow in action.
### Result:
Increased CI coverage
Use nightly_next as swift version
see https://github.com/apple/swift-nio/pull/3122
Motivation:
To not have to rename threshold directories when the nightly branch
changes.
Modifications:
* Use nightly_next as swift version in the matrix generation script
which
is picked up by the benchmark script.
* Move nightly-next thresholds and add legacy symlink
Result:
Benchmark thresholds will attempt to find directories named
nightly_next, not nightly_6_1.
Update generate matrix stript to use nightly_next
### Motivation:
Updating the CI parameters for changing nightly job targets with each
new Swift version (6.0, 6.1, ...) is onerous. Whilst this gives us some
value in the non-nightly pipelines by forcing adopters to apply thought
to what is applied to each supported version, the nightlies arguably
deliver less value in this regard as they tend to be non-required.
### Modifications:
Change flags to refer to nightly_next rather than nightly_6_1, implement
a shim so that the old environment variables still work.
### Result:
We will be able to modify adopter workflows to use nightly_next
terminology.
Example of a job where workflows are using the new matrix
https://github.com/apple/swift-nio/actions/runs/13520016728
### Motivation:
We recently added support for nightly 6.1 Swift and old scripts making
use of `check-matrix-job.sh` do not set `COMMAND_OVERRIDE_NIGHTLY_6_1`
which causes the script to fail. We should tolerate this.
### Modifications:
Add a default case for `COMMAND_OVERRIDE_NIGHTLY_6_1`.
### Result:
No failures if a caller doesn't specify `COMMAND_OVERRIDE_NIGHTLY_6_1`.
```
❯ SWIFT_VERSION=nightly-6.1 COMMAND="echo foo" COMMAND_OVERRIDE_5_9="" COMMAND_OVERRIDE_5_10="" COMMAND_OVERRIDE_6_0="" COMMAND_OVERRIDE_NIGHTLY_MAIN="" ./scripts/check-matrix-job.sh
** Running default command
foo
❯ SWIFT_VERSION=nightly-6.1 COMMAND="echo foo" COMMAND_OVERRIDE_5_9="" COMMAND_OVERRIDE_5_10="" COMMAND_OVERRIDE_6_0="" COMMAND_OVERRIDE_NIGHTLY_MAIN="" COMMAND_OVERRIDE_NIGHTLY_6_1="echo bar" ./scripts/check-matrix-job.sh
** Running nightly 6.1 command override
bar
```
### Motivation:
Some changes were missed in #3076, passing through parameters for the
new 6.1 version.
### Modifications:
Pass through more parameters, clean up misleading comments.
### Result:
6.1 nightly runs will be more like 6.0 nightly runs were
### Motivation:
Active development has moved to 6.1 so the Swift 6.0 nightlies no longer
deliver much value.
### Modifications:
* Deprecate 6.0 nightly CI configuration parameters
* Introduce new 6.1 nightly CI configuration parameters
* Update the matrix generation script to handle 6.1
### Result:
* No more nightly 6.0 CI, 6.1 nightlies are introduced.
* Configuration for 6.0 nightlies no longer has any effect.
* All repos are automatically opted-in to 6.1 nightly linux pipelines
**even if they opted out of 6.0**.
### Motivation:
The use of 'swift' here is very arbitrary and adds no value. The object
is only there at all because GitHub Actions seems to require it.
### Modifications:
Rename the top-level object from 'swift' to 'config' to maybe add value
to the intent of the object and hopefully be less cryptic.
### Result:
Everything should just keep working.
Introduce and adopt a new method for defining test matrices,
`swift_test_matrix.yml`.
⚠️ Any external adopters of the unit tests, Cxx interop and benchmarks
workflows are automatically opted in to use the new infrastructure.
### Motivation:
* The current matrix workflow has the limitation that it only supports
pre-defined sets of variables which are explored in the test matrix. At
the moment this is a pre-defined set of Swift versions on Linux and
Windows.
* Adding more means hard-coding them at multiple levels of the workflow
hierarchy.
* Currently skipped Windows matrix jobs show up as successes in the
GitHub UI leading to a misleading impression of good coverage.
### Modifications:
Introduce and adopt a new method for defining test matrices,
`swift_test_matrix.yml`. The new method is based around the approach of
defining the test matrix via a JSON object which may be supplied via an
input string from another workflow or from a file on-disk in a
repository.
Taking this approach means that we have the ability to add new targets
to the matrix simply by adding new elements to the JSON array,
increasing flexibility and the scope for future growth.
The unit tests, Cxx interop and benchmarks workflows are all modified to
use the new approach, this opts-in all downstream adopters. This should
be transparent to all downstream adopters.
In order to unify the Linux and Windows jobs I removed the use of the
`container:` GitHub Actions affordance in the Linux jobs which
transparently means all steps are executed within the nested container.
Instead we must manually call in to the docker container which
complicates scripting a little. I tested to see if doing this slowed
down the jobs (perhaps GitHub was caching the docker images more
intelligently) but it does not.
This approach follows the pattern of @FranzBusch 's open PR
https://github.com/apple/swift-nio/pull/2942
### Result:
* More flexible test matrix definitions
* No more false-passes for disabled Windows targets
Cxx interop CI appends `swiftSettings` to any existing settings rather
than overriding them.
### Motivation:
This better accommodates workflows for example which use Swift 5
language mode on a per-target basis.
### Modifications:
Cxx interop CI appends `swiftSettings` to any existing settings rather
than overriding them.
### Result:
Cxx interop CI jobs don't cause unexpected build configurations.
### Motivation:
At the moment the Cxx interoperability CI workflow doesn't actually test
Cxx interoperability.
### Modifications:
The script now looks for the precise way that the `Package.swift`
currently formats the target entry and appends a `swiftSettings` entry
to it.
This approach is pretty brittle but has the advantage that it's the same
on Linux and Darwin. If this formatting changes too often then we could
give up on this and assume the availability of gnu-sed instead (and make
sure it's present on our CI containers).
### Result:
All Cxx interoperability checks will check what they are named for.
---------
Co-authored-by: George Barnett <gbarnett@apple.com>
Motivation:
netstat -s can identify many system-wide issues that may be useful extra
data. There's no reason not to collect it.
Modifications:
Add netstat -s to the nio-diagnose script.
Result:
Better diagnostics
# Motivation
In the long term we want to add support for Windows in our Swift
packages. To do this we need a CI setup to ensure that our packages are
building and the tests are passing.
# Modification
This PR adds a windows job to our swift matrix that installs Swift on
the runner and then uses Swift PM to build and test the package. Windows
jobs are disabled by default since most of our packages require some
work to add Windows support.
# Result
We have a Windows CI pipeline that allows us to verify that our packages
are working.
# Motivation
Our support strategy includes the last three released Swift versions.
After the recent release of Swift 6 this means we now support 5.9, 5.10
and 6.0.
# Modification
This PR updates our CI configuration and scripts to align with our
supported Swift versions.
# Result
Up to date CI coverage.
# Motivation
Switch to the reusable workflows from the swiftlang organization for
soundness.
# Modification
This PR moves over to the new reusable soundness workflow. Additionally,
it also deletes all files in this repo for the soundness checks since
they are no longer needed.
# Motivation
Some of repos have `.in` and `.cmake` files including header license
comments.
# Modification
This PR adds support to the license check for those type of files.
# Result
Even more license headers.
…and remove docc plugin
# Motivation
Some repositories might want to run their soundness and matrix jobs
checks on different container images e.g. a repo might build against the
nightly toolchains only so their soundness jobs should also use nightly
images. Other repos might want to build against a different linux
distribution since their usual deployment target is not Ubuntu based.
Additionally, we currently require our repos to depend on the docc
plugin for building documentation which adds an unnecessary dependency
to all of our packages.
# Modification
This PR makes the container images for the soundness and matrix jobs
configurable. Furthermore, it modifies the docs check script to add the
docc-plugin dependency and removes the current dependency from our
manifest.
# Result
Our workflows can be adopted by more repositories.
# Motivation
We have quite a lot of shell scripts in our repo and want to make sure that they all pass `shellcheck`.
# Modification
This PR adds a GH action workflow to the soundness script for `shellcheck` and fixes up all errors and warnings.
# Result
No more shell/bash discussions
# Motivation
Some repositories may want to exclude files from being format checked.
# Modification
This PR moves the formatting into a separate script and adds an optional `.swiftformatignore` file that can be placed at the repo root similar to the `.licenseignore` file.
# Result
Repos can now decide what files are ignored when running the formatter.
# Motivation
Another reusable check is to make sure that all library products of a package are successfully building when consumed from a module that has Cxx interoperability enabled. Another check that's missing is running the integration tests.
# Modification
This PR adds two new checks to the reusable workflow. One to check for Cxx interoperability compatibility and another one to run the integration tests. I also fixed a misalgined name for the nightly benchmarks.
# Result
This should be one of the last reusable workflow checks.
* [GHA] Introduce reusable matrix workflow
# Motivation
We have a few checks that we want to run across all supported Swift versions e.g. unit tests and benchmarks. Right now we already added two matrix based jobs to the reusable workflow. This leads to a lot of duplication and harder maintenance.
# Modification
This PR splits off the current reusable workflow to only contain soundness related checks and introduces a new reusable matrix based workflow which can be customized to execute different commands.
# Result
It is now easy to setup multiple matrix based workflows.
* Code review
* Change nightly next
# Motivation
The next two reusable checks that we need to create are for broken symlinks and formatting. The latter is disabled for this repo for now since we need to discuss the formatting rules separately.
# Modification
This PR adds two new checks - broken symlinks and formatting.
# Result
Closer to having everything on GitHub actions.
* [GHA] Add license header check
# Motivation
We need to make sure all code files have the appropriate license headers in place. This is currently part of the soundness check and we need to replace this with a GH action.
# Modification
Adds a new job to the reusable workflow the check all files for license headers. Since some files are ignored every repo can specify a `.licenseignore` file.
# Result
Last part of the soundness script migrated.
* Review
* Remove default excludes
# Motivation
For longer scripts we want to use a standalone shell script instead of using a multiline script inside the workflow yml.
# Modification
This PR uses the existing doc script instead of an inline script in the workflow
# Result
Better maintainability of the scripts.
* [GHA] Unacceptable language check
# Motivation
Next up replacing part of our soundness script that checked for unacceptable language.
# Modification
This PR adds a new GH action that checks for unacceptable language.
# Result
One more script replaced
* PR review
* Change homepage link to repo
* Introduce `assumeIsolated()` methods on `EventLoop`, `EventLoopPromise` and `EventLoopFuture`
> All methods/types are currently `internal` so we don't have to bikeshed just yet but we can move forward to get `NIOCore` warning free under strict concurrency
# Motivation
Methods on the above types are often called from the same event loop; however, we cannot prove to the compiler that this is true so we had to mark many methods on those types with `@Sendable` or require the generic type to be `Sendable`. This leads to unnecessary usage of `NIOLoopBound` when instead we should just dynamically assert that we are on the event loop. @dnadoba opened a very similar PR https://github.com/apple/swift-nio/pull/2228.
# Modification
This PR provides a method called `assumeIsolated()` on the three types that returns a type which re-declaration of all methods of the wrapped typed that have `Sendable` annotations. This new type is asserting at runtime that we are on the right event loop; hence, we don't need a `Sendable` value.
# Result
This PR makes it easier for our adopters to avoid newly introduced `Sendable` warnings
* Review
Motivation:
`scripts/analyze_performance_results.rb` has `-o markdown`. But the output that it currently generates isn't compliant. It outputs:
`|--|--|...` as the header separator. There need to be three or more `-` in that (or `:` replacing one of the dashes).
I've also put the test names in code blocks, this improves the rendering slightly.
Modifications:
* Fix header separator output in markdown
* Put the test names in code blocks
Result:
Markdown output renders properly in standard markdown viewers (github, MacDown).
Motivation:
Up until recently, it has not been possible to regression check our
documentation. However, in recent releases of the DocC plugin it has
become possible to make warnings into errors, making it possible for us
to CI our docs.
This patch adds support for doing that, and also cleans up our
documentation so that it successfully passes the check.
Along the way I accidentally wrote an `index.md` for `NIOCore` so I
figure we may as well keep it.
Modifications:
- Structure the documentation for NIOCore
- Fix up DocC issues
- Add `check-docs.sh` script to check the docs cleanly build
- Wire things up to our docker-compose scripts.
Result:
We can CI our docs.
Co-authored-by: George Barnett <gbarnett@apple.com>
Motivation:
SwiftNIO periodically drops support for older Swift versions. Now that
5.7 has been released, 5.4 will be dropped.
Modifications:
- Remove 5.4 specific Package.swift and docker-compose
- Update the 5.7 docker-compose to use the released 5.7 and move from
focal (2004) to jammy (2204)
- Remove unused swiftformat from Dockerfile
- Update tools version in syscall wrapper tests to 5.5
- Update docs
Results:
Minimum Swift version is 5.5
Motivation:
The node.js HTTP parser library that we use has been unmaintained for some time. We should move to the maintained replacement, which is llhttp. This patch will update our dependency and bring us over to the new library, as well as make any changes we need.
Modifications:
This patch comes in 4 parts, each contained in a separate commit in the PR.
The first commit drops the existing http_parser code and updates some of the repo state for using llhttp.
The second commit rewrites the update script to bring in llhttp instead of http_parser.
The third runs the actual script. You can skip reviewing this except to sanity check the outcome.
The fourth commit updates the NIO code and the tests to get everything working.
In general the substance of the product modifications was minimal. The logic around keeping track of where we are in the buffer and how upgrades work has changed a bit, so that required some fiddling. I also had to add an error reporting path for the delegates to be able to throw specific errors that llhttp no longer checks for. Finally, I removed two tests that were a little overzealous and that llhttp does not police.
Result:
Back on the supported path.
* initial adoption of DocC based documentaiton
motivation: use DocC for docs
change:
* add DocC catalog to NIO target, as the "landing page"
* remove jazzy doc generation script
* add Package.swift with tools-version 5.6 so that DocC can be used, and add a dependency on the DocC plugin
* udpate soundness script
* * add packge.swift files for older vrsions
* adjust waning-as-error and enable-test-discovery to the different permutations
* fixup
* remove jazzy
Co-authored-by: Cory Benfield <lukasa@apple.com>
Motivation:
Cocoapods appears to have a different idea of the dependency graph to
SPM which has led to a handful of build failures (e.g.
https://github.com/apple/swift-nio/issues/2073).
This appears to have originated when we dropped the explicit dependency
on `CNIOAtomics` from `NIO` (https://github.com/apple/swift-nio/pull/1719).
We can work around this by listing all transitive dependencies as
requirements in the generated podspecs.
Modifications:
- Add a script to list transitive dependencies for a given module.
- Fix a bug in build_podspecs.sh which did a `pod repo update` even if
the pods were not being uploaded.
- Update build_podspecs.sh to use transitive dependencies.
- Use Python3 in 'dev/stackdiff-dtrace.py' and update soundness.sh
Result:
Podspecs should include all transitive dependencies.
* Drop support for 5.2 and 5.3
As outlined in a [Swift forums post in November ’21](https://forums.swift.org/t/swiftnio-swift-version-support/53232), SwiftNIO will only support the latest non-patch Swift release and the 2 immediately prior non-patch versions.
In this commit we drop support for Swift 5.2 and 5.3. We update CI for Swift 5.4 to run on bionic instead of focal to ensure that we still test bionic.
* Added Versions paragraph to Security document
* Apply suggestions from code review
Co-authored-by: Cory Benfield <lukasa@apple.com>
Co-authored-by: Cory Benfield <lukasa@apple.com>
### Motivation:
SPM has built in functionality to check the API of modules against a target git treeish. We can use this to simplify our `check_no_api_breakages.sh` script. Closes https://github.com/apple/swift-nio/issues/1239
### Modifications:
This PR, exchanges the direct calls to Swift's API checker with the new SPM `diagnose-api-breaking-changes` tool. This allows us to get rid of the manual module parsing, build invocations and result comparisons.
### Result:
We are now using SPMs `diagnose-api-breaking-changes` to check for breaking changes.