Commit Graph
42 Commits
Author SHA1 Message Date
Raghav RoyandCory Benfield 12d8a4c6d7 Add Peek API for Multiple Integers (#3175)
### Motivation:

The current readMultipleIntegers methods are used to consume multiple
integer values at once, but lack an equivalent nonmutating API. This can
lead to unnecessary complexity when users want to inspect values without
advancing readerIndex. THis patch introduces peekMultipleIntegers which
uses the current readerIndex, improving safety. This extends previous
work on peek methods (issue #2034, issue #2736).

### Modifications:

Updated the generation script (.sh file) to produce peekMultipleIntegers
variants for each arity of readMultipleIntegers.
    Added new tests mirroring the existing multi-read/write tests

### Result:

Users can now inspect multiple integer values in a ByteBuffer without
consuming them, enabling safer workflows.

---------

Co-authored-by: Cory Benfield <lukasa@apple.com>
2025-04-10 10:23:27 +00:00
Rick Newton-Rogers 771810f5f8 A script for updating benchmark thresholds from CI output (#3151)
### Motivation:

We have GitHub Actions CI running Swift Package Benchmark benchmarks on
each commit but when the values don't match updating them can be time
consuming. Particularly because you need to may need to exactly match
the running environment in CI to get matching numbers.

### Modifications:

Added a new dev script which pulls the output for the benchmark GitHub
actions runs on a PR and updates your local threshold files

### Result:

Easier benchmark threshold updates

Example interaction (actually on my branch of NIO SSL where I've messed
with the thresholds so they need an update):
```
❯ FETCH_PR_URL=https://github.com/rnro/swift-nio-ssl/pull/5 OUTPUT_DIRECTORY=Benchmarks/Thresholds/ ../swift-nio/dev/thresholds-from-benchmark-output.sh
** Updating: Benchmarks/Thresholds//nightly-main/NIOSSHBenchmarks.SimpleHandshake.p90.json job:39048331461
** Nothing to update: rnro/swift-nio-ssl 5.10 job:39048331486
** Nothing to update: rnro/swift-nio-ssl 5.9 job:39048331454
** Nothing to update: rnro/swift-nio-ssl 6.0 job:39048331469
** Nothing to update: rnro/swift-nio-ssl nightly-next job:39048331466
2025-03-21 16:17:14 +00:00
Cory Benfield 6ac01d1734 Clean up sendability in the bootstraps (#3051)
### Motivation:

The bootstraps are the next most obvious target for cleaning up
Sendability issues. These issues are mostly just missing annotations,
but there are a few places where we had actual latent threading bugs
that were missed. A lot more of the control flow is made more explicit
in this patch, and in general it should get a lot easier to be confident
that the code is correct.

### Modifications:

- Add necessary `@Sendable` annotations
- Clean up some incorrect `self` captures of bootstraps, which are not
`Sendable`, and which could lead to real threading bugs
- Make a few methods `static` to avoid needing to capture `self` at all.
- Make a few threading assumptions clear by using the isolated views or
the sync operations
- Add some missing `Sendable` constraints on interior generic functions.

### Result:

Better safety, better correctness in the bootstraps.
2025-01-14 20:49:33 +00:00
George Barnett 100b1f4ff9 Print alloc-limits-from-test-output usage when no args (#3041)
Motivation:

The alloc-limits-from-test-output doesn't print usage if you don't pass
in a format arg, it just sits and waits for stdin.

Modifications:

Print usage if running interactively

Result:

Easier to get usage
2025-01-07 10:36:03 +00:00
Cory Benfield 035d3a57c9 Add back alloc limits script with JSON mode (#2987)
Motivation:

We removed the alloc limits script, but it is still very useful. It just
couldn't produce JSON. So now it can!

Modifications:

Re-adds the alloc limits script.
Makes it produce JSON.

Result:

JSON can be produced from the CI output.
2024-11-21 16:55:29 +00:00
Peter AdamsandCory Benfield e25bf7ecfe Remove unused dockerfiles and allocation limit scripts (#2957)
Motivation:

Remove unused clutter

Modifications:

Remove dockerfiles and scripts to update allocation limits.

Result:

Cleaner repo

Co-authored-by: Cory Benfield <lukasa@apple.com>
2024-10-31 15:59:53 +00:00
Franz Busch 290b349e2c [CI] Fix python lint (#2925)
# Motivation

Fix the python lint check.

# Modification

This PR fixes all the warnings related to the new python lint check.

# Result

Only green CI.
2024-10-17 09:57:40 +00:00
Franz Busch f17f7e5909 [CI] Add shellcheck and fix up warnings (#2809)
# Motivation

We have quite a lot of shell scripts in our repo and want to make sure that they all pass `shellcheck`.

# Modification

This PR adds a GH action workflow to the soundness script for `shellcheck` and fixes up all errors and warnings.

# Result

No more shell/bash discussions
2024-07-25 14:55:44 +00:00
Si Beaumont f938b6a19c Fix compose file used in update-benchmark-thresholds script (#2808) 2024-07-25 10:52:14 +01:00
Franz Busch 7b03cb247b Fix benchmark thresholds update script (#2783)
# Motivation

We added new CI to run both nightly main and nightly next but forgot to update the benchmark thresholds script

# Modification

This PR updates the benchmark thresholds script and adds a nightly next docker compose image.

# Result

We can use the update benchmark thresholds script again.
2024-07-13 02:51:05 -07:00
Franz Busch e4102aecb5 Remove @unchecked Sendable conformance from ChannelOptions.Storage (#2638)
# Motivation
We currently have an unchecked `Sendable` conformance on `ChannelOptions.Storage` which is not necessary and in fact we are missing a proper `Sendable` requirement on the `ChannelOption.Value` protocol right now.

# Modification
This PR adds the `Sendable` requirement on the `ChannelOption.Value` and replaces the unchecked conformance on the storage with a regular one.

# Result
No more `Sendable` warnings in strict concurrency mode inside `ChannelOptions`.
2024-02-12 09:48:54 +00:00
Gustavo Cairo 0c648c91c6 Fix broken tests (#2621)
* Update swift-tools-version to 5.7 in integration tests

* Fix flaky test

* Add missing AtomicCounter dependency in generated Package.swift

* Update swift-tools-version in integration test helper Swift projects

* Make test clearer

* Don't run MTELG integration test on Swift 5.11+

It hasn't been properly tested
2024-01-08 10:02:09 +00:00
Franz Busch d782de84df Adopt package-benchmark (#2534)
# Motivation
We want to migrate our allocation and later on also our performance tests to use the `package-benchmark` plugin. This plugin makes writing benchmarks way easier than our current setup. Furthermore, debugging benchmarks is also possible from within Xcode now.

# Modification
This PR adds the setup for the benchmarking infrastructure and connects it with out

# Result
Allocations tests are more accessible and easier to iterate.
2023-10-08 13:31:22 -07:00
Felix Schlegel a22a35a0ff Update update-alloc-limits script (#2430)
Motivation:

Running the script failed as it looked for a CI build for Swift 5.5,
which is not run by our CI anymore.

Modifications:

* don't look for CI builds for Swift 5.5
* additionally, also check for CI build for Swift 5.9
2023-05-19 09:22:53 -07:00
George Barnett 4dfae01cc6 Add a pooled recv buffer allocator (#2362)
Motivation:

Channels can read `ChannelOptions.maxMessagesPerRead` times from a
socket in each read cycle. They typically re-use the same buffer for
each read and rely on it CoWing if necessary. If we read more than once
in a cycle then we may CoW the buffer. Instead of reusing one buffer we
can reuse a pool of buffers limited by `maxMessagesPerRead` and cycle
through each, reducing the chance of CoWing the buffers.

Modifications:

- Extend `RecvByteBufferAllocator` to provide the size of the next
  buffer with a default implementation returning `nil`.
- Add an recv buffer pool which lazily grows up to a fixed size and
  attempts to reuse buffers where possible if doing so avoids CoWing.

Results:

Fewer allocations
2023-02-20 17:00:19 +00:00
Cory Benfield 0760383b38 Measure allocations applying WS mask (#2333)
Motivation:

We're missing some alloc tests on the WS mask.

Modifications:

- Add extra alloc tests for the WS mask

Result:

We'll have more tests
2022-12-13 18:05:27 +00:00
Cory Benfield b1b44ca67b Reduce the alloc limits for main (#2095)
The nightly builds have improved, so the alloc limits can drop slightly
2022-05-04 15:49:40 +01:00
George Barnett f32314f82f Script to list transitive dependencies (#2082)
Motivation:

Cocoapods appears to have a different idea of the dependency graph to
SPM which has led to a handful of build failures (e.g.
https://github.com/apple/swift-nio/issues/2073).

This appears to have originated when we dropped the explicit dependency
on `CNIOAtomics` from `NIO` (https://github.com/apple/swift-nio/pull/1719).

We can work around this by listing all transitive dependencies as
requirements in the generated podspecs.

Modifications:

- Add a script to list transitive dependencies for a given module.
- Fix a bug in build_podspecs.sh which did a `pod repo update` even if
  the pods were not being uploaded.
- Update build_podspecs.sh to use transitive dependencies.
- Use Python3 in 'dev/stackdiff-dtrace.py' and update soundness.sh

Result:

Podspecs should include all transitive dependencies.
2022-04-20 14:39:43 +01:00
Cory Benfield dde891c624 Update alloc limits for nightly. (#2071)
We regressed allocs in nightly, probably for a Swift bug. Fixing it is
tracked in #2070, for now we're going to raise the limits so CI isn't
moaning all the time.
2022-03-23 10:05:11 -07:00
Fabian FettandCory Benfield c74c3bbabf [SelectableEventLoop] SR-15872; Save allocations when appending task (#2046)
* [SelectableEventLoop] SR-15872; Save allocations when appending task

* Add pr to workaround list

* Update alloc counters

* Update alloc script to include 5.6

* Update alloc counters

Co-authored-by: Cory Benfield <lukasa@apple.com>
2022-02-17 12:18:21 -08:00
Johannes Weiss b2629903ca ByteBuffer: provide multi read/write int methods (#1987)
Motivation:

Many network protocols (especially for example NFS) have quite a number
of integer values next to each other. In NIO, you'd normally parse/write
them with multiple read/writeInteger calls.

Unfortunately, that's a bit wasteful because we're checking the bounds
as well as the CoW state every time.

Modifications:

- Provide read/writeMultipleIntegers for up to 15 FixedWidthIntegers.
- Benchmarks

Result:

Faster code. For 10 UInt32s, this is a 5x performance win on my machine,
see benchmarks.
2021-11-22 14:58:52 +00:00
David Evans b0effbcfd9 Make Swift 5.2 the minimum requirement (#1860)
Make Swift 5.2 the minimum requirement, dropping support for Swift 5.0 and 5.1.

Motivation:

Whenever we have problems, Swift 5.0 and 5.1 seem to be the culprits. Dropping support for these very old versions will require less maintenance and free up our time to work on new features.

Modifications:

Set the tools version in Package.swift to 5.2
Remove CI configurations for 5.0 and 5.1
Update the various readmes to reflect that this change will be rolled out in NIO 2.30.0
Result:

Swift 5.2 is the minimum version of Swift required to use NIO.
2021-06-18 21:02:41 +01:00
Johannes Weiss b23f40b4fa add Swift 5.5 job (#1845) 2021-06-01 18:58:56 +01:00
Johannes Weiss 06195d2f46 dev/scripts: add missing license headers (#1852)
Motivation:

We need license headers but the soundness script couldn't check if the
scripts don't have a `.sh` extension.

Modification:

- Add missing license headers.
- Better license checking (search for shell shebangs too)

Result:

Proper license attribution.
2021-05-07 10:28:07 +01:00
Johannes Weiss c51907f551 alloc limit updater: allow other repos (#1844)
Motivation:

The allocation limit updater script is handy but so far only worked for
the SwiftNIO core repo.

Modifications:

- allow the user to pass other URL prefixes and repos

Result:

Can be used for other repos
2021-05-04 19:15:38 +01:00
Johannes Weiss 0504ff04dd dev/lldb-smoker: LLDB smoke test using NIO (#1840)
Motivation:

We often hit a bunch of problems with LLDB, especially on Linux. So
maybe it would be cool if we could smoke test LLDB using SwiftNIO.

Modifications:

Add script `dev/lldb-smoker` which does the following things:

- takes 2000 random picks amongst all the files in the package
- for each of those files, pick a random line number
- write an LLDB script which sets break points for each of the file/line
  pairs (one-shot breakpoints)
- runs the program
- for each breakpoint hit, run `frame variable` to list all the
  variables

Result:

LLDB smoke test
2021-05-04 11:05:40 +01:00
Cory Benfield 8369411d6f Stop measuring alloc counts on 5.0 (#1826)
Motivation:

5.0 is several years old now, and while we still support it there is
beginning to be some conflict with performance optimizations for newer
Swift versions. We should stop counting the allocation counts on this
version now: we no longer care as much about regressions.

Modifications:

- Stop setting alloc counter limits on 5.0
- Remove the script that tries to add them back.

Result:

No longer measure allocs on 5.0
2021-04-28 14:02:13 +01:00
Johannes Weiss eb1590a48a script to download & apply alloc limits from Jenkins (#1817)
Motivation:

When doing bigger alloc count changes, it's really tedious to apply them
manually given that the CI has the right numbers already.

Modifications:

Add a script to download and update them automatically.

Result:

Less toil.
2021-04-23 17:54:52 +01:00
Johannes Weiss 4d1a72c712 alloc-limits-from-test-output: support more formats (#1812)
Motivation:

The dev/alloc-limits-from-test-output script is handy to get correctly
formatted (and sorted) alloc counter limits from test output.

Modifications:

When running the alloc counter script itself, it outputs suggested
`export MAX_ALLOCS_ALLOWED` lines. We're now also parsing those, so you
can easily go from `export ...*` to the docker-compose form :).

Result:

Even easier alloc counter limits.
2021-04-21 14:39:55 +01:00
Johannes Weiss 2d27890eb8 normalise alloc counter slack (#1789)
Motivation:

Usually, we add a more or less random number of slack allocations to
make sure the tests don't spuriously fail. This makes it quite costly to
support new Swift versions.

Modifications:

Add a script which can spit you out the right allocation limits
including slack.

Result:

Easier to support new Swift versions
2021-03-30 14:42:33 +01:00
George Barnett 990b82d698 Make stackdiff-dtrace.py a little more tolerant (#1771)
Motivation:

Some of the output from 'malloc-aggregation.d' trips up the
'stackdiff-dtrace.py' script -- especially when used with the output
from recent 5.4 builds. The regex for numbers includes hex which trips
us up when we try to convert that match to an int. Since we only use
this pattern for matching allocation counts, where we don't expect hex,
we can just modify the pattern.

Also the input parsing is a little tricky to follow without context so I
tidied it up little.

Modifications:

- relax the number regex (we don't expect counts to be in hex)
- drop the stack regex (it wasn't used)
- aggregate the current stack in a list (instead of a string)
- filter out non-stack lines (which were erroneously used to key the
  first stack!)
- a few comments

Result:

stackdiff-dtrace.py doesn't blow up on output from newer builds
2021-03-02 12:26:52 +00:00
Johannes WeissandCory Benfield 3a2fc0d39b alloc tests: implement malloc_zone_* (#1569)
Motivation:

The Swift runtime is now using malloc_zone_*, we need to implement
replacements for these too. This is just a first pass, eventually, we
should implement _all_ replacements as `malloc_zone_memalign` which is
powerful enough to implement all others.

Modifications:

Provide new replacements.

Result:

Alloc tests work again on macOS.

Co-authored-by: Cory Benfield <lukasa@apple.com>
2020-06-26 11:04:37 +01:00
Peter Adams c5fa0b4565 Allocation diffing script no longer drops data. (#1513)
Allocation diffing script no longer drops data.

Motivation:

Looking at allocations is hard enough without the script getting the allocation counts wrong.

Modifications:

Where multiple allocations resolved to the same "key" only the last allocation was accounted for. I have changed the script to store all allocations - still keyed in the same way. When diffing the total number of allocations is used - if they are different the total and all contributing stack traces are output.

Added a total for all allocations everywhere at the end together with a difference number.

Output all stack traces before and after contributing to a detected diff.

Update the documentation to reflect changes.

Change the threshold for reporting diffs from > 1000 to >= 1000. This means if allocations go from 1000 to 2000 they are reported as a difference rather than a new allocation.

Result:

It is now easier - if somewhat more verbose to compare allocations.
2020-05-12 09:11:56 +01:00
Johannes Weiss ad0c671acf make-single-file-spm: better directory name (#1446)
Motivation:

The make-single-file-spm script used to use a hidden folder in /tmp
which isn't very helpful when tar'ing up the result for a bug report.

Modifications:

Switch to a non-hidden folder.

Result:

Easier bugs.swift.org reports.
2020-03-13 17:01:37 +00:00
George BarnettandJohannes Weiss ddb7109142 Include stackdiff-ing in the debugging-allocations doc (#1384)
Motivation:

We recently grew a script to diff the output of `malloc-aggregation.d`,
we should document how this can be used when debugging allocation
regressions.

Modifications:

- Update the doc to include an example of using the script to debug an
  allocation regression
- Fix a bug in the script where before/after were mixed up
- Fix a bug in the script where usage would print '()\n' instead of '\n'

Result:

- Better info on debugging allocations

Co-authored-by: Johannes Weiss <johannesweiss@apple.com>
2020-02-06 16:10:04 +00:00
Johannes Weiss a76a98eb00 dev/stackdiff-dtrace: stack aggregation diffing script (#1373)
Motivation:

NIO has allocation counter tests, NIO also has dev/malloc-aggregation.d
which can give you aggregated stacks that allocated alongside how many
times they allocated. What's missing is a script that can diff two
aggregated stack traces.

Modifications:

Add dev/stacktrace-dtrace which allows aggregated stack trace diffing.

Result:

Easier to track performance regressions.
2020-02-03 09:09:20 +00:00
Johannes Weiss 9dd9f36bfd make-single-file-spm: support lines ending in n (#1251)
Motivation:

make-single-file-spm has a weird bug where it would chop off trailing
`n` characters in all lines due to a badly set `IFS` bash variable.

Modifications:

Set `IFS=""` which is correct.

Result:

make-single-file-spm will not chop off trailing `n`s.
2019-11-18 15:24:18 +01:00
Johannes Weiss 1c160afa95 add make-single-file-spm (#1236)
Motivation:

make-single-file-spm is a simple bash script that can turn a single file
into a SwiftPM project with multiple modules (all defined in a single
file with special markers such as `// MODULE: Foo`. This is mostly
useful for performance evaluation of Swift becuase in almost all cases,
you'll need multiple modules which is quite a bit of work to set up.

Modifications:

Add make-single-file-spm

Result:

More useful scripts in the NIO repo.
2019-11-12 19:07:01 +00:00
Johannes Weiss 9480c27148 malloc-aggregation.d: don't forget posix_memalign (#785)
Motivation:

When counting the number of allocations it's important not to forget an
allocation function that Swift uses.

Modifications:

add posix_memalign to dev/malloc-aggregation.d

Result:

malloc aggregations accurate with recent Swift versions
2019-01-29 11:23:42 +00:00
Johannes Weiß ec47847b4b add other memory allocation functions (#366)
Motivation:

The dtrace scripts dev/malloc-aggregation.d was missing a bunch of more
rarely used memory allocation functions.

Modifications:

Added the more rarely used functions.

Result:

Better output when using dev/malloc-aggregation.d
2018-04-27 10:29:25 +01:00
Johannes Weiß 5599bfe4c8 add useful dtrace scripts to debug allocations (#352)
Motivation:

dtrace is extrmely useful when debugging allocation related issues. This
adds two scripts that have helped me many times.

Modifications:

- `dev/malloc-aggregation.d` which prints an aggregation of all
  stacks that have allocated
- `dev/boxed-existentials.d` which live prints all allocations of boxed
  existentials

Result:

sharing debugging tools is great
2018-04-25 15:23:09 +01:00
Norman Maurer 3bfdd3f55b Adapt git template that is also used by Netty 2018-02-09 12:19:12 +01:00