### Motivation:
One of my project uses `NIOHTTP1TestServer` for testing and for
responses that contain the `content-length` header + `connection:
close`,
the test was flaky - failing intermittently, because the client closed
the connection before the test managed to submit `.end` to the test
server.
The cause was a race condition between the test and the client's and
server's threads hosting their respective event loops.
### Modifications:
Now `NIOHTTP1TestServer` captures the channel that sent the `.head`, and
even once `currentClientChannel` changes (but before the next `.head` is
sent),
when a call to send `.end` comes in, uses the channel associated with
the previous `.head`, rather than reading `currentClientChannel`.
This test is skipped on Darwin as it requires the platform to detect
early EOF.
### Result:
More reliable tests that involve two requests that come in at once (in a
deterministic order), when the second one is parked and ready right
after the first one finishes.
Create temporary directories with owner-only permissions.
### Motivation:
`FileSystem.createTemporaryDirectory(template:)` (and
`withTemporaryDirectory`, which uses it) creates the directory with
`FilePermissions.defaultsForDirectory`, i.e. 0755. When the template
points into a shared directory such as `/tmp`, other local users can
list the temporary directory and read anything written into it with the
default 0644 file permissions. `mkdtemp(3)`, Go's `os.MkdirTemp` and
Python's `tempfile.mkdtemp` all use 0700 for this reason.
Repro with a umask of 022:
```swift
let dir = try await FileSystem.shared.createTemporaryDirectory(template: "/tmp/nio-repro-XXXXXX")
try await FileSystem.shared.withFileHandle(forWritingAt: "\(dir)/secret", options: .newFile(replaceExisting: false)) { _ in }
// before: dir is drwxr-xr-x, secret is -rw-r--r--, readable by every local user
// after: dir is drwx------
```
### Modifications:
- Pass `.ownerReadWriteExecute` when `_createTemporaryDirectory` creates
the directory, in both `NIOFS` and `_NIOFileSystem`. Intermediary
directories created from the template get the same permissions, since
they are created by the same `_createDirectory` call.
- Document the permissions on `createTemporaryDirectory(template:)`.
- Add `testCreateTemporaryDirectoryIsOnlyAccessibleByOwner`.
### Result:
Temporary directories are only accessible by their owner.
I verified the before/after modes for both modules with a small
executable against this branch on macOS (`swift build` is clean). I
couldn't run the XCTest target locally because this machine only has the
Command Line Tools.
---------
Co-authored-by: Cory Benfield <lukasa@apple.com>
### Motivation:
We introduced a new configuration type in #3746 to make further
extensions to the server pipeline (#3745 for example) easier to
introduce.
The plan was to merge #3745 soon after #3746 was merged, because #3745
added a breaking change to the new config type.
However, #3745 needs some more consideration. We should therefore revert
the change in #3746 to unblock new releases from being made until we
come up with a better solution.
### Modifications:
A clean revert of PR #3746.
### Result:
Unblocks new releases.
Drop `@unchecked` from `TokenBucket`'s `Sendable` conformance.
### Motivation:
`TokenBucket` (in both `NIOFS` and `_NIOFileSystem`) is marked
`@unchecked Sendable`, but its only stored property is `private let
lock: NIOLockedValueBox<State>`, and `NIOLockedValueBox` is `Sendable`
when its value is (`State` holds an `Int` and a
`Deque<CheckedContinuation<Void, Never>>`). The compiler can verify this
conformance on its own, and keeping `@unchecked` means it would not
catch a future change that breaks sendability.
### Modifications:
- Replace `@unchecked Sendable` with `Sendable` on `TokenBucket` in
`NIOFS` and `_NIOFileSystem`.
### Result:
No behavior change; the conformance is now checked by the compiler.
Verified locally on macOS (Swift 6.4.2): `swift build --build-tests
-Xswiftc -strict-concurrency=complete` produces no new diagnostics, and
`NIOFSTests`, `NIOFSIntegrationTests` and `NIOFSFoundationCompatTests`
pass.
### Motivation:
`NIOHTTP1.HTTPPipelineSetup` contains helper methods which add HTTP/1
client/server handlers to the provided channel pipeline.
Over time, we have had to duplicate these methods when introducing a new
configuration option in order to not break API.
As a result, there is now lots of duplication in
`NIOHTTP1.HTTPPipelineSetup` and introducing more configuration options
is very cumbersome. We should instead add a configuration type to solve
this problem.
### Modifications:
- Added new configuration types named
`NIOHTTPServerPipelineConfiguration`, and
`NIOHTTPServerPipelineSynchronousConfiguration`.
- Added a new pipeline setup helper method on `ChannelPipeline` and
`ChannelPipeline.SynchronousOperations` that requires the configuration
type as an argument.
- Deprecated all existing helper methods on `ChannelPipeline` and
`ChannelPipeline.SynchronousOperations`.
### Result:
Better API.
## Motivation
`QUICBinaryEncodingStrategy.writeInteger(_:reservedCapacity:to:)`
switches on `max(reservedCapacity, bytesNeededForInteger(integer))` and
handles only 1, 2, 4 and 8, falling through to
`fatalError("Unreachable")` otherwise.
`bytesNeededForInteger` only ever returns those four values, but
`reservedCapacity` is guarded solely against being greater than 8. A
reserved capacity of 3, 5, 6 or 7 therefore reaches the unreachable case
and traps:
```swift
var buffer = ByteBuffer()
let strategy = ByteBuffer.QUICBinaryEncodingStrategy.quic
strategy.writeInteger(1, reservedCapacity: 3, to: &buffer)
// NIOCore/ByteBuffer-quicBinaryEncodingStrategy.swift:126: Fatal error: Unreachable
```
This is reachable without calling that method directly.
`requiredBytesHint` is a public `var Int` with no validation — the
initialiser takes the 1/2/4/8 `IntegerLength` enum, but the property can
be assigned any value afterwards, and
`writeLengthPrefixed(strategy:writeData:)` passes it straight through as
the reserved capacity:
```swift
var strategy = ByteBuffer.QUICBinaryEncodingStrategy(requiredBytesHint: .one)
strategy.requiredBytesHint = 3 // public var, accepted silently
var buffer = ByteBuffer()
buffer.writeLengthPrefixed(strategy: strategy) { $0.writeString("hi") }
// Fatal error: Unreachable
```
## Modifications
Round the length up to the next valid QUIC integer length rather than
trapping, which is what the surrounding comment already intends by "use
more space than necessary in order to fill the reserved space":
```diff
- case 4:
+ case 3, 4:
- case 8:
+ case 5, 6, 7, 8:
```
RFC 9000 § 16 permits encoding a value in more bytes than it strictly
requires, and `writeLengthPrefixed` already shifts the payload when the
integer turns out longer than was reserved (`case reservedCapacity...`),
so writing 4 bytes for a reservation of 3 is handled by existing code.
## Result
No trap for any `reservedCapacity` in `0...8`, and the encoded value
still round-trips.
I checked every combination of `reservedCapacity` in `0...8` against the
boundary values `0, 1, 63, 64, 16383, 16384, 1073741823, 1073741824,
4611686018427387903`: all 81 encode, decode back to the original value,
and leave no bytes behind. On `main` the same check traps.
## Testing
The existing `testRoundtripWithReservedCapacity` iterates over `[0, 1,
2, 4, 8]` — skipping exactly the values that trap, which is presumably
why this went unnoticed. I've added a companion
`testRoundtripWithNonPowerOfTwoReservedCapacity` covering 3, 5, 6 and 7,
asserting both the encoded length and that the value round-trips. It
fails on `main` and passes with this change.
Full test suite: **2455 tests, 0 failures.**
## A note on the alternative
Rounding *down* to the largest valid length that still fits the
reservation would also avoid the trap and waste fewer bytes — for
`reservedCapacity: 3` that would write 2 bytes rather than 4. I went
with rounding up because the existing comment says to use more space
rather than less, and because it keeps the guarantee that at least the
reserved number of bytes is written. Happy to switch if you'd prefer the
other behaviour.
If you would rather fix this at the source by validating
`requiredBytesHint` or changing its type to `IntegerLength`, that would
be source-breaking, so I left it alone — but the public setter is what
makes this reachable in ordinary use.
---------
Co-authored-by: Raphael Hiesgen <r_hiesgen@apple.com>
### Motivation:
Users may want to move a non-Sendable value into an EventLoopFuture.
This is safe if we take the value as `sending`
### Modifications:
Change functions whcih currently require Sendable parameters to instead
take the parameter as `sending` where possible. This seems to not break
API
Some functions take closures whcih return Sendable values, those ones
now take closures which return sending values. This has been done as an
overload, because a straight replacement seems to break API
### Motivation:
Stay up-to-date with the vendored llhttp version.
### Modifications:
* Update script to work with current releases.
* Run script to update vendored llhttp version.
### Result:
New llhttp version.
### Motivation
SwiftNIO provides convenience functions for a range of socket options.
This PR adds convenience options for fragmentation-related socket
options on Darwin and Linux. One particular use case is disabling
fragmentation when using QUIC, which disallows IP fragmentation of its
datagrams.
### Modifications
* Add `NIOBSDSocket.Option.ip_dontfrag` (Darwin).
* Add `NIOBSDSocket.Option.ipv6_dontfrag` (Darwin and Linux).
* Add `NIOBSDSocket.Option.ip_mtu_discover` and `.ipv6_mtu_discover`
(Linux). They set the path MTU discovery mode, which influences
fragmentation.
### Result
Adopters can easily configure IP fragmentation.
Corrected `compatability` to `compatibility` in the default
`RecvByteBufferAllocator` extension comment.
Co-authored-by: Cory Benfield <lukasa@apple.com>
### Motivation
Decoders are state machines and state machines are a natural fit for
non-copyable types. Today `ByteToMessageDecoder` and
`NIOSingleStepByteToMessageDecoder` require their conformers to be
copyable.
While we are here we also want `NIOSingleStepByteToMessageDecoder` to be
able to declare a concrete error type, so that callers can handle decode
errors without going through an existential.
### Changes
- Make `ByteToMessageDecoder` and `NIOSingleStepByteToMessageDecoder`
`~Copyable`.
- Add an `associatedtype DecodeError: Error = any Error` to
`NIOSingleStepByteToMessageDecoder` and spell decode and decodeLast as
`throws(DecodeError)`. Decoders that don't declare it keep behaving
exactly as before.
- Allow `~Copyable` `ByteToMessageDecoder` in `ByteToMessageHandler`.
- Since `~Copyable` types cannot be type erased into an existential
before macOS 15, we have to work around the existing `CanDequeueWrites`
dance: instead of casting `self` to `CanDequeueWrites` in
`handlerAdded`, ByteToMessageHandler gains an optional
`deliverQueuedWrite` closure. It is formed in the constrained
`ChannelOutboundHandler.write`, the only place where `Decoder`'s
`WriteObservingByteToMessageDecoder` conformance is statically known,
the first time a write arrives re-entrantly.
This has to be a closure rather than a protocol requirement on
`ByteToMessageDecoder`. `Decoders` may conform to
`WriteObservingByteToMessageDecoder` conditionally while conforming to
`ByteToMessageDecoder` unconditionally — `HTTPDecoder` does exactly this
— and all generic instantiations share one `ByteToMessageDecoder`
witness table, so such a requirement could only ever be witnessed by a
non-observing default and the write would be silently dropped.
- Split the delivery out of `tryDecodeWrites` into an `@inline(never)
dequeueWrites()`. `decodeLoop` calls `tryDecodeWrites` on every
iteration, so the empty-queue check has to stay cheap; keeping the
delivery inline tripled that function's size (50 → 141 instructions) and
showed up directly on the normal decode path.
- `ByteToMessageDecoderVerifier` allows `~Copyable`
`ByteToMessageDecoder`s.
- Added tests that use the newly lifted `~Copyable` restriction, plus
regression coverage for a decoder whose write-observing conformance is
conditional (the HTTPDecoder shape) and for constructing the handler in
a generic context.
### Result
Decoders may be non-copyable and may declare a concrete error type.
The normal decode path does not change performance. The re-entrant write
path costs one extra allocation per handler, but is meaningfully cheaper
in instructions. I'm happy to provide the benchmarks used to generate
those numbers in a separate pr – leaving them out here, as we would end
up with as many benchmarks as we have unit tests sooner or later.
Instructions, macOS arm64, per 1000 iterations, min of 5 interleaved
rounds:
```
┌──────────────────────────────┬────────────┬────────────┬───────────────────────────────────┐
│ │ main │ this PR │ │
├──────────────────────────────┼────────────┼────────────┼───────────────────────────────────┤
│ decode, no write observation │ 9,404,415 │ 9,360,896 │ −0.46% (noise ~0.8%, i.e. parity) │
├──────────────────────────────┼────────────┼────────────┼───────────────────────────────────┤
│ decode + re-entrant write │ 13,656,063 │ 12,541,951 │ −8.16% │
├──────────────────────────────┼────────────┼────────────┼───────────────────────────────────┤
│ handler add / remove │ 85,983,231 │ 83,361,791 │ −3.05% (noise ~2.2%) │
└──────────────────────────────┴────────────┴────────────┴───────────────────────────────────┘
```
Replacing the CanDequeueWrites existential with the closure is what
improves the reentrant write path: delivering a queued write is now a
direct closure call rather than a witness-table dispatch, and
handlerAdded no longer performs a runtime conditional-conformance
lookup.
The saving grows with the number of writes queued per decode — around
−16% with 8 queued writes, though that benchmark is noisier.
Allocations (nio-alloc-counter-tests, Swift 6.2):
```
┌────────────────────────────┬─────────┬─────────┬────────┐
│ │ main │ this PR │ │
├────────────────────────────┼─────────┼─────────┼────────┤
│ 1000_reqs_1_conn │ 27,341 │ 27,342 │ +1 │
├────────────────────────────┼─────────┼─────────┼────────┤
│ 1_reqs_1000_conn │ 368,000 │ 369,000 │ +1,000 │
├────────────────────────────┼─────────┼─────────┼────────┤
│ ping_pong_1000_reqs_1_conn │ 296 │ 296 │ — │
├────────────────────────────┼─────────┼─────────┼────────┤
│ decode_1000_ws_frames │ 2,000 │ 2,000 │ — │
└────────────────────────────┴─────────┴─────────┴────────┘
```
Forming the closure captures Decoder.OutboundIn's metadata, so it
allocates once per handler that actually queues a re-entrant write —
1_reqs_1000_conn builds 1000 such handlers, hence +1000. main's
existential cast did not allocate, so this is a new cost rather than
staying flat. The 1_reqs_1000_conn threshold is raised by 1000 for
6.1/6.2/6.3; the nightly toolchains are unchanged (10,000 on both — they
specialize the closure away entirely).
---------
Co-authored-by: Cory Benfield <lukasa@apple.com>
### Motivation:
`EventLoopFuture.get()` violates Structured Concurrency in two ways:
1. It doesn't automatically propagate cancellation
2. It doesn't return control back on cancellation
(1) is unfixable without user help but (2) we can address.
### Modifications:
- Add `EventLoopFuture.getAbandoningOnCancel()`
### Result:
- Users now have a way to at least get back control.
Add channel options to read a vsock socket's local and remote addresses
### Motivation:
NIOPosix supports vsock via `VsockAddress`, `ServerBootstrap.bind(to:)`
and `ClientBootstrap.connect(to:)`, but `Channel.localAddress` and
`Channel.remoteAddress` are always `nil` on vsock channels because
`NIOCore.SocketAddress` has no vsock representation.
Two things are unreachable as a result:
- A server accepting a vsock connection cannot tell which context
connected. The peer's context ID is the only identity a vsock connection
carries, and it is supplied by the kernel rather than the peer, which
makes it a natural basis for authorization.
- A listener bound to `VsockAddress/Port/any` cannot learn the port the
kernel assigned, so callers are forced to hard-code ports and race for
them.
`ChannelOptions.localVsockContextID` already exists for the first half
of this problem, but it reports only the context ID, not the port.
Adding a `.vsock` case to `SocketAddress` would fix both, but it is
source-breaking for every exhaustive switch over that enum, in NIO and
downstream.
### Modifications:
- Add `ChannelOptions.Types.LocalVsockAddress` and `RemoteVsockAddress`,
both get-only with `Value == VsockAddress`, alongside the existing
`LocalVsockContextID`.
- Add `BaseSocket.getLocalVsockAddress()` and `getRemoteVsockAddress()`,
sharing a helper that reads into a `sockaddr_storage` via
`withMutableSockAddr` and `convert()`, mirroring `BaseSocket.get_addr`.
On a non-vsock socket `getsockname`/`getpeername` fill a different
`sockaddr`, and reinterpreting those bytes would yield a context ID
derived from unrelated data such as a peer's IP address and port, so the
helper checks the address family and throws
`SocketAddressError.unsupported` on a mismatch.
- Service both options in `SocketChannel.getOption0`, and
`LocalVsockAddress` in `ServerSocketChannel.getOption0`.
`RemoteVsockAddress` is not serviced on `ServerSocketChannel`: a
listener has no peer.
- Tests: both helpers reject non-vsock sockets; over vsock loopback the
client's peer port matches the port the listener bound and the accepted
channel reports the same CID the client saw; a `Port/any` listener
reports a concrete port and a CID of `.any`, matching the channel
option. The loopback tests are gated on `System.supportsVsockLoopback`
in the same way as `testGetLocalCID` and `testEchoVsock`, so they run in
the `vsock-tests` CI job and skip elsewhere.
This is purely additive: two new option types, four new static
accessors, and new arms in two internal switches. No existing
declaration changes.
### Result:
A vsock server can identify the connecting context, and a wildcard-bound
listener can report its port:
```swift
let peer = try await channel.getOption(.remoteVsockAddress).get()
peer.cid // the connecting context's CID
let local = try await listener.getOption(.localVsockAddress).get()
local.port // the port a `Port/any` bind was given
```
Reading either option on a channel that isn't backed by a vsock socket
throws `SocketAddressError.unsupported`.
Co-authored-by: George Barnett <gbarnett@apple.com>
### Motivation:
`NIOBSDSocket.Option` provides shorthand API for socket options, e.g.
`.so_reuseaddr` for `SO_REUSEADDR`. We don't have API for
`SO_REUSEPORT`.
`SO_REUSEPORT` is used to bind more than one socket to the same address
and port to spread accepts or datagrams over multiple sockets and
support hot restarts.
Users can already use it today, but must use `init(rawValue:)`, and
import the platform C lib, etc:
```swift
.channelOption(.socketOption(.init(rawValue: SO_REUSEPORT)), value: 1)
```
There's no reason not to provide a convenience for this option, like we
do for many others.
### Modifications:
This PR adds `NIOBSDSocket.Option.so_reuseport`, wrapping
`SO_REUSEPORT`, alongside the other options in `BSDSocketAPI.swift`.
It's not provided on Windows or WASI, which don't have `SO_REUSEPORT`,
but there was already precedent for other options like this, e.g.
`SO_TIMESTAMP`.
### Result:
Users can pass `.socketOption(.so_reuseport)` like other supported
socket options:
```swift
DatagramBootstrap(group: group)
.channelOption(.socketOption(.so_reuseport), value: 1)
.bind(host: "127.0.0.1", port: 12345)
```
### What's wrong
`Sources/NIOCore/DispatchQueue+WithFuture.swift:30`, the return-value
line of the
public `DispatchQueue.asyncWithFuture(eventLoop:_:)`:
```swift
/// - returns a new `EventLoopFuture<ReturnType>` with value returned by the `block` parameter.
```
There is no colon after `returns`, so DocC does not recognise it as a `-
Returns:`
callout. The method renders **no Return Value section**; the sentence
falls into the
Discussion as a stray bullet instead.
Two further identifiers in that same sentence do not exist on this
method:
- `ReturnType` — the generic parameter is `NewValue`
(`asyncWithFuture<NewValue: Sendable>`).
- `block` — the closure parameter is `callbackMayBlock`.
The `- Parameters:` block directly above it is correct and unaffected.
### Reproduction
I could not build NIO to render this page (see Testing below), so I
reproduced the
DocC behaviour on a standalone module carrying this method's doc comment
**verbatim**,
alongside a copy with only the colon restored. Same file, same DocC, so
the callout
spelling is the only variable:
```
broken(eventloop:_:).json
HEADING: Discussion
codeListing ...
unorderedList: "returns a new `EventLoopFuture<ReturnType>` with value returned by the `block` parameter."
fixed(eventloop:_:).json
HEADING: Return Value
paragraph: "A new `EventLoopFuture<NewValue>` with value returned by the `callbackMayBlock` parameter."
```
The broken form produces a `Discussion` heading with the text demoted to
a list item.
The fixed form produces a `Return Value` heading with a proper
paragraph.
Commands, if you want to confirm it against the real module:
```
swift build --target NIOCore \
-Xswiftc -emit-symbol-graph -Xswiftc -emit-symbol-graph-dir -Xswiftc /tmp/sg
xcrun docc convert --fallback-display-name NIOCore \
--fallback-bundle-identifier org.swift.NIOCore \
--additional-symbol-graph-dir /tmp/sg --output-path /tmp/out.doccarchive
```
then read
`data/documentation/niocore/dispatchqueue/asyncwithfuture(eventloop:_:).json`.
Worth noting: **DocC emits no diagnostic for this.** `docc convert`
exits 0 with an
empty warning list, so nothing in a normal build or docs check surfaces
it. The
rendered output is the only evidence.
### Scope, and a judgement call
One line. This is the only occurrence in the repository — I grepped all
of `Sources/`
for `- returns`/`- Returns` without a colon and for backticked parameter
names, and
there is nothing else.
**The judgement call is fixing `ReturnType` and `block` in the same
change.** The
narrower alternative is to add only the colon and leave the two names
for a separate
PR. I did not do that, because restoring the colon is exactly what
*promotes this
sentence from invisible to visible* — shipping the colon alone would put
two
identifiers that do not exist on the method onto the rendered page for
the first time.
Correcting them is what makes the fix an improvement rather than a
regression in
visible accuracy. If you would rather have the colon alone and handle
the names
separately, say so and I will split it.
### Testing
**I did not run NIO's test suite, and I want to be straightforward about
why rather
than imply a green run.** The machine I am working on is
disk-constrained and a NIO
build did not fit; that is a limitation on my side, not a judgement that
the suite is
unnecessary. What I did verify:
- The defect and its uniqueness by reading, against `main` at
`8c063f043d94c120d0f8d6303ef4fc7918e3561d`.
- The identifier mismatches against the actual declaration in the same
file.
- DocC's parsing behaviour, via the standalone reproduction above (real
output, not asserted).
- The new line is 110 characters, within the repository's configured
`lineLength` of 120.
This is a comment-only change to a doc comment; it cannot affect
compiled behaviour.
CI will confirm the build and the format check.
### Disclosure
This change was written with AI assistance. The DocC output above is
real output from
my machine, not generated text.
Co-authored-by: hxperl <16929288+hxperl@users.noreply.github.com>
### Motivation
`ByteBuffer._Storage`'s `_capacity` and `_bytes` are only modified, if
the `ByteBuffer._Storage` is uniquely referenced. This is enforced by
the `isKnownUniquelyReferenced` checks in ByteBuffer itself. For this
reason unexclusive read and write access to these properties can never
happen.
Since the compiler sadly isn't able to spot this. Let's help it, by
marking the properties with `@exclusivity(unchecked)`
### Changes
- Mark ByteBuffer._Storage's _capacity and _bytes
`@exclusivity(unchecked)`
### Result
Significant reduction in instructions, which leads to faster ByteBuffer
code.
### Motivation
The new `NoUseUnstructuredThrowingTask` diagnostic on nightly-main flags
every `Task { try await ... }`
whose result is dropped. We have a few of those.
### Changes
- Use `async let _` instead of a fire-and-forget `Task` in
`NIOAsyncWriterTests` and
`NIOThrowingAsyncSequenceTests`.
- `NIOPerformanceTester`: rethrow the error from the `Task` in
`measureAndPrint` instead of
swallowing it.
- Name the `swift-nio` dependency explicitly in
`Benchmarks/Package.swift`; SwiftPM otherwise infers
it from the directory name, which breaks in git worktrees.
### Result
No warnings. Suspended yields in the tests are cancelled when the test
returns instead of leaking
past it, and async benchmarks fail loudly.
Fix the following warnings:
> setter for property 'xxx' is private and cannot be referenced from an
'@inlinable' function; this will be an error in a future Swift language
mode
### Motivation:
Eight doc comments name an argument that the declaration below them does
not have, so DocC has nothing to bind the description to and quietly
drops it. Nothing warns about this: Swift has no equivalent of
`-Wdocumentation`, and DocC stays silent rather than complaining, so
these sit until somebody reads the generated page and wonders where the
description went
| file | the comment says | the declaration says |
|---|---|---|
| `NIOPosix/ServerSocket.swift` | `descriptor` | `socket` |
| `NIOPosix/Socket.swift` ×2 | `descriptor` | `socket` |
| `NIOHTTP1/ByteCollectionUtils.swift` | `bytes` | `to` |
| `NIOHTTP1/HTTPTypes.swift` | `buffer` | `headers` |
| `NIOFS/BufferedStream.swift` ×2 | `sequence` | `element` |
| `NIOFS/SystemFileHandle.swift` | `executor` | `threadPool` |
The socket ones are worth a word. Each of those files has a pair: a
deprecated `init(descriptor:)` and its replacement `init(socket:)`, and
the rename is recorded right there in `@available(*, deprecated,
renamed: "init(socket:setNonBlocking:)")`. The replacement inherited the
wording of the one it replaced, so the doc still says `descriptor`. The
deprecated ones are correct and I left them alone
And the last two are the same defect in two copies of one module,
`NIOFS` and `_NIOFileSystem`, which is what a verbatim copy of a file
does to a mistake inside it. Fixing one and not the other would leave
them out of sync, so both are in here
### Modifications:
Renamed the documented argument to the one in the declaration, eleven
lines across eight files
One is slightly more than a rename: in `ByteBuffer.write(headers:)` the
entry described the buffer the method is called on rather than its
argument, so it now describes `headers` and keeps the note about the
writer index
### Result:
The parameter descriptions reach the generated documentation. No API
change, no behaviour change, comments only
btw these came out of a checker I wrote for exactly this, run over the
tree and then read by hand one by one. Six of the things it flagged
turned out to be my parser rather than your code, and those are fixed on
my side rather than sent to you 🙌🏼
all of it on a MacBook Air M3, of course
Motivation:
A number of NIO APIs are asynchronous and return futures but have
synchronous implementations. If a caller wants to operate on this they
need to register a callback with the future and then operate on the
result. However, now that we have an isolated view on the event loop
future we can provide a fast path: just check whether the result is
already present.
This is particularly helpful for channel initializers which are
typically synchronous. Using the fast path means calls can avoid an
escaping closure from registering a callback on the future.
Modifications:
Add three new APIs to the isolated event loop future:
- `result` which returns the underlying `Result<Value, Error>` or `nil`
if it hasn't been completed yet
- `value` which call `try result.get()`, and
- `isComplete`
Result:
APIs for accessing the result of an ELF from its isolated view if it's
already completed
### Motivation:
`CNIOWindows_sendmmsg` was a stub that asserted and then called
`abort()`, so any code path reaching it took the whole process down:
```
Assertion failed: !"sendmmsg not implemented", file Sources\CNIOWindows\shim.c, line 26
```
### Modifications:
Winsock has no `sendmmsg`, so emulate it by sending the messages one at
a time — exactly what the Darwin shim does for the same reason.
Error semantics match the Darwin implementation: a failure on the
first message is reported to the caller.
### Result:
Further progress on windows.
Windows-only; no other platform is affected.
---------
Co-authored-by: Fabian Fett <fabianfett@apple.com>
Motivation:
The module map for glibc is broken which can lead libc symbols being
attributed to the wrong module. In NIO, SocketAddresses.swift imports
CNIOLinux (and friends) but, as far as I can tell, doesn't use any of
their symbols directly. The result is that various libc symbols
incorrectly get attributed to CNIOLinux and compilation can fail for
downstream packages with member import visibility checks enabled.
See also: https://github.com/swiftlang/swift/issues/85427
Modifications:
- Remove the CNIO* imports from SocketAddresses.swift as they appear to
be unused
- Simplify the imoport block
Result:
Can build grpc with member import visibility checks enabled.
Co-authored-by: Cory Benfield <lukasa@apple.com>
### Motivation:
`System.enumerateDevices()` crashes on Windows, which in turn makes
`UtilitiesTest.testEnumeratingDevices` and the whole of
`SocketOptionProviderTest` (whose `setUp` enumerates devices) unrunnable
there; both are currently skipped.
`GetAdaptersAddresses` reports the size its output requires in *bytes*,
but the buffer was allocated as `Int(ulSize) /
MemoryLayout<IP_ADAPTER_ADDRESSES>.stride` *elements*. That integer
division truncates, so the buffer was always smaller than what the OS
asked for, and the OS then wrote its linked list of variably sized
records (adapter names, sockaddrs, unicast address entries) past the end
of it, corrupting the heap.
### Modifications:
- Size the allocation in bytes, starting from the 15KB initial
allocation the
[documentation](https://learn.microsoft.com/en-us/windows/win32/api/iphlpapi/nf-iphlpapi-getadaptersaddresses)
recommends, and retry on `ERROR_BUFFER_OVERFLOW` — the set of adapters
can change between the sizing call and the filling call.
- Remove the corresponding skips in
`UtilitiesTest.testEnumeratingDevices` and `SocketOptionProviderTest`.
- Running `SocketOptionProviderTest` for the first time on Windows
showed that `testPassingInvalidSizeToSetComplexSocketOptionFails` needs
a Windows-specific expectation. Winsock's `SO_RCVTIMEO` takes a `DWORD`
rather than a `timeval`, so the `Int` the test passes is in fact large
enough and `setsockopt` correctly accepts it. On Windows the test now
passes a single byte, which is too small for any option, and asserts the
`WSAEFAULT` that Winsock reports.
No behaviour change on any other platform; everything is behind `#if
os(Windows)`.
### Result:
`System.enumerateDevices()` works on Windows, and 13 previously skipped
tests run there.
Verified on a Windows ARM64 VM (Swift 6.3.2): a full unfiltered `swift
test` passes — 2354 tests, 0 failures.
### Motivation:
The WSAPoll selector traps with `preconditionFailure("Invalid fd
supplied.")` whenever `WSAPoll` reports `POLLNVAL`, and that is
reachable from ordinary shutdown, so
`MultiThreadedEventLoopGroupTests.testShuttingDownFailsRegistration` is
currently skipped on Windows.
The root cause is not in the selector. `SelectableEventLoop.deregister`
guards on `isOpen`, which is false both when the loop is closed *and*
when it has merely stopped accepting **new** registrations. The latter
is precisely the state the loop is in during `Selector.closeGently`,
which closes each remaining channel — so those channels skip
deregistration entirely and then close their sockets anyway.
The epoll and kqueue backends tolerate that, because there a
registration lives in the kernel and is dropped along with the
descriptor. The WSAPoll backend keeps its registrations in an array of
its own, so it is left holding a stale entry for a closed socket, and
the next `WSAPoll` correctly reports `POLLNVAL` for it.
Traced on a Windows ARM64 VM by instrumenting the selector and the event
loop:
```
register0 fd=704 idx=1
EL.deregister SKIPPED state=runningButNotAcceptingNewRegistrations
EL.deregister SKIPPED state=runningButNotAcceptingNewRegistrations
POLLNVAL fd=704 i=1 revents=4 events=256 dereg=[] indexes=[704: 1, 712: 2, 728: 3]
Fatal error: Invalid fd supplied.
```
### Modifications:
- Guard `deregister` on the *selector's* lifecycle rather than on
`isOpen`. Deregistering an existing channel while the loop is closing is
valid, because the selector itself is still open at that point.
- Remove the skip on
`MultiThreadedEventLoopGroupTests.testShuttingDownFailsRegistration`.
`POLLNVAL` deliberately keeps trapping. Like `EBADF`, which NIO already
treats as an unacceptable errno, it means we have lost track of a
descriptor; Windows recycles `SOCKET` handles, so continuing against a
possibly-reused handle would misroute I/O to the wrong channel. An
earlier version of this patch mapped `POLLNVAL` to `.error` instead —
that silently retained the stale entry and gave up that guarantee, so it
was dropped in favour of fixing the cause.
**Note for reviewers:** unlike most of the Windows porting patches this
touches shared code rather than sitting behind `#if os(Windows)`. The
observable behaviour on Linux/Darwin is unchanged, but they will now
issue one `epoll_ctl(EPOLL_CTL_DEL)`/kevent delete per channel during
event loop shutdown where previously they issued none.
### Result:
The WSAPoll selector no longer traps during event loop shutdown, and
`testShuttingDownFailsRegistration` runs on Windows.
Verified:
- Windows ARM64 (Swift 6.3.2), full unfiltered `swift test`: 2354 tests,
0 failures.
- macOS, full `swift test` with and without this change: identical
results (2451 tests, same 28 pre-existing failures caused by the network
mount used for this port, none in NIOPosix).
Co-authored-by: Fabian Fett <fabianfett@apple.com>
Saturate the `ByteCount` unit factories so large counts no longer trap
on overflow.
### Motivation:
`ByteCount` exposes unit factories such as `kilobytes`, `megabytes`,
`gigabytes`, `kibibytes`, `mebibytes`, and `gibibytes`. Each one
computed the byte total with a plain multiplication, for example `1024 *
1024 * 1024 * count` in `gibibytes`. When a caller passes a very large
count, that multiplication overflows `Int64` and traps, crashing the
process. This is exactly what issue #2877 reports:
`ByteCount(contentsOf:maximumSizeAllowed: .gibibytes(.max))` crashes
with an arithmetic overflow inside `gibibytes`, even though passing a
huge value is a natural way to express "effectively no limit".
### Modifications:
I added a small `fileprivate` saturating multiplication helper on
`Int64` and routed every unit factory through it. The helper uses
`multipliedReportingOverflow(by:)` and, on overflow, clamps to
`Int64.max` for positive counts or `Int64.min` for negative counts.
Counts that fit continue to produce the exact same value as before, so
there is no behavior change for normal inputs. I extended
`ByteCountTests` with positive and negative overflow cases, and applied
the identical change to both copies of `ByteCount.swift` (`NIOFS` and
`_NIOFileSystem`) to keep them in sync, matching the convention used by
recent changes to those targets.
### Result:
Passing extreme values such as `.gibibytes(.max)` saturates to
`Int64.max` instead of trapping, so disabling a size limit via a large
value no longer crashes. I verified this by running `swift test --filter
ByteCountTests`, which passes after the change. To confirm the tests are
load bearing I reverted the source fix and re-ran the new tests, which
crashed with signal 5 (the overflow trap) exactly as the issue
describes, and they pass again with the fix in place. `swift format lint
--strict` reports no issues on the changed files.
Fixes#2877
### Motivation:
`Selector.description` prints `self.selectorFD`, which only the epoll,
kqueue and io_uring backends ever assign. On Windows it therefore always
reads `descriptor = -1` — the value that by convention means the
selector has been closed — even while the selector is open and perfectly
usable.
That made
`MultiThreadedEventLoopGroupTests.testStructuredConcurrencyMTELGStartStop`
unrunnable on Windows, since it asserts that a running event loop does
*not* describe itself as having descriptor -1, and that a shut down one
does.
### Modifications:
The WSAPoll backend has no single descriptor standing for the selector
itself, but the read end of its wakeup socket pair has exactly the same
lifetime: it is created in `initialiseState0` and reset to
`NIOBSDSocket.invalidHandle` in `close0`, under the same
`externalSelectorFDLock` that this description already takes. Report
that on Windows, mapping the invalid handle to -1 so the established
meaning of -1 is preserved.
Remove the corresponding skip.
The change is behind `#if os(Windows)`; other platforms keep printing
`selectorFD` exactly as before.
### Result:
A running event loop on Windows now describes its selector accurately,
and `testStructuredConcurrencyMTELGStartStop` runs there.
Verified on a Windows ARM64 VM (Swift 6.3.2): a full unfiltered `swift
test` passes — 2354 tests, 0 failures.
This change clarifies the documentation of
`replaceItem(at:withItemAt:)`. It explicitly states that this operation
is identical to a move, and that the item at `existingPath` will be
moved to the new location and no longer exist at its original path.
For small fixed-size allocations, we can use stack instead of heap.
### Motivation:
Allocations are costly. Stack allocations are much cheaper than heap
allocations.
### Modifications:
Use `withUnsafeTemporaryAllocation(of:capacity:_:)` instead of
allocating an `Array`.
`withUnsafeTemporaryAllocation` allocates on the stack for up to 1000
bytes.
### Result:
Better performance.
Motivation:
The schedule callback mechanism on `EventLoop` has an equivalent on an
isolated `EventLoop`. The default isolated implementation puts the
hander in a loop bound box and calls the non-isolated version.
This is okay but introduces a number of additional "in event loop"
checks which are required for correctness in the general case (i.e. not
called from an isolated event loop).
Modifications:
- Implement the customisation point directly to avoid the additional
checks (the are unecessary by definition of being on the isolated event
loop).
- Add benchmarks
Result:
On the isolated event loop, in micro benchmarks:
- scheduleTask is ~1.6x faster
- scheduleCallback is ~2x faster
This PR addresses **#3613** where writing to `/dev/stdout` fails with
`ENXIO (6) Device not configured` when running in a terminal environment
instead of a pipe.
Currently, `SystemFileHandle` checks for `errno == .illegalSeek`
(ESPIPE) when deciding whether to fall back from positional
`pwrite`/`pread` to non-positional `write`/`read` for unseekable files
like pipes. However, macOS/Darwin returns `ENXIO`
(`Errno.noSuchDeviceOrAddress`) instead of `ESPIPE` when attempting a
positional write/read on a terminal device.
**Changes:**
- Added `.noSuchDeviceOrAddress` to the fallback conditions in
`SystemFileHandle.swift` for both `readChunk` and `write`.
- Updated `FileSystemError+Syscall.swift` to map
`.noSuchDeviceOrAddress` to `.unsupported` alongside `.illegalSeek`.
- Replicated these changes across both the `_NIOFileSystem` and `NIOFS`
modules.
This allows `FileSystem.shared.withFileHandle` and other APIs to work
transparently with terminal devices like `/dev/stdout`!
### Motivation:
Currently, EventLoopFuture.flatMap requires the inner eventloop to have
a Sendable type. This makes sense, because the value might move between
eventloops, if the inner and outer future are not on the same EventLoop.
However, this makes it impossible for users to use non Sendable values
even when they _know_ they're on the right eventloop
### Modifications:
Add a new `flatMapIsolated` function which can be used in this case. It
requires users to return an EventLoopFuture.Isolated to prove that they
are on the same eventloop
### Example:
```swift
class T { // Not Sendable
init() {
}
}
func test(eventLoop: any EventLoop) -> EventLoopFuture<T> {
let a = eventLoop.submit { T() }
let b = eventLoop.submit { T() }
return a.flatMap { _ in b }
}
```
The above example gives a warning on the final line, because flatMap
requires the inner future (`b`) to have a Sendable type. That is because
it doesn't know that a and b are on the same EventLoop.
```swift
func test(eventLoop: any EventLoop) -> EventLoopFuture<T> {
let a = eventLoop.submit { T() }
let b = eventLoop.submit { T() }
return a.flatMapIsolated { _ in b.assumeIsolated() }
}
```
Here, it compiles without warnings. We can write assumeIsolated inside
the flatMapIsolated closure because we know that the eventloops are the
same. And therefore the new flatMapIsolated function does not have a
Sendable requirement on its value
Co-authored-by: Cory Benfield <lukasa@apple.com>
## Motivation
Follow-up to #3649 (all non-test targets now build on Windows). This
gets the bulk of the **test** targets compiling on Windows so we can
start moving toward running them in CI.
Of the 16 test targets, 6 already compiled unchanged. This brings **8
more** to a clean compile, leaving only two as dedicated follow-ups:
- `NIOPosixTests` — large, many distinct Winsock/POSIX issues; deserves
its own PR.
- `NIOFoundationCompatTests` — a `missing required module 'CNIOPosix'`
issue to investigate separately.
The changes are **compile-only** — they don't aim to make the tests
*pass* on Windows yet. Verified by building each target on a Windows
ARM64 VM; not yet wired into CI (CI still builds products only, since
the two deferred targets would fail `--build-tests`).
## What's here
Three commits:
1. **Make `Posix.close` available on Windows** — it was gated out only
because it sat inside the socket-syscalls `#if !os(Windows)` block (next
to `shutdown`), even though its body already handled Windows. Define
`sysClose = _close` and lift `close(descriptor:)` out of that block. No
behavior change off Windows.
2. **Compile the low-hanging test targets** — a
gate-out-plus-minimal-shim approach:
- A tiny `#if os(Windows)` `usleep` shim (backed by `Sleep`), once per
affected target, plus a Windows branch for the `#error` C-library check.
- Real Windows implementations for the temp-file/pipe test helpers
rather than stubs: `withPipe` → `_pipe`, `openTemporaryFile` →
`_mktemp_s` + `_sopen_s`, `createTemporaryDirectory` → a GUID +
`CreateDirectoryW`.
- Gate out genuinely-unavailable test logic behind `#if !os(Windows)`:
`System.enumerateInterfaces`, `ChannelOptions.socket(IPPROTO_IP,
IP_TTL)`, `mkfifo`-based tests, the POSIX-`stat`-shaped `FileInfoTests`,
the `S_IFSOCK`/`DT_*` conversion tests, and the `openat` oflag-bit test.
- Two Windows-only shim tweaks make the NIOFS `OpenOptions` gap values
and `Errno.noData` `@_spi(Testing) public`, matching how swift-system
exposes the equivalents elsewhere (the tests reach them via an SPI
import).
- The `NIOHTTP1Tests` and `NIOPosixTests` copies of `TestUtils.swift`
are intentional duplicates and get identical Windows helper
implementations to stay in sync.
3. **Document why `SocketOption`'s integer API is unavailable on
Windows** — comment-only: WinSDK imports the socket option level/name
constants as enumeration cases, not plain integers, so they don't fit
the integer `SocketOptionLevel`/`SocketOptionName` typealiases.
## Notes for reviewers
- No behavior change off Windows: every change is behind `#if
os(Windows)` / `#if !os(Windows)`; the one shared edit (a test buffer
typed `CChar` instead of `CInterop.PlatformChar`) is identical on POSIX.
- The `usleep` shim is duplicated per target rather than shared: there's
no test-only support target these all depend on (the shared deps,
`NIOCore`/`NIOTestUtils`, are shipping libraries). It can be centralised
later if desired.
---------
Co-authored-by: George Barnett <gbarnett@apple.com>
## Motivation
This continues the incremental Windows port of SwiftNIO. `NIOPosix`
already builds on Windows (on Winsock2 with a POSIX-shim mapping); this
PR gets the **L4 NIOFileSystem family** to **compile** on Windows:
- `_NIOFileSystem`
- `NIOFS`
- `NIOFileSystem`
- `_NIOFileSystemFoundationCompat`
- `NIOFSFoundationCompat`
With these compiling, **all non-test targets now build on Windows**, so
the Windows CI job is simplified to a bare `swift build` (which builds
the whole non-test graph) rather than an explicit per-target list.
## Scope — compile-only
**This PR gets the targets compiling; it does not yet make them
functional.** The POSIX file-system syscalls that don't exist (or aren't
exposed by swift-system) on Windows are provided as `fatalError("…
unavailable on Windows")` **stubs** with the correct signatures, so the
library type-checks and links. Real Win32 implementations, and bringing
the test targets online in CI to validate them, are deliberately left to
follow-up PRs.
The intent is to land the smallest reviewable slice first and **get
feedback on the overall shape and organization** before building on it.
## What this does
All changes are behind `#if os(Windows)` / `#if defined(_WIN32)`, with
**zero behavior change on Darwin/Linux/BSD**:
- **`Sources/*/Internal/System Calls/SystemPackage+Windows.swift`**
(new, one per tree): a module-scope compatibility shim supplying the
pieces the rest of the target expects on Windows —
- the swift-system-gap extensions swift-system doesn't provide on
Windows
(`FileDescriptor.OpenOptions.{noFollow,closeOnExec,nonBlocking,directory}`,
`Errno.noData`);
- `CInterop.Stat`/`DirEnt`/`FTS`/`FTSEnt` Windows stand-ins and the
`S_IF*`/`DT_*`/`FTS_*`/`UTIME_*` constants the call sites reference;
- `fatalError` stubs for the bare-libc syscalls (`openat`,
`stat`/`lstat`/`fstat`, `readdir`/`fdopendir`/`closedir`, `fts_*`,
`futimens`, `rename`, `symlink`, xattr-adjacent helpers, `getcwd`,
pthread TLS keys, …), matching the exact signatures the existing
`system_*` / `libc_*` wrappers call, so those wrappers and their
non-Windows call sites need no changes.
- Small `#elseif os(Windows)` branches in the existing platform switches
(`FileInfo`, `FileType`, `Errno`, `Mocking`, `Syscall(s)`,
`SystemFileHandle`, `FileSystem`) — including `fatalError` stubs for
`_copyRegularFile` / `_copySymbolicLink` /
file-materialization-via-rename.
- A tiny `CNIOWindows` addition: `errno` accessors (`CNIOWindows_errno`
/ `CNIOWindows_set_errno`, typed `errno_t` via
`_get_errno`/`_set_errno`) so the Swift errno indirection can bridge the
CRT `errno` macro.
- `CNIOWindows` added to the `NIOFS` / `_NIOFileSystem` target
dependencies in `Package.swift`.
`NIOFS` and `_NIOFileSystem` are lockstep near-duplicates, so every
Windows change is applied identically to both.
## Verification
Verified by source review and by confirming a clean `swift build` on
`aarch64-unknown-windows-msvc`. Since the syscalls are stubbed, there is
no runtime behavior to test yet.
## What I'm looking for
Feedback on the **high-level organization and shape** of the NIOFS
Windows shim before the real implementations follow:
1. Is the layering right — a single module-scope
`SystemPackage+Windows.swift` gap-shim, with the upstream `system_*` /
`libc_*` call sites left untouched?
2. When the implementations land, is emulating the POSIX seams (a
`fts(3)`-shaped tree walk, POSIX directory streams) the right approach,
or should Windows fork at a higher layer (e.g. a Windows-specific
enumerator in `DirectoryEntries`) rather than reimplementing
`fts`/`readdir`?
3. Anything about the swift-system-gap extensions or the `CInterop.*`
Windows stand-ins that should be shaped differently.
## Next steps
Follow-up PRs will replace the stubs with real Win32 implementations,
then bring the NIOFileSystem **test targets** building and running in CI
— where these shims get real validation and we start gaining deeper
confidence in Windows support across the stack.
---------
Co-authored-by: George Barnett <gbarnett@apple.com>
### Motivation:
On the server side, `HTTPResponseEncoder` chooses a response's body
framing only from the response status
(`HTTPResponseStatus.mayHaveResponseBody`): it has no knowledge of the
request method as it's an outbound-only handler. But HTTP requires that
a response to a HEAD request (and a 2xx response to CONNECT) carry no
message body, regardless of status or framing headers.
As a result, a response to a HEAD request with a status that allows a
body (like 200) and no `Content-Length` gets framed as
`Transfer-Encoding: chunked`, and the encoder emits the chunked
terminator (`0\r\n\r\n`). This is illegal.
A conformant client won't read those bytes when receiving a HEAD
response, so on a keep-alive connection they get misinterpreted as the
start of the next response, corrupting the connection. This surfaces as
intermittent, hard-to-diagnose failures (e.g. "I/O on closed channel")
on the following request.
### Modifications:
Made the server response framing method-aware without changing any
public API:
- The request decoder now hands each decoded request's method to the
response encoder.
- `HTTPResponseEncoder` consults the method and, for responses to
HEAD/CONNECT requests, omits the body and the terminator.
- This is wired up in `configureHTTPServerPipeline`. When the encoder is
constructed and used on its own, its behaviour is unchanged (framing is
still derived solely from the status), so there is no behavioural or API
change for users who may be creating this handler manually.
### Result:
Responses to `HEAD` and `CONNECT` requests are no longer emitted with a
message body, keeping keep-alive connections well-formed and
RFC-compliant. No public API changes.
Motivation:
GHSA-ffw9-jrfx-pfvc raised a question about whether `setBytes(_:at:)`,
`setInteger(_:at:...)`, and `setString(_:at:)` correctly validate an
out-of-range `index`. On review, this turned out to be
working-as-intended: these APIs trap on an invalid index by design,
unlike the `get*` family, which validates via `rangeWithinReadableBytes`
and returns `nil` instead. That asymmetry wasn't previously called out
in the doc comments, which made it easy to mistake for an unmitigated
validation gap rather than documented behavior.
Modifications:
Added a `- Precondition:` note to the doc comments of `setBytes(_:at:)`
(all overloads), `setInteger(_:at:...)`, and `setString(_:at:)` in
NIOCore, making the trapping behavior explicit and noting that callers
must validate any externally-derived `index` before use.
Result:
No behavior changes. The trapping contract on these `set*` write APIs is
now documented, matching the existing convention used by `get*`'s
optional-return documentation.One line description of your change
### Motivation:
The old NIOCrashTester doesn't work on Windows and would be hard to
port. Also, Swift Testing has wonderful 'exit test' ability, let's use
that!
### Modifications:
- Rewrite all NIOCrashTester tests in Swift Testing
### Result:
- Less code to maintain
- (hopefully) works on Windows
- fixes#3657
Add the two remaining non-test, non-NIOFS targets to the Windows CI
build list.
NIOPerformanceTester builds and runs on Windows. The only obstacle was
the `setbuf(stdout, nil)` call: `stdout` is a macro on the Windows CRT
that Swift cannot reference directly. Add a
`CNIOWindows_setStdoutUnbuffered()` wrapper so the C compiler expands
the macro (keeping us correct if its definition ever changes), and call
it on Windows.
NIOCrashTester cannot function on Windows: its OutputGrepper captures a
child process's output through a pipe channel (`NIOPipeBootstrap`),
which deliberately `fatalError`s on Windows, and relies on
`Foundation.Pipe` fd conversion plus Unix-signal crash detection. Extend
the existing `!canImport(Darwin) || os(macOS)` guard with `&&
!os(Windows)` across the target so it compiles to a no-op on Windows,
just as it already does on the non-macOS Darwin platforms.
This PR gets TCP (Servers) on Windows mostly working. I'll annotate my
PR for clarity.
**The one bug:**
If you open a TCP client connection to a Windows TCP server, it doesn't
read/write yet until a second connection comes in. This happens because
`reregister0` is called after `WSAPoll` is called - but WSAPoll isn't
woken up to know this so it doesn't know the new socket requests
read/writes/...
Also the EventLoop cannot be woken up during `el.execute { .. }` because
`SleepEx` isn't running. WSAPoll doesn't respond to this signal it
seems.
This makes the TCP server functionality not yet usable, but I thought
with this draft we could figure out what I'm missing.
Alternative considered: We can make `WSAPoll` limited to wake up every 1
ms for example, that way we can still get to these events at _some_
point.
---------
Co-authored-by: Cory Benfield <lukasa@apple.com>
Co-authored-by: Fabian Fett <fabianfett@apple.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Oliver Drobnik <oliver@cocoanetics.com>
Co-authored-by: Jake Petroules <jake.petroules@apple.com>
Co-authored-by: George Barnett <gbarnett@apple.com>
Co-authored-by: Gabriele Mondada <gab@ijk.ch>
### Motivation:
NIO's historic limit was that the maximum header block was limited to 80
kB which is a sensible limit and originates from Nginx's http_parser.c.
The history of the changes:
- commit e67fc995: Introduces the 80 kB limit
- commit 69180342: Accidentally removes the limit entirely
- commit b24872d3: Introduced the 2 MB / 256 headers limit
I would argue that 2MB & 256 headers is a bad limit:
- 2MB is way too big, that allows pretty powerful slowloris-style
attacks forcing NIO to hold on to 2 MB per connection
- 256 headers is too little for software that may be a proxy
- in HTTP, the number of headers is _non-semantic_
- A: `a: 1\r\n a: 2\r\n a: 3\r\n a: 4\r\n a: 5\r\n a: 6\r\n[...]\r\na:
257\r\n`
- B: `a: 1, 2, 3, 4, 5, ..., 257\r\n`
- A & B are _equivalent_. Yet, A would be rejected and B accepted.
That's not right.
### Modifications:
Restore the old limits.
### Result:
- More compatible (because we allow more than 256 headers)
- More secure (because the default goes back to 80kB)
Motivation:
Users can write arbitrary code in their deinits, and that arbitrary code
_may_ end up hitting Objective-C autoreleases. We don't want to leak
those objects into the per-thread autoreleasepool, as that may not be
popped for a very long time, if ever, so we need to to ensure that
deiniting the task objects has an autoreleasepool.
Modifications:
Add an autoreleasepool around the task drop, and tests to validate that
we do things correctly.
Result:
Better memory management around deinit
### Motivation
`HTTPResponseEncoder` writes hand-optimised status lines for the
well-known `HTTPResponseStatus` values instead of going through the
general `default` path that formats `HTTP/<version> <code>
<reasonPhrase>`. Several of those fast paths had drifted out of sync
with `HTTPResponseStatus.reasonPhrase`, so the exact bytes on the wire
depended on whether a status happened to have a fast path:
* **HTTP/1.0 + 511** emitted the wrong HTTP version (`HTTP/1.1 511 ...`)
— a wire-format correctness bug, not just cosmetic.
* **HTTP/1.1 + 431** emitted the reason phrase `Range Not Satisfiable`
instead of `Request Header Fields Too Large`.
* **HTTP/1.1 + 416** emitted `Request Range Not Satisified` (also
misspelled) instead of `Range Not Satisfiable`.
* **HTTP/1.0 and HTTP/1.1 + 307** emitted `Tempory Redirect`
(misspelled) instead of `Temporary Redirect`.
Resolves#3630.
### Modifications
* Correct the five diverging status lines in `HTTPResponseEncoder` so
they match `HTTPResponseStatus.reasonPhrase` and carry the correct HTTP
version.
* Add a regression test that, for every well-known status at both
HTTP/1.0 and HTTP/1.1, asserts the encoded status line equals the
canonical `HTTP/<version> <code> <reasonPhrase>` form, keeping the fast
paths in lockstep with `reasonPhrase`.
### Result
The encoder now produces correct, consistent status lines for all
well-known statuses regardless of whether they use a fast path. The new
test fails on the pre-fix code (the 511 case fails on the version, the
431/416/307 cases on the reason phrase) and passes after the fix.
---------
Co-authored-by: George Barnett <gbarnett@apple.com>
Off-by-one in the QUIC variable-length integer length boundaries.
### Motivation:
`bytesNeededForInteger` classifies an integer into the 1/2/4/8 byte QUIC
encoding using half-open ranges (`0..<63`, `0..<16383`,
`0..<1_073_741_823`, `0..<4_611_686_018_427_387_903`). Those upper
bounds are the *largest* value each size can hold (2^6-1, 2^14-1, 2^30-1
and 2^62-1 per RFC 9000 § 16), so each maximum is wrongly excluded:
- 63, 16383 and 2^30-1 report one byte too many, so the encoder writes
them with a larger encoding than needed.
- 2^62-1, the largest value the encoding can represent, falls through to
the `fatalError` and crashes the encoder.
The existing one-byte write test only iterates `0..<63`, so the 63 case
was never exercised.
### Modifications:
- Switch the four ranges in `bytesNeededForInteger` to inclusive bounds.
- Extend the one-byte write test to include 63, and add boundary checks
plus a 2^62-1 round-trip test.
### Result:
Boundary values encode in the minimal number of bytes and 2^62-1
round-trips instead of crashing.
Co-authored-by: Cory Benfield <lukasa@apple.com>
This PR is just a cleanup of the source code.
### Motivation:
The `CNIOFreeBSD` source code contains inconsistent trailing
whitespaces, which hinders future modifications.
### Modifications:
Removed all trailing whitespaces from `CNIOFreeBSD`.
### Result:
The source code remains exactly the same; only the trailing whitespace
has been removed.
Co-authored-by: Cory Benfield <lukasa@apple.com>
We are running into some issues when doing:
```swift
let writeOptions: OpenOptions.Write =
byteOffset > 0 ? .modifyFile(createIfNecessary: true) : .newFile(replaceExisting: true)
try await FileSystem.shared.withFileHandle(
forWritingAt: FilePath(outputFilePath),
options: writeOptions
) { fileHandle in
```
which results in crashes on some Android devices where `renameat2` is
not available:
```
('renameat2' system call failed with '(38) Function not implemented'.)
```
This PR adds functionality similar to what is already done for `EINVAL`
on Linux, such that both lead to the same recovery: stat the
destination, then plain rename() if absent.
And for non-exclusive renames where it also fails to function not
implemented, it falls back to plain `rename`.