Compare commits

..
126 Commits
Author SHA1 Message Date
David Smith 41daa93a5d Merge pull request #235 from apple/merge-1.5.0-into-main
Merge 1.5.0 into main
2025-07-24 22:18:56 +00:00
David Smith 2adb6b41cc Merge branch 'release/1.5.0' into merge-1.5.0-into-main 2025-07-24 14:29:18 -07:00
Guillaume Lessard 6ee9a58c36 Merge pull request #234 from azarovalex/fix-api-docs
Fix API documentation links in README.md
2025-07-14 15:18:35 -07:00
David Smith 7f9dba80cf Merge pull request #208 from apple/david/ioring
Add io_uring support to System
2025-07-12 00:39:04 +00:00
David Smith 08ed962064 Don't abort the test if we can't open /proc/sys/kernel/io_uring_disabled 2025-07-11 23:05:26 +00:00
David Smith 08ea3343a5 Merge branch 'david/ioring' of https://github.com/apple/swift-system into david/ioring 2025-07-11 22:42:41 +00:00
David Smith a66176f518 Only run IORing tests if io_uring support is enabled 2025-07-11 22:42:35 +00:00
David Smith 33b58a8110 Fix excludes harder 2025-07-11 13:44:46 -07:00
David Smith 52d9c99a94 Fix excludes 2025-07-11 13:42:49 -07:00
David Smith b93169b8e2 Work around windows bugs harder 2025-07-11 20:37:23 +00:00
David Smith 9a046498d0 Try adding newlines to the ends of files to work around a compiler bug 2025-07-11 19:23:24 +00:00
David Smith a56bcda3c1 Un-combine conditionals but flip their order. This should fix the Windows build with older compilers 2025-07-11 19:07:04 +00:00
David Smith f7c05e1fb8 Combine platform conditionals 2025-07-11 18:53:55 +00:00
David Smith 7fc787e6ba Move IORing files to a subdirectory 2025-07-11 17:00:32 +00:00
David Smith 779c9ac628 Fix misuse of _CANCEL_ANY, and add support for non-redicated cancellation 2025-07-11 16:50:58 +00:00
David Smith 2f6c4e0f03 Add support for non-IORING_FEAT_SINGLE_MMAP kernels 2025-07-11 16:14:36 +00:00
David Smith d121081c01 More Ubuntu 22 workarounds. This should error at runtime rather than not compiling, since we don't have Linux version checks at compile time 2025-07-10 23:30:42 +00:00
David Smith a6675d7e5c timespec should be compatible here, let's try it to fix the Ubuntu 22 build 2025-07-10 23:22:11 +00:00
David Smith 329c5f59a2 Conditionalize IORing on compiler >= 6.2 2025-07-09 21:17:42 +00:00
David Smith 6dd0eba8e3 Merge branch 'david/ioring' of https://github.com/apple/swift-system into david/ioring 2025-07-09 20:53:28 +00:00
David Smith 4b70ffe375 Fix short submits and also fix the registration ops enum, which somehow stopped compiling overnight 2025-07-09 20:53:23 +00:00
David Smith 8757fae97c Remove some stray commas the API breakage test complains about 2025-07-08 16:57:30 -07:00
David Smith d2b78f801b Remove unneeded defines 2025-07-08 23:53:50 +00:00
David Smith d6687d2bbb Add deleted file back 2025-07-08 16:51:24 -07:00
David Smith 528e7485ba Document completion flags and disable ones we don't currently support 2025-07-08 16:38:21 -07:00
David Smith a6c5a39ca5 Remove the proposal doc 2025-07-08 16:24:28 -07:00
David Smith 55e55c1978 Add test and fix submission queue capacity check 2025-07-08 23:15:17 +00:00
David Smith bc097e91dc Remove more blocking enqueue infrastructure 2025-07-08 22:42:15 +00:00
David Smith 227a38f230 Remove blocking enqueue 2025-07-08 22:41:06 +00:00
David Smith bbe2f1c192 Add doc comments, Span/MutableSpan support, unregistering registered fds, variants of submission operations that don't block, address various TODOs 2025-07-08 00:41:50 +00:00
David Smith e580c13ca8 Merge branch 'david/ioring' of https://github.com/apple/swift-system into david/ioring 2025-07-04 00:11:17 +00:00
David Smith a6149c215e Cleanup, implement buffer unregistering, inline almost everything because we're not ABI stable on Linux 2025-07-04 00:09:19 +00:00
David Smith de2d6e2d39 Conditionalize tests too 2025-07-03 16:17:51 -07:00
David Smith 0d4c6fc971 Fix Musl 2025-07-03 16:12:56 -07:00
David Smith 27ccce55db Conditionalize IORing stuff for Linux 2025-07-03 16:04:52 -07:00
David Smith 321a71c2e9 Expand test coverage 2025-07-03 00:13:29 +00:00
David Smith 131326c8ae Fix tests 2025-07-01 01:02:40 +00:00
Alex Azarov c0273af7e5 Fix API documentation links in README.md 2025-06-28 17:49:11 +02:00
David Smith f91cb6b8de Improve tests. Not quite working but close 2025-06-27 00:39:35 +00:00
David Smith f9c2a2b8e7 Work around io_uring.h in ways that will work on both versions 2025-06-18 23:08:45 +00:00
Guillaume Lessard 61e4ca4b81 Merge pull request #231 from apple/main
Merge main into release/1.5.0
2025-05-28 16:25:04 -07:00
Guillaume Lessard f3154e33ec Merge branch 'release/1.4.0' into release/1.5.0 2025-05-23 15:09:25 -07:00
Guillaume Lessard 99feed67fe Merge pull request #225 from marcprux/main
Add Android imports to tests
2025-05-22 10:07:29 -07:00
Guillaume Lessard 40d22dace6 Merge pull request #230 from owenv/owenv/windows-release-path-parsing
Fix Windows path parsing when the package is built in release configuration
2025-05-22 09:48:39 -07:00
Owen Voorhees 595c862c19 Fix Windows path parsing when the package is built in release configuration
forceWindowsPaths can be used to force Windows path parsing both on and off.
Instead of defaulting it to false in release builds, we should not express any preference
2025-05-21 21:12:58 -07:00
David Smith 3d6cba22b6 Work around compiler bugs harder 2025-04-28 23:03:26 +00:00
David Smith f5b8e02ba7 Merge remote-tracking branch 'origin/main' into david/ioring 2025-04-28 22:33:21 +00:00
David Smith 7e2f797fb4 Fix some errors on newer linux systems 2025-04-28 22:30:00 +00:00
David Smith 3a649e32a6 Fix type of features, and make it clear that Completion is a nested type 2025-04-24 16:39:41 -07:00
Jake Petroules 5695a8831b Transparently add the \\?\ prefix to Win32 calls for extended length path handling
On Windows, there is a built-in maximum path limitation of 260 characters under most conditions. This can be extended to 32767 characters under either of the following two conditions:

- Adding the longPathAware attribute to the executable's manifest AND enabling the LongPathsEnabled system-wide registry key or group policy.
- Ensuring fully qualified paths passed to Win32 APIs are prefixed with \\?\

Unfortunately, the former is not realistic for the Swift ecosystem, since it requires developers to have awareness of this specific Windows limitation, AND set longPathAware in their apps' manifest AND expect end users of those apps to change their system configuration.

Instead, this patch transparently prefixes all eligible paths in calls to Win32 APIs with the \\?\ prefix to allow them to work with paths longer than 260 characters without requiring the caller of System to manually prefix the paths.

See https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation for more info.
2025-04-23 14:52:03 -07:00
David Smith f38d72f310 Nested types, some renames, misc other updates 2025-04-22 16:32:12 -07:00
David Smith a49ba8a8b3 Give flags a default value 2025-04-22 23:20:37 +00:00
David Smith c47779fac0 Try making requests and completions nested types 2025-04-22 23:07:01 +00:00
David Smith 344b3b85f8 Proposal updates 2025-04-21 16:59:39 -07:00
David Smith a98cd3f2da Build fix and rename IOResource 2025-04-21 23:47:00 +00:00
David Smith 3760dc265a Nest IOResource, and update proposal to match 2025-04-21 16:35:08 -07:00
David Smith baa3d62569 API tweak 2025-04-21 23:26:14 +00:00
David Smith ea77ef8a21 Minor cleanup 2025-04-21 23:24:09 +00:00
David Smith 9d325b29ce Change IORingFileSlot and IORingBuffer to be nested types in IORing 2025-04-21 23:23:26 +00:00
David Smith 7798f0c957 Add cancellation support and drop gerunds 2025-04-21 23:18:33 +00:00
Guillaume Lessard 5691e92a4b Merge pull request #227 from apple/enable-windows-ci 2025-04-19 09:59:09 -07:00
Jake Petroules 28ce835363 Enable Windows CI
FileOperationsTestWindows.testPermissions fails on Windows in a container environment with Hyper-V isolated containers; see https://learn.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/persistent-storage#permissions for some potentially-relevant information.

Skip it if we detect we're in a container.

Closes #223
2025-04-19 00:23:55 -07:00
David Smith 05eac87c08 It compiles again! 2025-04-18 23:38:43 +00:00
David Smith c029c1bfbd Proposal updates and a minor change to setup flags 2025-04-18 15:22:17 -07:00
David Smith 261ca250fe Centralize on Errno for errors, and work around lifetime compiler crash. There's a second compiler crash still to work around though. 2025-04-18 22:20:09 +00:00
David Smith 014f8b78eb Rename userData to context 2025-04-17 23:36:51 +00:00
David Smith 41f9676907 WIP, crashes the compiler 2025-04-17 23:31:01 +00:00
David Smith d4ca203c09 Most of feature querying and setup flags 2025-04-17 00:26:13 +00:00
David Smith 2f91217c90 Fix a few things I missed 2025-04-16 23:44:21 +00:00
David Smith 27275f843a Merge remote-tracking branch 'refs/remotes/origin/david/ioring' into david/ioring 2025-04-16 23:38:35 +00:00
David Smith b40b09440c Revamp error handling 2025-04-16 23:37:50 +00:00
David Smith 0e652c4d25 Naming updates, cancellation support, IOResources are structs now, other editorial updates 2025-04-04 16:42:33 -07:00
Marc Prud'hommeaux 009b07e140 Add Android imports to tests 2025-03-13 08:59:59 -04:00
Karoy Lorentey 1b8167d354 Merge pull request #222 from apple/github-actions-support
[CI] Add support for GitHub Actions
2025-03-10 14:50:35 -07:00
Mishal Shah 2ac80d6f4e Disable a few checks 2025-03-09 18:29:05 -07:00
David Smith 7fff872441 Add the draft proposal so I can link to it 2025-03-06 12:09:57 -08:00
Mishal Shah d6b40a7284 [CI] Add support for GitHub Actions 2025-03-04 23:22:34 -08:00
David Smith 72c316ba86 Make completions noncopyable again 2025-03-03 23:59:36 +00:00
David Smith 48455a9135 Fix plumbing 2025-03-03 23:04:52 +00:00
David Smith 880ec9086e Add a pointer convenience for getting the user data 2025-03-03 22:51:35 +00:00
David Smith 9ad16c0aae Plumb through userData 2025-03-03 22:37:51 +00:00
David Smith 91155fd527 Plumb error handling through completion consumers 2025-03-03 22:15:32 +00:00
David Smith 49dd7977ca Add a combined "submit and consume" operation 2025-03-03 22:07:28 +00:00
David Smith 5e24673259 Switch from unsafe pointers to FilePaths, using hacks 2025-03-03 21:53:26 +00:00
David Smith d338de1478 Only get one completion if we asked for one completion 2025-02-28 00:49:31 +00:00
David Smith f3b8cc473d Actually consume events we waited for 2025-02-28 00:45:31 +00:00
David Smith d895f2a828 Revert "Reapply "Turn on single issuer as an experiment""
This reverts commit 283e8d6b7b.
2025-02-14 18:27:17 +00:00
David Smith 283e8d6b7b Reapply "Turn on single issuer as an experiment"
This reverts commit 901f4c80c4.
2025-02-14 18:26:31 +00:00
David Smith 901f4c80c4 Revert "Turn on single issuer as an experiment"
This reverts commit 0064649631.
2025-02-14 18:25:46 +00:00
David Smith 0064649631 Turn on single issuer as an experiment 2025-02-14 18:24:09 +00:00
David Smith 5ba137703c Dubious approach to this, but I want to try it out a bit 2025-02-13 23:43:19 +00:00
David Smith 6983196ad6 Add unlinkAt support 2025-02-13 23:31:06 +00:00
David Smith a22e5f6858 Add a version of registerBuffers that isn't varargs 2025-02-13 22:09:42 +00:00
David Smith 0c6ef16ca9 Fix type 2025-02-12 23:45:18 +00:00
David Smith 7f6e673750 Adjust registered resources API 2025-02-12 23:43:29 +00:00
David Smith 74366c3341 Improve submit API 2025-02-12 20:18:51 +00:00
David Smith 5bfed03c97 Switch to static methods for constructing requests 2025-02-11 23:27:03 +00:00
David Smith d4ca4129a0 Some renaming, and implement linked requests 2025-02-11 23:01:33 +00:00
David Smith a396967e3a More registration tweaks 2025-02-11 22:38:01 +00:00
David Smith bb03f0fba3 Redesign registered resources API 2025-02-11 22:33:10 +00:00
David Smith 02481e0daa More import fixes 2025-02-11 21:06:16 +00:00
David Smith 7107a570b4 Fix import visibility 2025-02-11 21:01:49 +00:00
David Smith fdbcecab0b Eliminate internal locking and add multiple consume support 2025-02-06 00:42:21 +00:00
David Smith e5fdf9ea96 Remove managed abstractions for now 2025-02-06 00:41:30 +00:00
David Smith 55fd6e7150 Add support for timeout-on-wait to IORing, don't have tests yet 2024-12-12 01:20:24 +00:00
David Smith 7ea32aefce More refactoring, and working timeout support on ManagedIORing 2024-12-11 23:00:50 +00:00
David Smith ef94a37193 Refactoring, and give up on resources being noncopyable structs 2024-12-05 03:11:11 +00:00
David Smith f369347b73 Fix mismerge 2024-10-28 17:25:31 -07:00
David Smith 1a3e37d791 Fix mismerge 2024-10-28 17:24:46 -07:00
David Smith a9f92a6e31 Delete stray Package.resolved changes 2024-10-28 17:23:25 -07:00
David Smith 6f793cfe60 Merge branch 'main' into david/ioring 2024-10-29 00:21:42 +00:00
David Smith fcb0b69df6 Merge branch 'main' into david/ioring 2024-10-28 17:20:36 -07:00
David Smith 822e4814d1 Adopt isolation parameters 2024-10-28 23:49:37 +00:00
David Smith 32966f975d It builds again! With some horrible hacks 2024-10-28 23:43:59 +00:00
David Smith 10c070abd6 WIP, more ~Copyable adoption 2024-10-28 20:00:07 +00:00
David Smith 60299362ed One more noncopyable struct 2024-10-24 21:27:22 +00:00
David Smith baab9b26b7 Starting to move to noncopyable structs, and away from swift-atomics 2024-10-24 21:21:07 +00:00
Lucy Satheesan 6b4084c69f introduce IORing unit tests 2023-08-11 09:08:43 -07:00
Lucy Satheesan b596783bd9 implement closing 2023-08-11 09:07:47 -07:00
Lucy Satheesan b58450413e fix off-by-one in IORequest.openat 2023-08-11 09:07:08 -07:00
Lucy Satheesan d099546787 fix access control 2023-08-11 09:06:17 -07:00
Lucy Satheesan 0762f57d80 migrate CSystem to systemLibrary
for some reason, the linker on my linux machine fails
to link the tests otherwise. investigate / fix before merging.
2023-08-10 16:03:21 -07:00
Lucy Satheesan 1f821a8539 AsyncSequence draft implementation 2023-08-08 09:44:32 -07:00
Lucy Satheesan 996e940942 initial AsyncFileDescriptor work 2023-07-31 22:28:54 -07:00
Lucy Satheesan 6338029b74 stage 2: IORequest enum 2023-07-31 22:28:43 -07:00
Lucy Satheesan 15794f5cad ioring pitch: first steps 2023-07-18 21:04:39 -07:00
25 changed files with 2046 additions and 38 deletions

No files matched your search

+23
View File
@@ -0,0 +1,23 @@
name: Pull request
on:
pull_request:
types: [opened, reopened, synchronize]
jobs:
tests:
name: Test
uses: swiftlang/github-workflows/.github/workflows/swift_package_test.yml@main
with:
linux_exclude_swift_versions: '[{"swift_version": "5.8"}]'
soundness:
name: Soundness
uses: swiftlang/github-workflows/.github/workflows/soundness.yml@main
with:
license_header_check_project_name: "Swift.org"
# https://github.com/apple/swift-system/issues/224
docs_check_enabled: false
unacceptable_language_check_enabled: false
license_header_check_enabled: false
format_check_enabled: false
python_lint_check_enabled: false
+1 -1
View File
@@ -1,5 +1,5 @@
LinuxSwiftPackageJob {
swift_version_tag = "5.10-jammy"
repo = "swift-system"
branch = "release/1.4.0"
branch = "main"
}
+1 -1
View File
@@ -1,5 +1,5 @@
LinuxSwiftPackageJob {
swift_version_tag = "5.9-jammy"
repo = "swift-system"
branch = "release/1.4.0"
branch = "main"
}
+1 -1
View File
@@ -1,5 +1,5 @@
macOSSwiftPackageJob {
swift_version = "6.0"
repo = "swift-system"
branch = "release/1.4.0"
branch = "main"
}
+1 -1
View File
@@ -1,5 +1,5 @@
LinuxSwiftPackageJob {
nightly_docker_tag = "nightly-6.0-jammy"
repo = "swift-system"
branch = "release/1.4.0"
branch = "main"
}
+1 -1
View File
@@ -1,5 +1,5 @@
macOSSwiftPackageJob {
swift_version = "main"
repo = "swift-system"
branch = "release/1.4.0"
branch = "main"
}
+1 -1
View File
@@ -1,5 +1,5 @@
LinuxSwiftPackageJob {
nightly_docker_tag = "nightly-jammy"
repo = "swift-system"
branch = "release/1.4.0"
branch = "main"
}
+1 -1
View File
@@ -1,7 +1,7 @@
WindowsSwiftPackageWithDockerImageJob {
docker_image = "swiftlang/swift:nightly-windowsservercore-1809"
repo = "swift-system"
branch = "release/1.4.0"
branch = "main"
sub_dir = "swift-system"
label = "windows-server-2019"
}
+15 -1
View File
@@ -22,8 +22,21 @@ let swiftSettings: [SwiftSetting] = [
.when(platforms: [.macOS, .macCatalyst, .iOS, .watchOS, .tvOS, .visionOS])),
.define("SYSTEM_PACKAGE"),
.define("ENABLE_MOCKING", .when(configuration: .debug)),
.enableExperimentalFeature("Lifetimes"),
]
#if os(Linux)
let filesToExclude = ["CMakeLists.txt"]
#else
let filesToExclude = ["CMakeLists.txt", "IORing"]
#endif
#if os(Linux)
let testsToExclude:[String] = []
#else
let testsToExclude = ["IORequestTests.swift", "IORingTests.swift"]
#endif
let package = Package(
name: "swift-system",
products: [
@@ -40,12 +53,13 @@ let package = Package(
name: "SystemPackage",
dependencies: ["CSystem"],
path: "Sources/System",
exclude: ["CMakeLists.txt"],
exclude: filesToExclude,
cSettings: cSettings,
swiftSettings: swiftSettings),
.testTarget(
name: "SystemTests",
dependencies: ["SystemPackage"],
exclude: testsToExclude,
cSettings: cSettings,
swiftSettings: swiftSettings),
])
+1 -1
View File
@@ -24,7 +24,7 @@ try fd.closeAfter {
}
```
[API documentation](https://swiftpackageindex.com/apple/swift-system/main/documentation/SystemPackage)
[API documentation](https://swiftpackageindex.com/apple/swift-system/documentation/SystemPackage)
## Adding `SystemPackage` as a Dependency
+1
View File
@@ -21,5 +21,6 @@
#include <pthread.h>
#include <sched.h>
#include <unistd.h>
#include "io_uring.h"
#endif
+60
View File
@@ -0,0 +1,60 @@
#include <unistd.h>
#include <sys/syscall.h>
#include <sys/uio.h>
#include <signal.h>
#include <linux/io_uring.h>
#ifndef SWIFT_IORING_C_WRAPPER
#define SWIFT_IORING_C_WRAPPER
# ifndef __NR_io_uring_setup
# define __NR_io_uring_setup 425
# endif
# ifndef __NR_io_uring_enter
# define __NR_io_uring_enter 426
# endif
# ifndef __NR_io_uring_register
# define __NR_io_uring_register 427
# endif
/*
struct io_uring_getevents_arg {
__u64 sigmask;
__u32 sigmask_sz;
__u32 min_wait_usec; //used to be called `pad`. This compatibility wrapper avoids dealing with that.
__u64 ts;
};
*/
struct swift_io_uring_getevents_arg {
__u64 sigmask;
__u32 sigmask_sz;
__u32 min_wait_usec;
__u64 ts;
};
static inline int io_uring_register(int fd, unsigned int opcode, void *arg,
unsigned int nr_args)
{
return syscall(__NR_io_uring_register, fd, opcode, arg, nr_args);
}
static inline int io_uring_setup(unsigned int entries, struct io_uring_params *p)
{
return syscall(__NR_io_uring_setup, entries, p);
}
static inline int io_uring_enter2(int fd, unsigned int to_submit, unsigned int min_complete,
unsigned int flags, void *args, size_t sz)
{
return syscall(__NR_io_uring_enter, fd, to_submit, min_complete,
flags, args, _NSIG / 8);
}
static inline int io_uring_enter(int fd, unsigned int to_submit, unsigned int min_complete,
unsigned int flags, sigset_t *sig)
{
return io_uring_enter2(fd, to_submit, min_complete, flags, sig, _NSIG / 8);
}
#endif
@@ -40,7 +40,7 @@ fileprivate func forEachFile(
try searchPath.withPlatformString { szPath in
var findData = WIN32_FIND_DATAW()
let hFind = FindFirstFileW(szPath, &findData)
let hFind = try szPath.withCanonicalPathRepresentation({ szPath in FindFirstFileW(szPath, &findData) })
if hFind == INVALID_HANDLE_VALUE {
throw Errno(windowsError: GetLastError())
}
@@ -95,8 +95,8 @@ internal func _recursiveRemove(
let subpath = path.appending(component)
if (findData.dwFileAttributes & DWORD(FILE_ATTRIBUTE_DIRECTORY)) == 0 {
try subpath.withPlatformString {
if !DeleteFileW($0) {
try subpath.withPlatformString { subpath in
if try !subpath.withCanonicalPathRepresentation({ DeleteFileW($0) }) {
throw Errno(windowsError: GetLastError())
}
}
@@ -105,7 +105,7 @@ internal func _recursiveRemove(
// Finally, delete the parent
try path.withPlatformString {
if !RemoveDirectoryW($0) {
if try !$0.withCanonicalPathRepresentation({ RemoveDirectoryW($0) }) {
throw Errno(windowsError: GetLastError())
}
}
@@ -461,3 +461,78 @@ extension SystemString {
return lexer.current
}
}
#if os(Windows)
import WinSDK
// FIXME: Rather than canonicalizing the path at every call site to a Win32 API,
// we should consider always storing absolute paths with the \\?\ prefix applied,
// for better performance.
extension UnsafePointer where Pointee == CInterop.PlatformChar {
/// Invokes `body` with a resolved and potentially `\\?\`-prefixed version of the pointee,
/// to ensure long paths greater than MAX_PATH (260) characters are handled correctly.
///
/// - seealso: https://learn.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation
internal func withCanonicalPathRepresentation<Result>(_ body: (Self) throws -> Result) throws -> Result {
// 1. Normalize the path first.
// Contrary to the documentation, this works on long paths independently
// of the registry or process setting to enable long paths (but it will also
// not add the \\?\ prefix required by other functions under these conditions).
let dwLength: DWORD = GetFullPathNameW(self, 0, nil, nil)
return try withUnsafeTemporaryAllocation(of: WCHAR.self, capacity: Int(dwLength)) { pwszFullPath in
guard (1..<dwLength).contains(GetFullPathNameW(self, DWORD(pwszFullPath.count), pwszFullPath.baseAddress, nil)) else {
throw Errno(rawValue: _mapWindowsErrorToErrno(GetLastError()))
}
// 1.5 Leave \\.\ prefixed paths alone since device paths are already an exact representation and PathCchCanonicalizeEx will mangle these.
if let base = pwszFullPath.baseAddress,
base[0] == UInt8(ascii: "\\"),
base[1] == UInt8(ascii: "\\"),
base[2] == UInt8(ascii: "."),
base[3] == UInt8(ascii: "\\") {
return try body(base)
}
// 2. Canonicalize the path.
// This will add the \\?\ prefix if needed based on the path's length.
var pwszCanonicalPath: LPWSTR?
let flags: ULONG = numericCast(PATHCCH_ALLOW_LONG_PATHS.rawValue)
let result = PathAllocCanonicalize(pwszFullPath.baseAddress, flags, &pwszCanonicalPath)
if let pwszCanonicalPath {
defer { LocalFree(pwszCanonicalPath) }
if result == S_OK {
// 3. Perform the operation on the normalized path.
return try body(pwszCanonicalPath)
}
}
throw Errno(rawValue: _mapWindowsErrorToErrno(WIN32_FROM_HRESULT(result)))
}
}
}
@inline(__always)
fileprivate func HRESULT_CODE(_ hr: HRESULT) -> DWORD {
DWORD(hr) & 0xffff
}
@inline(__always)
fileprivate func HRESULT_FACILITY(_ hr: HRESULT) -> DWORD {
DWORD(hr << 16) & 0x1fff
}
@inline(__always)
fileprivate func SUCCEEDED(_ hr: HRESULT) -> Bool {
hr >= 0
}
// This is a non-standard extension to the Windows SDK that allows us to convert
// an HRESULT to a Win32 error code.
@inline(__always)
fileprivate func WIN32_FROM_HRESULT(_ hr: HRESULT) -> DWORD {
if SUCCEEDED(hr) { return DWORD(ERROR_SUCCESS) }
if HRESULT_FACILITY(hr) == FACILITY_WIN32 {
return HRESULT_CODE(hr)
}
return DWORD(hr)
}
#endif
+71
View File
@@ -0,0 +1,71 @@
#if compiler(>=6.2)
#if os(Linux)
import CSystem
public extension IORing {
struct Completion: ~Copyable {
@inlinable init(rawValue inRawValue: io_uring_cqe) {
rawValue = inRawValue
}
@usableFromInline let rawValue: io_uring_cqe
}
}
public extension IORing.Completion {
struct Flags: OptionSet, Hashable, Codable {
public let rawValue: UInt32
@inlinable public init(rawValue: UInt32) {
self.rawValue = rawValue
}
///`IORING_CQE_F_BUFFER` Indicates the buffer ID is stored in the upper 16 bits
@inlinable public static var allocatedBuffer: Flags { Flags(rawValue: 1 << 0) }
///`IORING_CQE_F_MORE` Indicates more completions will be generated from the request that generated this
@inlinable public static var moreCompletions: Flags { Flags(rawValue: 1 << 1) }
//`IORING_CQE_F_SOCK_NONEMPTY`, but currently unused
//@inlinable public static var socketNotEmpty: Flags { Flags(rawValue: 1 << 2) }
//`IORING_CQE_F_NOTIF`, but currently unused
//@inlinable public static var isNotificationEvent: Flags { Flags(rawValue: 1 << 3) }
//IORING_CQE_F_BUF_MORE will eventually be (1U << 4) if we add IOU_PBUF_RING_INC support
}
}
public extension IORing.Completion {
@inlinable var context: UInt64 {
get {
rawValue.user_data
}
}
@inlinable var userPointer: UnsafeRawPointer? {
get {
UnsafeRawPointer(bitPattern: UInt(rawValue.user_data))
}
}
@inlinable var result: Int32 {
get {
rawValue.res
}
}
@inlinable var flags: IORing.Completion.Flags {
get {
Flags(rawValue: rawValue.flags & 0x0000FFFF)
}
}
@inlinable var bufferIndex: UInt16? {
get {
if self.flags.contains(.allocatedBuffer) {
return UInt16(rawValue.flags >> 16)
} else {
return nil
}
}
}
}
#endif
#endif
+497
View File
@@ -0,0 +1,497 @@
#if compiler(>=6.2)
#if os(Linux)
import CSystem
@usableFromInline
internal enum IORequestCore {
case nop // nothing here
case openat(
atDirectory: FileDescriptor,
path: FilePath,
FileDescriptor.AccessMode,
options: FileDescriptor.OpenOptions = FileDescriptor.OpenOptions(),
permissions: FilePermissions? = nil,
context: UInt64 = 0
)
case openatSlot(
atDirectory: FileDescriptor,
path: FilePath,
FileDescriptor.AccessMode,
options: FileDescriptor.OpenOptions = FileDescriptor.OpenOptions(),
permissions: FilePermissions? = nil,
intoSlot: IORing.RegisteredFile,
context: UInt64 = 0
)
case read(
file: FileDescriptor,
buffer: IORing.RegisteredBuffer,
offset: UInt64 = 0,
context: UInt64 = 0
)
case readUnregistered(
file: FileDescriptor,
buffer: UnsafeMutableRawBufferPointer,
offset: UInt64 = 0,
context: UInt64 = 0
)
case readSlot(
file: IORing.RegisteredFile,
buffer: IORing.RegisteredBuffer,
offset: UInt64 = 0,
context: UInt64 = 0
)
case readUnregisteredSlot(
file: IORing.RegisteredFile,
buffer: UnsafeMutableRawBufferPointer,
offset: UInt64 = 0,
context: UInt64 = 0
)
case write(
file: FileDescriptor,
buffer: IORing.RegisteredBuffer,
offset: UInt64 = 0,
context: UInt64 = 0
)
case writeUnregistered(
file: FileDescriptor,
buffer: UnsafeMutableRawBufferPointer,
offset: UInt64 = 0,
context: UInt64 = 0
)
case writeSlot(
file: IORing.RegisteredFile,
buffer: IORing.RegisteredBuffer,
offset: UInt64 = 0,
context: UInt64 = 0
)
case writeUnregisteredSlot(
file: IORing.RegisteredFile,
buffer: UnsafeMutableRawBufferPointer,
offset: UInt64 = 0,
context: UInt64 = 0
)
case close(
FileDescriptor,
context: UInt64 = 0
)
case closeSlot(
IORing.RegisteredFile,
context: UInt64 = 0
)
case unlinkAt(
atDirectory: FileDescriptor,
path: FilePath,
context: UInt64 = 0
)
case cancel(
flags:UInt32
)
case cancelContext(
flags: UInt32,
targetContext: UInt64
)
case cancelFD(
flags: UInt32,
targetFD: FileDescriptor
)
case cancelFDSlot(
flags: UInt32,
target: IORing.RegisteredFile
)
}
@inline(__always) @inlinable
internal func makeRawRequest_readWrite_registered(
file: FileDescriptor,
buffer: IORing.RegisteredBuffer,
offset: UInt64,
context: UInt64 = 0,
request: consuming RawIORequest
) -> RawIORequest {
request.fileDescriptor = file
request.buffer = buffer.unsafeBuffer
request.rawValue.buf_index = UInt16(exactly: buffer.index)!
request.offset = offset
request.rawValue.user_data = context
return request
}
@inline(__always) @inlinable
internal func makeRawRequest_readWrite_registered_slot(
file: IORing.RegisteredFile,
buffer: IORing.RegisteredBuffer,
offset: UInt64,
context: UInt64 = 0,
request: consuming RawIORequest
) -> RawIORequest {
request.rawValue.fd = Int32(exactly: file.index)!
request.flags = .fixedFile
request.buffer = buffer.unsafeBuffer
request.rawValue.buf_index = UInt16(exactly: buffer.index)!
request.offset = offset
request.rawValue.user_data = context
return request
}
@inline(__always) @inlinable
internal func makeRawRequest_readWrite_unregistered(
file: FileDescriptor,
buffer: UnsafeMutableRawBufferPointer,
offset: UInt64,
context: UInt64 = 0,
request: consuming RawIORequest
) -> RawIORequest {
request.fileDescriptor = file
request.buffer = buffer
request.offset = offset
request.rawValue.user_data = context
return request
}
@inline(__always) @inlinable
internal func makeRawRequest_readWrite_unregistered_slot(
file: IORing.RegisteredFile,
buffer: UnsafeMutableRawBufferPointer,
offset: UInt64,
context: UInt64 = 0,
request: consuming RawIORequest
) -> RawIORequest {
request.rawValue.fd = Int32(exactly: file.index)!
request.flags = .fixedFile
request.buffer = buffer
request.offset = offset
request.rawValue.user_data = context
return request
}
extension IORing {
public struct Request {
@usableFromInline var core: IORequestCore
@inlinable internal init(core inCore: IORequestCore) {
core = inCore
}
@inlinable internal consuming func extractCore() -> IORequestCore {
return core
}
}
}
extension IORing.Request {
@inlinable public static func nop(context: UInt64 = 0) -> IORing.Request {
.init(core: .nop)
}
@inlinable public static func read(
_ file: IORing.RegisteredFile,
into buffer: IORing.RegisteredBuffer,
at offset: UInt64 = 0,
context: UInt64 = 0
) -> IORing.Request {
.init(core: .readSlot(file: file, buffer: buffer, offset: offset, context: context))
}
@inlinable public static func read(
_ file: FileDescriptor,
into buffer: IORing.RegisteredBuffer,
at offset: UInt64 = 0,
context: UInt64 = 0
) -> IORing.Request {
.init(core: .read(file: file, buffer: buffer, offset: offset, context: context))
}
@inlinable public static func read(
_ file: IORing.RegisteredFile,
into buffer: UnsafeMutableRawBufferPointer,
at offset: UInt64 = 0,
context: UInt64 = 0
) -> IORing.Request {
.init(core: .readUnregisteredSlot(file: file, buffer: buffer, offset: offset, context: context))
}
@inlinable public static func read(
_ file: FileDescriptor,
into buffer: UnsafeMutableRawBufferPointer,
at offset: UInt64 = 0,
context: UInt64 = 0
) -> IORing.Request {
.init(core: .readUnregistered(file: file, buffer: buffer, offset: offset, context: context))
}
@inlinable public static func write(
_ buffer: IORing.RegisteredBuffer,
into file: IORing.RegisteredFile,
at offset: UInt64 = 0,
context: UInt64 = 0
) -> IORing.Request {
.init(core: .writeSlot(file: file, buffer: buffer, offset: offset, context: context))
}
@inlinable public static func write(
_ buffer: IORing.RegisteredBuffer,
into file: FileDescriptor,
at offset: UInt64 = 0,
context: UInt64 = 0
) -> IORing.Request {
.init(core: .write(file: file, buffer: buffer, offset: offset, context: context))
}
@inlinable public static func write(
_ buffer: UnsafeMutableRawBufferPointer,
into file: IORing.RegisteredFile,
at offset: UInt64 = 0,
context: UInt64 = 0
) -> IORing.Request {
.init(core: .writeUnregisteredSlot(
file: file, buffer: buffer, offset: offset, context: context))
}
@inlinable public static func write(
_ buffer: UnsafeMutableRawBufferPointer,
into file: FileDescriptor,
at offset: UInt64 = 0,
context: UInt64 = 0
) -> IORing.Request {
.init(
core: .writeUnregistered(file: file, buffer: buffer, offset: offset, context: context)
)
}
@inlinable public static func close(
_ file: FileDescriptor,
context: UInt64 = 0
) -> IORing.Request {
.init(core: .close(file, context: context))
}
@inlinable public static func close(
_ file: IORing.RegisteredFile,
context: UInt64 = 0
) -> IORing.Request {
.init(core: .closeSlot(file, context: context))
}
@inlinable public static func open(
_ path: FilePath,
in directory: FileDescriptor,
into slot: IORing.RegisteredFile,
mode: FileDescriptor.AccessMode,
options: FileDescriptor.OpenOptions = FileDescriptor.OpenOptions(),
permissions: FilePermissions? = nil,
context: UInt64 = 0
) -> IORing.Request {
.init(
core: .openatSlot(
atDirectory: directory, path: path, mode, options: options,
permissions: permissions, intoSlot: slot, context: context))
}
@inlinable public static func open(
_ path: FilePath,
in directory: FileDescriptor,
mode: FileDescriptor.AccessMode,
options: FileDescriptor.OpenOptions = FileDescriptor.OpenOptions(),
permissions: FilePermissions? = nil,
context: UInt64 = 0
) -> IORing.Request {
.init(
core: .openat(
atDirectory: directory, path: path, mode, options: options,
permissions: permissions, context: context
))
}
@inlinable public static func unlink(
_ path: FilePath,
in directory: FileDescriptor,
context: UInt64 = 0
) -> IORing.Request {
.init(core: .unlinkAt(atDirectory: directory, path: path, context: context))
}
// Cancel
/*
* ASYNC_CANCEL flags.
*
* IORING_ASYNC_CANCEL_ALL Cancel all requests that match the given key
* IORING_ASYNC_CANCEL_FD Key off 'fd' for cancelation rather than the
* request 'user_data'
* IORING_ASYNC_CANCEL_ANY Match any request
* IORING_ASYNC_CANCEL_FD_FIXED 'fd' passed in is a fixed descriptor
* IORING_ASYNC_CANCEL_USERDATA Match on user_data, default for no other key
* IORING_ASYNC_CANCEL_OP Match request based on opcode
*/
@inlinable internal static var SWIFT_IORING_ASYNC_CANCEL_ALL: UInt32 { 1 << 0 }
@inlinable internal static var SWIFT_IORING_ASYNC_CANCEL_FD: UInt32 { 1 << 1 }
@inlinable internal static var SWIFT_IORING_ASYNC_CANCEL_ANY: UInt32 { 1 << 2 }
@inlinable internal static var SWIFT_IORING_ASYNC_CANCEL_FD_FIXED: UInt32 { 1 << 3 }
@inlinable internal static var SWIFT_IORING_ASYNC_CANCEL_USERDATA: UInt32 { 1 << 4 }
@inlinable internal static var SWIFT_IORING_ASYNC_CANCEL_OP: UInt32 { 1 << 5 }
public enum CancellationMatch {
case all
case first
}
@inlinable public static func cancel(
_ matchAll: CancellationMatch,
matchingContext: UInt64
) -> IORing.Request {
switch matchAll {
case .all:
.init(core: .cancelContext(flags: SWIFT_IORING_ASYNC_CANCEL_ALL | SWIFT_IORING_ASYNC_CANCEL_USERDATA, targetContext: matchingContext))
case .first:
.init(core: .cancelContext(flags: SWIFT_IORING_ASYNC_CANCEL_USERDATA, targetContext: matchingContext))
}
}
@inlinable public static func cancel(
_ matchAll: CancellationMatch,
matching: FileDescriptor
) -> IORing.Request {
switch matchAll {
case .all:
.init(core: .cancelFD(flags: SWIFT_IORING_ASYNC_CANCEL_ALL | SWIFT_IORING_ASYNC_CANCEL_FD, targetFD: matching))
case .first:
.init(core: .cancelFD(flags: SWIFT_IORING_ASYNC_CANCEL_FD, targetFD: matching))
}
}
@inlinable public static func cancel(
_ matchAll: CancellationMatch,
matching: IORing.RegisteredFile
) -> IORing.Request {
switch matchAll {
case .all:
.init(core: .cancelFDSlot(flags: SWIFT_IORING_ASYNC_CANCEL_ALL | SWIFT_IORING_ASYNC_CANCEL_FD_FIXED, target: matching))
case .first:
.init(core: .cancelFDSlot(flags: SWIFT_IORING_ASYNC_CANCEL_FD_FIXED, target: matching))
}
}
@inlinable public static func cancel(
_ matchAll: CancellationMatch,
) -> IORing.Request {
switch matchAll {
case .all:
.init(core: .cancel(flags: SWIFT_IORING_ASYNC_CANCEL_ALL))
case .first:
.init(core: .cancel(flags: SWIFT_IORING_ASYNC_CANCEL_ANY))
}
}
@inline(__always) @inlinable
internal consuming func makeRawRequest() -> RawIORequest {
var request = RawIORequest()
switch extractCore() {
case .nop:
request.operation = .nop
case .openatSlot(
let atDirectory, let path, let mode, let options, let permissions, let fileSlot,
let context):
// TODO: use rawValue less
request.operation = .openAt
request.fileDescriptor = atDirectory
request.rawValue.addr = UInt64(
UInt(
bitPattern: path.withPlatformString { ptr in
ptr //this is unsavory, but we keep it alive by storing path alongside it in the request
}))
request.rawValue.open_flags = UInt32(bitPattern: options.rawValue | mode.rawValue)
request.rawValue.len = permissions?.rawValue ?? 0
request.rawValue.file_index = UInt32(fileSlot.index + 1)
request.path = path
request.rawValue.user_data = context
case .openat(
let atDirectory, let path, let mode, let options, let permissions, let context):
request.operation = .openAt
request.fileDescriptor = atDirectory
request.rawValue.addr = UInt64(
UInt(
bitPattern: path.withPlatformString { ptr in
ptr //this is unsavory, but we keep it alive by storing path alongside it in the request
}))
request.rawValue.open_flags = UInt32(bitPattern: options.rawValue | mode.rawValue)
request.rawValue.len = permissions?.rawValue ?? 0
request.path = path
request.rawValue.user_data = context
case .write(let file, let buffer, let offset, let context):
request.operation = .writeFixed
return makeRawRequest_readWrite_registered(
file: file, buffer: buffer, offset: offset, context: context, request: request)
case .writeSlot(let file, let buffer, let offset, let context):
request.operation = .writeFixed
return makeRawRequest_readWrite_registered_slot(
file: file, buffer: buffer, offset: offset, context: context, request: request)
case .writeUnregistered(let file, let buffer, let offset, let context):
request.operation = .write
return makeRawRequest_readWrite_unregistered(
file: file, buffer: buffer, offset: offset, context: context, request: request)
case .writeUnregisteredSlot(let file, let buffer, let offset, let context):
request.operation = .write
return makeRawRequest_readWrite_unregistered_slot(
file: file, buffer: buffer, offset: offset, context: context, request: request)
case .read(let file, let buffer, let offset, let context):
request.operation = .readFixed
return makeRawRequest_readWrite_registered(
file: file, buffer: buffer, offset: offset, context: context, request: request)
case .readSlot(let file, let buffer, let offset, let context):
request.operation = .readFixed
return makeRawRequest_readWrite_registered_slot(
file: file, buffer: buffer, offset: offset, context: context, request: request)
case .readUnregistered(let file, let buffer, let offset, let context):
request.operation = .read
return makeRawRequest_readWrite_unregistered(
file: file, buffer: buffer, offset: offset, context: context, request: request)
case .readUnregisteredSlot(let file, let buffer, let offset, let context):
request.operation = .read
return makeRawRequest_readWrite_unregistered_slot(
file: file, buffer: buffer, offset: offset, context: context, request: request)
case .close(let file, let context):
request.operation = .close
request.fileDescriptor = file
request.rawValue.user_data = context
case .closeSlot(let file, let context):
request.operation = .close
request.rawValue.file_index = UInt32(file.index + 1)
request.rawValue.user_data = context
case .unlinkAt(let atDirectory, let path, let context):
request.operation = .unlinkAt
request.fileDescriptor = atDirectory
request.rawValue.addr = UInt64(
UInt(
bitPattern: path.withPlatformString { ptr in
ptr //this is unsavory, but we keep it alive by storing path alongside it in the request
})
)
request.path = path
request.rawValue.user_data = context
case .cancelContext(let flags, let targetContext):
request.operation = .asyncCancel
request.cancel_flags = flags
request.addr = targetContext
case .cancelFD(let flags, let targetFD):
request.operation = .asyncCancel
request.cancel_flags = flags
request.fileDescriptor = targetFD
case .cancelFDSlot(let flags, let target):
request.operation = .asyncCancel
request.cancel_flags = flags
request.rawValue.fd = Int32(target.index)
case .cancel(let flags):
request.operation = .asyncCancel
request.cancel_flags = flags
}
return request
}
}
#endif
#endif
+886
View File
@@ -0,0 +1,886 @@
#if compiler(>=6.2)
#if os(Linux)
import CSystem
// needed for mmap
#if canImport(Glibc)
import Glibc
#elseif canImport(Musl)
import Musl
#endif
import Synchronization
//This was #defines in older headers, so we redeclare it to get a consistent import
internal enum RegistrationOps: UInt32 {
case registerBuffers = 0
case unregisterBuffers = 1
case registerFiles = 2
case unregisterFiles = 3
case registerEventFD = 4
case unregisterEventFD = 5
case registerFilesUpdate = 6
case registerEventFDAsync = 7
case registerProbe = 8
case registerPersonality = 9
case unregisterPersonality = 10
}
extension UnsafeMutableRawPointer {
func advanced(by offset: UInt32) -> UnsafeMutableRawPointer {
return advanced(by: Int(offset))
}
}
extension UnsafeMutableRawBufferPointer {
func to_iovec() -> iovec {
iovec(iov_base: baseAddress, iov_len: count)
}
}
// all pointers in this struct reference kernel-visible memory
@usableFromInline struct SQRing: ~Copyable {
@usableFromInline let kernelHead: UnsafePointer<Atomic<UInt32>>
@usableFromInline let kernelTail: UnsafePointer<Atomic<UInt32>>
@usableFromInline var userTail: UInt32
// from liburing: the kernel should never change these
// might change in the future with resizable rings?
@usableFromInline let ringMask: UInt32
// let ringEntries: UInt32 - absorbed into array.count
// ring flags bitfield
// currently used by the kernel only in SQPOLL mode to indicate
// when the polling thread needs to be woken up
@usableFromInline let flags: UnsafePointer<Atomic<UInt32>>
// ring array
// maps indexes between the actual ring and the submissionQueueEntries list,
// allowing the latter to be used as a kind of freelist with enough work?
// currently, just 1:1 mapping (0..<n)
@usableFromInline let array: UnsafeMutableBufferPointer<UInt32>
}
@usableFromInline struct CQRing: ~Copyable {
@usableFromInline let kernelHead: UnsafePointer<Atomic<UInt32>>
@usableFromInline let kernelTail: UnsafePointer<Atomic<UInt32>>
@usableFromInline let ringMask: UInt32
@usableFromInline let cqes: UnsafeBufferPointer<io_uring_cqe>
}
@inline(__always) @inlinable
internal func _tryWriteRequest(
_ request: __owned RawIORequest, ring: inout SQRing,
submissionQueueEntries: UnsafeMutableBufferPointer<io_uring_sqe>
)
-> Bool
{
if let entry = _getSubmissionEntry(
ring: &ring, submissionQueueEntries: submissionQueueEntries) {
entry.pointee = request.rawValue
return true
}
return false
}
//TODO: omitting signal mask for now
//Tell the kernel that we've submitted requests and/or are waiting for completions
@inlinable
internal func _enter(
ring: borrowing SQRing,
ringDescriptor: Int32,
numEvents: UInt32,
minCompletions: UInt32,
flags: UInt32
) throws(Errno) -> Int32 {
// Ring always needs enter right now;
// TODO: support SQPOLL here
while true {
let ret = io_uring_enter(ringDescriptor, numEvents, minCompletions, flags, nil)
// error handling:
// EAGAIN / EINTR (try again),
// EBADF / EBADFD / EOPNOTSUPP / ENXIO
// (failure in ring lifetime management, fatal),
// EINVAL (bad constant flag?, fatal),
// EFAULT (bad address for argument from library, fatal)
if ret == -EAGAIN || ret == -EINTR {
//TODO: should we wait a bit on AGAIN?
continue
} else if ret < 0 {
throw(Errno(rawValue: -ret))
} else if _getSubmissionQueueCount(ring: ring) > 0 {
// See https://github.com/axboe/liburing/issues/309, in some cases not all pending requests are submitted
continue
} else {
return ret
}
}
}
@inlinable
internal func _submitRequests(ring: borrowing SQRing, ringDescriptor: Int32) throws(Errno) {
let flushedEvents = _flushQueue(ring: ring)
_ = try _enter(
ring: ring, ringDescriptor: ringDescriptor, numEvents: flushedEvents, minCompletions: 0, flags: 0)
}
@inlinable
internal func _getSubmissionQueueCount(ring: borrowing SQRing) -> UInt32 {
return ring.userTail - ring.kernelHead.pointee.load(ordering: .acquiring)
}
@inlinable
internal func _getRemainingSubmissionQueueCapacity(ring: borrowing SQRing) -> UInt32 {
return UInt32(truncatingIfNeeded: ring.array.count) - _getSubmissionQueueCount(ring: ring)
}
@inlinable
internal func _getUnconsumedCompletionCount(ring: borrowing CQRing) -> UInt32 {
return ring.kernelTail.pointee.load(ordering: .acquiring)
- ring.kernelHead.pointee.load(ordering: .acquiring)
}
@inlinable
internal func _flushQueue(ring: borrowing SQRing) -> UInt32 {
ring.kernelTail.pointee.store(
ring.userTail, ordering: .releasing
)
return _getSubmissionQueueCount(ring: ring)
}
@inlinable
internal func _getSubmissionEntry(
ring: inout SQRing, submissionQueueEntries: UnsafeMutableBufferPointer<io_uring_sqe>
) -> UnsafeMutablePointer<
io_uring_sqe
>? {
let next = ring.userTail &+ 1 //this is expected to wrap
let kernelHead: UInt32 = ring.kernelHead.pointee.load(ordering: .acquiring)
// FEAT: 128-bit event support (not in MVP)
if next - kernelHead <= ring.array.count {
// let sqe = &sq->sqes[(sq->sqe_tail & sq->ring_mask) << shift];
let sqeIndex = Int(
ring.userTail & ring.ringMask
)
let sqe = submissionQueueEntries
.baseAddress.unsafelyUnwrapped
.advanced(by: sqeIndex)
ring.userTail = next
return sqe
}
return nil
}
private func setUpRing(
queueDepth: UInt32, flags: IORing.SetupFlags
) throws(Errno) ->
(params: io_uring_params, ringDescriptor: Int32, ringPtr: UnsafeMutableRawPointer?, ringSize: Int, submissionRingPtr: UnsafeMutableRawPointer?, submissionRingSize: Int, completionRingPtr: UnsafeMutableRawPointer?, completionRingSize: Int, sqes: UnsafeMutableRawPointer) {
var params = io_uring_params()
params.flags = flags.rawValue
var err: Errno? = nil
let ringDescriptor = withUnsafeMutablePointer(to: &params) {
let result = io_uring_setup(queueDepth, $0)
if result < 0 {
err = Errno.current
}
return result
}
if let err {
throw err
}
if params.features & IORING_FEAT_NODROP == 0
{
close(ringDescriptor)
throw Errno.invalidArgument
}
let submitRingSize =
params.sq_off.array
+ params.sq_entries * UInt32(MemoryLayout<UInt32>.size)
let completionRingSize =
params.cq_off.cqes
+ params.cq_entries * UInt32(MemoryLayout<io_uring_cqe>.size)
let ringSize = Int(max(submitRingSize, completionRingSize))
var ringPtr: UnsafeMutableRawPointer!
var sqPtr: UnsafeMutableRawPointer!
var cqPtr: UnsafeMutableRawPointer!
if params.features & IORING_FEAT_SINGLE_MMAP != 0{
ringPtr = mmap(
/* addr: */ nil,
/* len: */ ringSize,
/* prot: */ PROT_READ | PROT_WRITE,
/* flags: */ MAP_SHARED | MAP_POPULATE,
/* fd: */ ringDescriptor,
/* offset: */ __off_t(IORING_OFF_SQ_RING)
)
if ringPtr == MAP_FAILED {
let errno = Errno.current
close(ringDescriptor)
throw errno
}
} else {
sqPtr = mmap(
/* addr: */ nil,
/* len: */ Int(submitRingSize),
/* prot: */ PROT_READ | PROT_WRITE,
/* flags: */ MAP_SHARED | MAP_POPULATE,
/* fd: */ ringDescriptor,
/* offset: */ __off_t(IORING_OFF_SQ_RING)
)
if sqPtr == MAP_FAILED {
let errno = Errno.current
close(ringDescriptor)
throw errno
}
cqPtr = mmap(
/* addr: */ nil,
/* len: */ Int(completionRingSize),
/* prot: */ PROT_READ | PROT_WRITE,
/* flags: */ MAP_SHARED | MAP_POPULATE,
/* fd: */ ringDescriptor,
/* offset: */ __off_t(IORING_OFF_CQ_RING)
)
if cqPtr == MAP_FAILED {
let errno: Errno = Errno.current
close(ringDescriptor)
throw errno
}
}
// map the submission queue
let sqes = mmap(
/* addr: */ nil,
/* len: */ Int(params.sq_entries) * MemoryLayout<io_uring_sqe>.size,
/* prot: */ PROT_READ | PROT_WRITE,
/* flags: */ MAP_SHARED | MAP_POPULATE,
/* fd: */ ringDescriptor,
/* offset: */ __off_t(IORING_OFF_SQES)
)
if sqes == MAP_FAILED {
let errno = Errno.current
if ringPtr != nil {
munmap(ringPtr, ringSize)
} else {
if sqPtr != nil {
munmap(sqPtr, Int(submitRingSize))
}
if cqPtr != nil {
munmap(cqPtr, Int(completionRingSize))
}
}
close(ringDescriptor)
throw errno
}
return (params: params, ringDescriptor: ringDescriptor, ringPtr: ringPtr, ringSize: ringSize, submissionRingPtr: sqPtr, submissionRingSize: Int(submitRingSize), completionRingPtr: cqPtr, completionRingSize: Int(completionRingSize), sqes: sqes!)
}
///IORing provides facilities for
/// * Registering and unregistering resources (files and buffers), an `io_uring` specific variation on Unix file IOdescriptors that improves their efficiency
/// * Registering and unregistering eventfds, which allow asynchronous waiting for completions
/// * Enqueueing IO requests
/// * Dequeueing IO completions
public struct IORing: ~Copyable {
let ringFlags: UInt32
@usableFromInline let ringDescriptor: Int32
@usableFromInline var submissionRing: SQRing
// FEAT: set this eventually
let submissionPolling: Bool = false
@usableFromInline let completionRing: CQRing
@usableFromInline let submissionQueueEntries: UnsafeMutableBufferPointer<io_uring_sqe>
// kept around for unmap / cleanup. TODO: we can save a few words of memory by figuring out how to handle cleanup for non-IORING_FEAT_SINGLE_MMAP better
let ringSize: Int
let ringPtr: UnsafeMutableRawPointer?
let submissionRingSize: Int
let submissionRingPtr: UnsafeMutableRawPointer?
let completionRingSize: Int
let completionRingPtr: UnsafeMutableRawPointer?
@usableFromInline var _registeredFiles: [UInt32]
@usableFromInline var _registeredBuffers: [iovec]
var features = Features(rawValue: 0)
/// RegisteredResource is used via its typealiases, RegisteredFile and RegisteredBuffer. Registering file descriptors and buffers with the IORing allows for more efficient access to them.
public struct RegisteredResource<T> {
public typealias Resource = T
@usableFromInline let resource: T
public let index: Int
@inlinable internal init(
resource: T,
index: Int
) {
self.resource = resource
self.index = index
}
}
public typealias RegisteredFile = RegisteredResource<UInt32>
public typealias RegisteredBuffer = RegisteredResource<iovec>
/// SetupFlags represents configuration options to an IORing as it's being created
public struct SetupFlags: OptionSet, RawRepresentable, Hashable {
public var rawValue: UInt32
@inlinable public init(rawValue: UInt32) {
self.rawValue = rawValue
}
@inlinable public static var pollCompletions: SetupFlags { .init(rawValue: UInt32(1) << 0) } //IORING_SETUP_IOPOLL
@inlinable public static var pollSubmissions: SetupFlags { .init(rawValue: UInt32(1) << 1) } //IORING_SETUP_SQPOLL
//TODO: figure out how to expose IORING_SETUP_SQ_AFF, IORING_SETUP_CQSIZE, IORING_SETUP_ATTACH_WQ
@inlinable public static var clampMaxEntries: SetupFlags { .init(rawValue: UInt32(1) << 4) } //IORING_SETUP_CLAMP
@inlinable public static var startDisabled: SetupFlags { .init(rawValue: UInt32(1) << 6) } //IORING_SETUP_R_DISABLED
@inlinable public static var continueSubmittingOnError: SetupFlags { .init(rawValue: UInt32(1) << 7) } //IORING_SETUP_SUBMIT_ALL
//TODO: do we want to expose IORING_SETUP_COOP_TASKRUN and IORING_SETUP_TASKRUN_FLAG?
//public static var runTasksCooperatively: SetupFlags { .init(rawValue: UInt32(1) << 8) } //IORING_SETUP_COOP_TASKRUN
//TODO: can we even do different size sqe/cqe? It requires a kernel feature, but how do we convince swift to let the types be different sizes?
//internal static var use128ByteSQEs: SetupFlags { .init(rawValue: UInt32(1) << 10) } //IORING_SETUP_SQE128
//internal static var use32ByteCQEs: SetupFlags { .init(rawValue: UInt32(1) << 11) } //IORING_SETUP_CQE32
@inlinable public static var singleSubmissionThread: SetupFlags { .init(rawValue: UInt32(1) << 12) } //IORING_SETUP_SINGLE_ISSUER
@inlinable public static var deferRunningTasks: SetupFlags { .init(rawValue: UInt32(1) << 13) } //IORING_SETUP_DEFER_TASKRUN
//pretty sure we don't want to expose IORING_SETUP_NO_MMAP or IORING_SETUP_REGISTERED_FD_ONLY currently
//TODO: should IORING_SETUP_NO_SQARRAY be the default? do we need to adapt anything to it?
}
/// Initializes an IORing with enough space for `queueDepth` prepared requests and completed operations
public init(queueDepth: UInt32, flags: SetupFlags = []) throws(Errno) {
let (params, tmpRingDescriptor, tmpRingPtr, tmpRingSize, tmpSQPtr, tmpSQSize, tmpCQPtr, tmpCQSize, sqes) = try setUpRing(queueDepth: queueDepth, flags: flags)
// All throws need to be before initializing ivars here to avoid
// "error: conditional initialization or destruction of noncopyable types is not supported;
// this variable must be consistently in an initialized or uninitialized state through every code path"
features = Features(rawValue: params.features)
ringDescriptor = tmpRingDescriptor
ringPtr = tmpRingPtr
ringSize = tmpRingSize
submissionRingPtr = tmpSQPtr
submissionRingSize = tmpSQSize
completionRingPtr = tmpCQPtr
completionRingSize = tmpCQSize
_registeredFiles = []
_registeredBuffers = []
submissionRing = SQRing(
kernelHead: UnsafePointer<Atomic<UInt32>>(
(ringPtr ?? submissionRingPtr!).advanced(by: params.sq_off.head)
.assumingMemoryBound(to: Atomic<UInt32>.self)
),
kernelTail: UnsafePointer<Atomic<UInt32>>(
(ringPtr ?? submissionRingPtr!).advanced(by: params.sq_off.tail)
.assumingMemoryBound(to: Atomic<UInt32>.self)
),
userTail: 0, // no requests yet
ringMask: (ringPtr ?? submissionRingPtr!).advanced(by: params.sq_off.ring_mask)
.assumingMemoryBound(to: UInt32.self).pointee,
flags: UnsafePointer<Atomic<UInt32>>(
(ringPtr ?? submissionRingPtr!).advanced(by: params.sq_off.flags)
.assumingMemoryBound(to: Atomic<UInt32>.self)
),
array: UnsafeMutableBufferPointer(
start: (ringPtr ?? submissionRingPtr!).advanced(by: params.sq_off.array)
.assumingMemoryBound(to: UInt32.self),
count: Int(
(ringPtr ?? submissionRingPtr!).advanced(by: params.sq_off.ring_entries)
.assumingMemoryBound(to: UInt32.self).pointee)
)
)
// fill submission ring array with 1:1 map to underlying SQEs
for i in 0 ..< submissionRing.array.count {
submissionRing.array[i] = UInt32(i)
}
submissionQueueEntries = UnsafeMutableBufferPointer(
start: sqes.assumingMemoryBound(to: io_uring_sqe.self),
count: Int(params.sq_entries)
)
completionRing = CQRing(
kernelHead: UnsafePointer<Atomic<UInt32>>(
(ringPtr ?? completionRingPtr!).advanced(by: params.cq_off.head)
.assumingMemoryBound(to: Atomic<UInt32>.self)
),
kernelTail: UnsafePointer<Atomic<UInt32>>(
(ringPtr ?? completionRingPtr!).advanced(by: params.cq_off.tail)
.assumingMemoryBound(to: Atomic<UInt32>.self)
),
ringMask: (ringPtr ?? completionRingPtr!).advanced(by: params.cq_off.ring_mask)
.assumingMemoryBound(to: UInt32.self).pointee,
cqes: UnsafeBufferPointer(
start: (ringPtr ?? completionRingPtr!).advanced(by: params.cq_off.cqes)
.assumingMemoryBound(to: io_uring_cqe.self),
count: Int(
(ringPtr ?? completionRingPtr!).advanced(by: params.cq_off.ring_entries)
.assumingMemoryBound(to: UInt32.self).pointee)
)
)
self.ringFlags = params.flags
}
@inlinable
internal func _blockingConsumeCompletionGuts<Err: Error>(
minimumCount: UInt32,
maximumCount: UInt32,
extraArgs: UnsafeMutablePointer<swift_io_uring_getevents_arg>? = nil,
consumer: (consuming IORing.Completion?, Errno?, Bool) throws(Err) -> Void
) throws(Err) {
var count = 0
while let completion = _tryConsumeCompletion(ring: completionRing) {
count += 1
if completion.result < 0 {
try consumer(nil, Errno(rawValue: -completion.result), false)
} else {
try consumer(completion, nil, false)
}
if count == maximumCount {
try consumer(nil, nil, true)
return
}
}
if count < minimumCount {
while count < minimumCount {
var sz = 0
if extraArgs != nil {
sz = MemoryLayout<swift_io_uring_getevents_arg>.size
}
let res = io_uring_enter2(
ringDescriptor,
0,
minimumCount,
IORING_ENTER_GETEVENTS,
extraArgs,
sz
)
// error handling:
// EAGAIN / EINTR (try again),
// EBADF / EBADFD / EOPNOTSUPP / ENXIO
// (failure in ring lifetime management, fatal),
// EINVAL (bad constant flag?, fatal),
// EFAULT (bad address for argument from library, fatal)
// EBUSY (not enough space for events; implies events filled
// by kernel between kernelTail load and now)
if res >= 0 || res == -EBUSY {
break
} else if res == -EAGAIN || res == -EINTR {
continue
}
fatalError(
"fatal error in receiving requests: "
+ Errno(rawValue: -res).debugDescription
)
}
var count = 0
while let completion = _tryConsumeCompletion(ring: completionRing) {
count += 1
if completion.result < 0 {
try consumer(nil, Errno(rawValue: -completion.result), false)
} else {
try consumer(completion, nil, false)
}
if count == maximumCount {
break
}
}
try consumer(nil, nil, true)
}
}
@inlinable
internal func _blockingConsumeOneCompletion(
extraArgs: UnsafeMutablePointer<swift_io_uring_getevents_arg>? = nil
) throws(Errno) -> Completion {
var result: Completion? = nil
try _blockingConsumeCompletionGuts(minimumCount: 1, maximumCount: 1, extraArgs: extraArgs) {
(completion: consuming Completion?, error, done) throws(Errno) in
if let error {
throw error
}
if let completion {
result = consume completion
}
}
return result.take()!
}
/// Synchronously waits for an operation to complete for up to `timeout` (or forever if not specified)
@inlinable
public func blockingConsumeCompletion(
timeout: Duration? = nil
) throws(Errno) -> Completion {
if let timeout {
var ts = timespec(
tv_sec: Int(timeout.components.seconds),
tv_nsec: Int(timeout.components.attoseconds / 1_000_000_000)
)
return try withUnsafePointer(to: &ts) { (tsPtr) throws(Errno) -> Completion in
var args = swift_io_uring_getevents_arg(
sigmask: 0,
sigmask_sz: 0,
min_wait_usec: 0,
ts: UInt64(UInt(bitPattern: tsPtr))
)
return try _blockingConsumeOneCompletion(extraArgs: &args)
}
} else {
return try _blockingConsumeOneCompletion()
}
}
/// Synchronously waits for `minimumCount` or more operations to complete for up to `timeout` (or forever if not specified). For each completed operation found, `consumer` is called to handle processing it
@inlinable
public func blockingConsumeCompletions<Err: Error>(
minimumCount: UInt32 = 1,
timeout: Duration? = nil,
consumer: (consuming Completion?, Errno?, Bool) throws(Err) -> Void
) throws(Err) {
if let timeout {
var ts = timespec(
tv_sec: Int(timeout.components.seconds),
tv_nsec: Int(timeout.components.attoseconds / 1_000_000_000)
)
try withUnsafePointer(to: &ts) { (tsPtr) throws(Err) in
var args = swift_io_uring_getevents_arg(
sigmask: 0,
sigmask_sz: 0,
min_wait_usec: 0,
ts: UInt64(UInt(bitPattern: tsPtr))
)
try _blockingConsumeCompletionGuts(
minimumCount: minimumCount, maximumCount: UInt32.max, extraArgs: &args,
consumer: consumer)
}
} else {
try _blockingConsumeCompletionGuts(
minimumCount: minimumCount, maximumCount: UInt32.max, consumer: consumer)
}
}
// public func peekNextCompletion() -> IOCompletion {
// }
/// Takes a completed operation from the ring and returns it, if one is ready. Otherwise, returns nil.
@inlinable
public func tryConsumeCompletion() -> Completion? {
return _tryConsumeCompletion(ring: completionRing)
}
@inlinable
func _tryConsumeCompletion(ring: borrowing CQRing) -> Completion? {
let tail = ring.kernelTail.pointee.load(ordering: .acquiring)
let head = ring.kernelHead.pointee.load(ordering: .acquiring)
if tail != head {
// 32 byte copy - oh well
let res = ring.cqes[Int(head & ring.ringMask)]
ring.kernelHead.pointee.store(head &+ 1, ordering: .releasing)
return Completion(rawValue: res)
}
return nil
}
/// Registers an event monitoring file descriptor with the ring. The file descriptor becomes readable whenever completions are ready to be dequeued. See `man eventfd(2)` for additional information.
public mutating func registerEventFD(_ descriptor: FileDescriptor) throws(Errno) {
var rawfd = descriptor.rawValue
let result = withUnsafePointer(to: &rawfd) { fdptr in
let result = io_uring_register(
ringDescriptor,
RegistrationOps.registerEventFD.rawValue,
UnsafeMutableRawPointer(mutating: fdptr),
1
)
return result >= 0 ? nil : Errno(rawValue: -result)
}
if let result {
throw result
}
}
/// Removes a registered event file descriptor from the ring
public mutating func unregisterEventFD() throws(Errno) {
let result = io_uring_register(
ringDescriptor,
RegistrationOps.unregisterEventFD.rawValue,
nil,
0
)
if result < 0 {
throw Errno(rawValue: -result)
}
}
/// Registers `count` files with the ring for later use in IO operations
public mutating func registerFileSlots(count: Int) throws(Errno) -> RegisteredResources<RegisteredFile.Resource> {
precondition(_registeredFiles.isEmpty)
precondition(count < UInt32.max)
let files = [UInt32](repeating: UInt32.max, count: count)
let regResult = files.withUnsafeBufferPointer { bPtr in
let result = io_uring_register(
self.ringDescriptor,
RegistrationOps.registerFiles.rawValue,
UnsafeMutableRawPointer(mutating: bPtr.baseAddress!),
UInt32(truncatingIfNeeded: count)
)
return result >= 0 ? nil : Errno(rawValue: -result)
}
if let regResult {
throw regResult
}
_registeredFiles = files
return registeredFileSlots
}
/// Removes registered files from the ring
public func unregisterFiles() throws {
let result = io_uring_register(
ringDescriptor,
RegistrationOps.unregisterFiles.rawValue,
nil,
0
)
if result < 0 {
throw Errno(rawValue: -result)
}
}
/// Allows access to registered files by index
@inlinable
public var registeredFileSlots: RegisteredResources<RegisteredFile.Resource> {
RegisteredResources(resources: _registeredFiles)
}
/// Registers buffers with the ring for later use in IO operations
public mutating func registerBuffers(_ buffers: some Collection<UnsafeMutableRawBufferPointer>) throws(Errno)
-> RegisteredResources<RegisteredBuffer.Resource>
{
precondition(buffers.count < UInt32.max)
precondition(_registeredBuffers.isEmpty)
let iovecs = buffers.map { $0.to_iovec() }
let regResult = iovecs.withUnsafeBufferPointer { bPtr in
let result = io_uring_register(
self.ringDescriptor,
RegistrationOps.registerBuffers.rawValue,
UnsafeMutableRawPointer(mutating: bPtr.baseAddress!),
UInt32(truncatingIfNeeded: buffers.count)
)
return result >= 0 ? nil : Errno(rawValue: -result)
}
if let regResult {
throw regResult
}
_registeredBuffers = iovecs
return registeredBuffers
}
/// Registers buffers with the ring for later use in IO operations
@inlinable
public mutating func registerBuffers(_ buffers: UnsafeMutableRawBufferPointer...) throws(Errno)
-> RegisteredResources<RegisteredBuffer.Resource>
{
try registerBuffers(buffers)
}
/// A view of the registered files or buffers in a ring
public struct RegisteredResources<T>: RandomAccessCollection {
@usableFromInline let resources: [T]
@inlinable public var startIndex: Int { 0 }
@inlinable public var endIndex: Int { resources.endIndex }
@inlinable init(resources: [T]) {
self.resources = resources
}
@inlinable public subscript(position: Int) -> RegisteredResource<T> {
RegisteredResource(resource: resources[position], index: position)
}
@inlinable public subscript(position: UInt16) -> RegisteredResource<T> {
RegisteredResource(resource: resources[Int(position)], index: Int(position))
}
}
/// Allows access to registered files by index
@inlinable
public var registeredBuffers: RegisteredResources<RegisteredBuffer.Resource> {
RegisteredResources(resources: _registeredBuffers)
}
public func unregisterBuffers() throws {
let result = io_uring_register(
self.ringDescriptor,
RegistrationOps.unregisterBuffers.rawValue,
nil,
0
)
guard result >= 0 else {
throw Errno(rawValue: -result)
}
}
/// Sends all prepared requests to the kernel for processing. Results will be delivered as completions, which can be dequeued from the ring.
@inlinable
public func submitPreparedRequests() throws(Errno) {
try _submitRequests(ring: submissionRing, ringDescriptor: ringDescriptor)
}
/// Sends all prepared requests to the kernel for processing, and then dequeues at least `minimumCount` completions, waiting up to `timeout` for them to become available. `consumer` is called to process each completed IO operation as it becomes available.
@inlinable
public func submitPreparedRequestsAndConsumeCompletions<Err: Error>(
minimumCount: UInt32 = 1,
timeout: Duration? = nil,
consumer: (consuming Completion?, Errno?, Bool) throws(Err) -> Void
) throws(Err) {
//TODO: optimize this to one uring_enter
do {
try submitPreparedRequests()
} catch (let e) {
try consumer(nil, e, true)
}
try blockingConsumeCompletions(
minimumCount: minimumCount,
timeout: timeout,
consumer: consumer
)
}
/// Attempts to prepare an IO request for submission to the kernel. Returns false if no space is available to enqueue the request
@inlinable
public mutating func prepare(request: __owned Request) -> Bool {
var raw: RawIORequest? = request.makeRawRequest()
return _tryWriteRequest(
raw.take()!, ring: &submissionRing, submissionQueueEntries: submissionQueueEntries)
}
/// Attempts to prepare a chain of linked IO requests for submission to the kernel. Returns false if not enough space is available to enqueue the request. If any linked operation fails, subsequent operations will be canceled. Linked operations always execute in order.
@inlinable
mutating func prepare(linkedRequests: some BidirectionalCollection<Request>) -> Bool {
guard linkedRequests.count > 0 else {
return true
}
let freeSQECount = _getRemainingSubmissionQueueCapacity(ring: submissionRing)
guard freeSQECount >= linkedRequests.count else {
return false
}
let last = linkedRequests.last!
for req in linkedRequests.dropLast() {
var raw = req.makeRawRequest()
raw.linkToNextRequest()
let successfullyAdded = _tryWriteRequest(
raw, ring: &submissionRing, submissionQueueEntries: submissionQueueEntries)
assert(successfullyAdded)
}
let successfullyAdded = _tryWriteRequest(
last.makeRawRequest(), ring: &submissionRing,
submissionQueueEntries: submissionQueueEntries)
assert(successfullyAdded)
return true
}
/// Prepares a sequence of requests for submission to the ring. Returns false if the submission queue doesn't have enough available space.
@inlinable
public mutating func prepare(linkedRequests: Request...) -> Bool {
prepare(linkedRequests: linkedRequests)
}
/// Prepares and submits a sequence of requests to the ring. Returns false if the submission queue doesn't have enough available space.
@inlinable
public mutating func submit(linkedRequests: Request...) throws(Errno) -> Bool {
if !prepare(linkedRequests: linkedRequests) {
return false
}
try submitPreparedRequests()
return true
}
/// Describes which io_uring features are supported by the kernel this program is running on
public struct Features: OptionSet, RawRepresentable, Hashable {
public let rawValue: UInt32
@inlinable public init(rawValue: UInt32) {
self.rawValue = rawValue
}
//IORING_FEAT_SINGLE_MMAP is handled internally
@inlinable public static var nonDroppingCompletions: Features { .init(rawValue: UInt32(1) << 1) } //IORING_FEAT_NODROP
@inlinable public static var stableSubmissions: Features { .init(rawValue: UInt32(1) << 2) } //IORING_FEAT_SUBMIT_STABLE
@inlinable public static var currentFilePosition: Features { .init(rawValue: UInt32(1) << 3) } //IORING_FEAT_RW_CUR_POS
@inlinable public static var assumingTaskCredentials: Features { .init(rawValue: UInt32(1) << 4) } //IORING_FEAT_CUR_PERSONALITY
@inlinable public static var fastPolling: Features { .init(rawValue: UInt32(1) << 5) } //IORING_FEAT_FAST_POLL
@inlinable public static var epoll32BitFlags: Features { .init(rawValue: UInt32(1) << 6) } //IORING_FEAT_POLL_32BITS
@inlinable public static var pollNonFixedFiles: Features { .init(rawValue: UInt32(1) << 7) } //IORING_FEAT_SQPOLL_NONFIXED
@inlinable public static var extendedArguments: Features { .init(rawValue: UInt32(1) << 8) } //IORING_FEAT_EXT_ARG
@inlinable public static var nativeWorkers: Features { .init(rawValue: UInt32(1) << 9) } //IORING_FEAT_NATIVE_WORKERS
@inlinable public static var resourceTags: Features { .init(rawValue: UInt32(1) << 10) } //IORING_FEAT_RSRC_TAGS
@inlinable public static var allowsSkippingSuccessfulCompletions: Features { .init(rawValue: UInt32(1) << 11) } //IORING_FEAT_CQE_SKIP
@inlinable public static var improvedLinkedFiles: Features { .init(rawValue: UInt32(1) << 12) } //IORING_FEAT_LINKED_FILE
@inlinable public static var registerRegisteredRings: Features { .init(rawValue: UInt32(1) << 13) } //IORING_FEAT_REG_REG_RING
@inlinable public static var minimumTimeout: Features { .init(rawValue: UInt32(1) << 15) } //IORING_FEAT_MIN_TIMEOUT
@inlinable public static var bundledSendReceive: Features { .init(rawValue: UInt32(1) << 14) } //IORING_FEAT_RECVSEND_BUNDLE
}
/// Describes which io_uring features are supported by the kernel this program is running on
public var supportedFeatures: Features {
return features
}
deinit {
if let ringPtr {
munmap(ringPtr, ringSize)
} else if let submissionRingPtr, let completionRingPtr {
munmap(submissionRingPtr, submissionRingSize)
munmap(completionRingPtr, completionRingSize)
}
munmap(
UnsafeMutableRawPointer(submissionQueueEntries.baseAddress!),
submissionQueueEntries.count * MemoryLayout<io_uring_sqe>.size
)
close(ringDescriptor)
}
}
extension IORing.RegisteredBuffer {
@unsafe @inlinable public var unsafeBuffer: UnsafeMutableRawBufferPointer {
return .init(start: resource.iov_base, count: resource.iov_len)
}
@inlinable public var mutableBytes: MutableRawSpan {
@_lifetime(&self)
mutating get {
let span = MutableRawSpan(_unsafeBytes: unsafeBuffer)
return unsafe _overrideLifetime(span, mutating: &self)
}
}
@inlinable public var bytes: RawSpan {
let span = RawSpan(_unsafeBytes: UnsafeRawBufferPointer(unsafeBuffer))
return unsafe _overrideLifetime(span, borrowing: self)
}
}
#endif
#endif
+203
View File
@@ -0,0 +1,203 @@
#if compiler(>=6.2)
#if os(Linux)
import CSystem
@usableFromInline
internal struct RawIORequest: ~Copyable {
@usableFromInline var rawValue: io_uring_sqe
@usableFromInline var path: FilePath? //buffer owner for the path pointer that the sqe may have
@inlinable public init() {
self.rawValue = io_uring_sqe()
}
}
extension RawIORequest {
@usableFromInline
enum Operation: UInt8 {
case nop = 0
case readv = 1
case writev = 2
case fsync = 3
case readFixed = 4
case writeFixed = 5
case pollAdd = 6
case pollRemove = 7
case syncFileRange = 8
case sendMessage = 9
case receiveMessage = 10
// ...
case asyncCancel = 14
case link_timeout = 15
// ...
case openAt = 18
case close = 19
case filesUpdate = 20
case statx = 21
case read = 22
case write = 23
// ...
case openAt2 = 28
// ...
case unlinkAt = 36
}
public struct Flags: OptionSet, Hashable, Codable {
public let rawValue: UInt8
@inlinable public init(rawValue: UInt8) {
self.rawValue = rawValue
}
@inlinable public static var fixedFile: RawIORequest.Flags { Flags(rawValue: 1 << 0) }
@inlinable public static var drainQueue: RawIORequest.Flags { Flags(rawValue: 1 << 1) }
@inlinable public static var linkRequest: RawIORequest.Flags { Flags(rawValue: 1 << 2) }
@inlinable public static var hardlinkRequest: RawIORequest.Flags { Flags(rawValue: 1 << 3) }
@inlinable public static var asynchronous: RawIORequest.Flags { Flags(rawValue: 1 << 4) }
@inlinable public static var selectBuffer: RawIORequest.Flags { Flags(rawValue: 1 << 5) }
@inlinable public static var skipSuccess: RawIORequest.Flags { Flags(rawValue: 1 << 6) }
}
@inlinable var operation: Operation {
get { Operation(rawValue: rawValue.opcode)! }
set { rawValue.opcode = newValue.rawValue }
}
@inlinable var cancel_flags: UInt32 {
get { rawValue.cancel_flags }
set { rawValue.cancel_flags = newValue }
}
@inlinable var addr: UInt64 {
get { rawValue.addr }
set { rawValue.addr = newValue }
}
@inlinable public var flags: Flags {
get { Flags(rawValue: rawValue.flags) }
set { rawValue.flags = newValue.rawValue }
}
@inlinable public mutating func linkToNextRequest() {
flags = Flags(rawValue: flags.rawValue | Flags.linkRequest.rawValue)
}
@inlinable public var fileDescriptor: FileDescriptor {
get { FileDescriptor(rawValue: rawValue.fd) }
set { rawValue.fd = newValue.rawValue }
}
@inlinable public var offset: UInt64? {
get {
if (rawValue.off == UInt64.max) {
return nil
} else {
return rawValue.off
}
}
set {
if let val = newValue {
rawValue.off = val
} else {
rawValue.off = UInt64.max
}
}
}
@inlinable public var buffer: UnsafeMutableRawBufferPointer {
get {
let ptr = UnsafeMutableRawPointer(bitPattern: UInt(exactly: rawValue.addr)!)
return UnsafeMutableRawBufferPointer(start: ptr, count: Int(rawValue.len))
}
set {
rawValue.addr = UInt64(Int(bitPattern: newValue.baseAddress!))
rawValue.len = UInt32(exactly: newValue.count)!
}
}
public enum RequestFlags {
case readWriteFlags(ReadWriteFlags)
// case fsyncFlags(FsyncFlags?)
// poll_events
// poll32_events
// sync_range_flags
// msg_flags
case timeoutFlags(TimeOutFlags)
// accept_flags
// cancel_flags
case openFlags(FileDescriptor.OpenOptions)
// statx_flags
// fadvise_advice
// splice_flags
}
public struct ReadWriteFlags: OptionSet, Hashable, Codable {
public var rawValue: UInt32
@inlinable public init(rawValue: UInt32) {
self.rawValue = rawValue
}
@inlinable public static var highPriority: RawIORequest.ReadWriteFlags { ReadWriteFlags(rawValue: 1 << 0) }
// sync with only data integrity
@inlinable public static var dataSync: RawIORequest.ReadWriteFlags { ReadWriteFlags(rawValue: 1 << 1) }
// sync with full data + file integrity
@inlinable public static var fileSync: RawIORequest.ReadWriteFlags { ReadWriteFlags(rawValue: 1 << 2) }
// return -EAGAIN if operation blocks
@inlinable public static var noWait: RawIORequest.ReadWriteFlags { ReadWriteFlags(rawValue: 1 << 3) }
// append to end of the file
@inlinable public static var append: RawIORequest.ReadWriteFlags { ReadWriteFlags(rawValue: 1 << 4) }
}
public struct TimeOutFlags: OptionSet, Hashable, Codable {
public var rawValue: UInt32
@inlinable public init(rawValue: UInt32) {
self.rawValue = rawValue
}
@inlinable public static var relativeTime: RawIORequest.TimeOutFlags { TimeOutFlags(rawValue: 0) }
@inlinable public static var absoluteTime: RawIORequest.TimeOutFlags { TimeOutFlags(rawValue: 1 << 0) }
}
}
extension RawIORequest {
@inlinable
static func nop() -> RawIORequest {
var req: RawIORequest = RawIORequest()
req.operation = .nop
return req
}
@inlinable
static func withTimeoutRequest<R>(
linkedTo opEntry: UnsafeMutablePointer<io_uring_sqe>,
in timeoutEntry: UnsafeMutablePointer<io_uring_sqe>,
duration: Duration,
flags: TimeOutFlags,
work: () throws -> R) rethrows -> R {
opEntry.pointee.flags |= Flags.linkRequest.rawValue
opEntry.pointee.off = 1
var ts = timespec(
tv_sec: Int(duration.components.seconds),
tv_nsec: Int(duration.components.attoseconds / 1_000_000_000)
)
return try withUnsafePointer(to: &ts) { tsPtr in
var req: RawIORequest = RawIORequest()
req.operation = .link_timeout
req.rawValue.timeout_flags = flags.rawValue
req.rawValue.len = 1
req.rawValue.addr = UInt64(UInt(bitPattern: tsPtr))
timeoutEntry.pointee = req.rawValue
return try work()
}
}
}
#endif
#endif
+1 -1
View File
@@ -130,7 +130,7 @@ internal var mockingEnabled: Bool {
@inline(__always)
internal var forceWindowsPaths: Bool? {
#if !ENABLE_MOCKING
return false
return nil
#else
return MockingDriver.forceWindowsPaths
#endif
@@ -35,17 +35,17 @@ internal func open(
bInheritHandle: decodedFlags.bInheritHandle
)
let hFile = CreateFileW(path,
decodedFlags.dwDesiredAccess,
DWORD(FILE_SHARE_DELETE
| FILE_SHARE_READ
| FILE_SHARE_WRITE),
&saAttrs,
decodedFlags.dwCreationDisposition,
decodedFlags.dwFlagsAndAttributes,
nil)
if hFile == INVALID_HANDLE_VALUE {
guard let hFile = try? path.withCanonicalPathRepresentation({ path in
CreateFileW(path,
decodedFlags.dwDesiredAccess,
DWORD(FILE_SHARE_DELETE
| FILE_SHARE_READ
| FILE_SHARE_WRITE),
&saAttrs,
decodedFlags.dwCreationDisposition,
decodedFlags.dwFlagsAndAttributes,
nil)
}), hFile != INVALID_HANDLE_VALUE else {
ucrt._set_errno(_mapWindowsErrorToErrno(GetLastError()))
return -1
}
@@ -77,17 +77,17 @@ internal func open(
bInheritHandle: decodedFlags.bInheritHandle
)
let hFile = CreateFileW(path,
decodedFlags.dwDesiredAccess,
DWORD(FILE_SHARE_DELETE
| FILE_SHARE_READ
| FILE_SHARE_WRITE),
&saAttrs,
decodedFlags.dwCreationDisposition,
decodedFlags.dwFlagsAndAttributes,
nil)
if hFile == INVALID_HANDLE_VALUE {
guard let hFile = try? path.withCanonicalPathRepresentation({ path in
CreateFileW(path,
decodedFlags.dwDesiredAccess,
DWORD(FILE_SHARE_DELETE
| FILE_SHARE_READ
| FILE_SHARE_WRITE),
&saAttrs,
decodedFlags.dwCreationDisposition,
decodedFlags.dwFlagsAndAttributes,
nil)
}), hFile != INVALID_HANDLE_VALUE else {
ucrt._set_errno(_mapWindowsErrorToErrno(GetLastError()))
return -1
}
@@ -242,7 +242,7 @@ internal func mkdir(
bInheritHandle: false
)
if !CreateDirectoryW(path, &saAttrs) {
guard (try? path.withCanonicalPathRepresentation({ path in CreateDirectoryW(path, &saAttrs) })) == true else {
ucrt._set_errno(_mapWindowsErrorToErrno(GetLastError()))
return -1
}
@@ -254,7 +254,7 @@ internal func mkdir(
internal func rmdir(
_ path: UnsafePointer<CInterop.PlatformChar>
) -> CInt {
if !RemoveDirectoryW(path) {
guard (try? path.withCanonicalPathRepresentation({ path in RemoveDirectoryW(path) })) == true else {
ucrt._set_errno(_mapWindowsErrorToErrno(GetLastError()))
return -1
}
@@ -14,6 +14,9 @@ import XCTest
#else
@testable import System
#endif
#if canImport(Android)
import Android
#endif
@available(/*System 0.0.1: macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0*/iOS 8, *)
final class FileOperationsTest: XCTestCase {
@@ -174,6 +174,9 @@ final class FileOperationsTestWindows: XCTestCase {
/// Test that the umask works properly
func testUmask() throws {
// See https://learn.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/persistent-storage#permissions
try XCTSkipIf(NSUserName() == "ContainerAdministrator", "containers use a different permission model")
// Default mask should be 0o022
XCTAssertEqual(FilePermissions.creationMask, [.groupWrite, .otherWrite])
@@ -205,6 +208,9 @@ final class FileOperationsTestWindows: XCTestCase {
/// Test that setting permissions on a file works as expected
func testPermissions() throws {
// See https://learn.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/persistent-storage#permissions
try XCTSkipIf(NSUserName() == "ContainerAdministrator", "containers use a different permission model")
try FilePermissions.withCreationMask([]) {
try withTemporaryFilePath(basename: "testPermissions") { path in
let tests = [
+3
View File
@@ -14,6 +14,9 @@ import SystemPackage
#else
import System
#endif
#if canImport(Android)
import Android
#endif
@available(/*System 0.0.1: macOS 11.0, iOS 14.0, watchOS 7.0, tvOS 14.0*/iOS 8, *)
final class FileDescriptorTest: XCTestCase {
+34
View File
@@ -0,0 +1,34 @@
#if compiler(>=6.2)
#if os(Linux)
import XCTest
#if SYSTEM_PACKAGE
@testable import SystemPackage
#else
import System
#endif
func requestBytes(_ request: consuming RawIORequest) -> [UInt8] {
return withUnsafePointer(to: request.rawValue) {
let requestBuf = UnsafeBufferPointer(start: $0, count: 1)
let rawBytes = UnsafeRawBufferPointer(requestBuf)
return .init(rawBytes)
}
}
// This test suite compares various IORequests bit-for-bit to IORequests
// that were generated with liburing or manually written out,
// which are known to work correctly.
final class IORequestTests: XCTestCase {
func testNop() {
let req = IORing.Request.nop().makeRawRequest()
let sourceBytes = requestBytes(req)
// convenient property of nop: it's all zeros!
// for some unknown reason, liburing sets the fd field to -1.
// we're not trying to be bug-compatible with it, so 0 *should* work.
XCTAssertEqual(sourceBytes, .init(repeating: 0, count: 64))
}
}
#endif
#endif
+132
View File
@@ -0,0 +1,132 @@
#if compiler(>=6.2)
#if os(Linux)
import XCTest
import CSystem //for eventfd
#if SYSTEM_PACKAGE
import SystemPackage
#else
import System
#endif
func uringEnabled() throws -> Bool {
do {
let procPath = FilePath("/proc/sys/kernel/io_uring_disabled")
let fd = try FileDescriptor.open(procPath, .readOnly)
let buffer = UnsafeMutableRawBufferPointer.allocate(byteCount: 1024, alignment: 0)
_ = try fd.read(into: buffer)
if buffer.load(fromByteOffset: 0, as: Int.self) == 0 {
return true
}
} catch (_) {
return false
}
return false
}
final class IORingTests: XCTestCase {
func testInit() throws {
guard try uringEnabled() else { return }
_ = try IORing(queueDepth: 32, flags: [])
}
func testNop() throws {
guard try uringEnabled() else { return }
var ring = try IORing(queueDepth: 32, flags: [])
_ = try ring.submit(linkedRequests: .nop())
let completion = try ring.blockingConsumeCompletion()
XCTAssertEqual(completion.result, 0)
}
func makeHelloWorldFile() throws -> (dir: FileDescriptor, file: FilePath) {
mkdir("/tmp/IORingTests/", 0o777)
let path: FilePath = "/tmp/IORingTests/test.txt"
let fd = try FileDescriptor.open(
path,
.readWrite,
options: .create,
permissions: .ownerReadWrite
)
try fd.writeAll("Hello, World!".utf8)
try fd.close()
let parent = try FileDescriptor.open("/tmp/IORingTests/", .readOnly)
return (parent, path)
}
func cleanUpHelloWorldFile(_ parent: FileDescriptor) throws {
try parent.close()
rmdir("/tmp/IORingTests/")
}
func setupTestRing(depth: Int, fileSlots: Int, buffers: [UnsafeMutableRawBufferPointer]) throws -> IORing {
var ring: IORing = try IORing(queueDepth: UInt32(depth))
_ = try ring.registerFileSlots(count: 1)
_ = try ring.registerBuffers(buffers)
return ring
}
func testUndersizedSubmissionQueue() throws {
guard try uringEnabled() else { return }
var ring: IORing = try IORing(queueDepth: 1)
let enqueued = ring.prepare(linkedRequests: .nop(), .nop())
XCTAssertFalse(enqueued)
}
// Exercises opening, reading, closing, registered files, registered buffers, and eventfd
func testOpenReadAndWriteFixedFile() throws {
guard try uringEnabled() else { return }
let (parent, path) = try makeHelloWorldFile()
let rawBuffer = UnsafeMutableRawBufferPointer.allocate(byteCount: 13, alignment: 16)
var ring = try setupTestRing(depth: 6, fileSlots: 1, buffers: [rawBuffer])
let eventFD = FileDescriptor(rawValue: eventfd(0, Int32(EFD_SEMAPHORE)))
try ring.registerEventFD(eventFD)
//Part 1: read the file we just created, and make sure the eventfd fires
let enqueued = try ring.submit(linkedRequests:
.open(path, in: parent, into: ring.registeredFileSlots[0], mode: .readOnly),
.read(ring.registeredFileSlots[0], into: ring.registeredBuffers[0]),
.close(ring.registeredFileSlots[0]))
XCTAssert(enqueued)
let efdBuf = UnsafeMutableRawBufferPointer.allocate(byteCount: 8, alignment: 0)
_ = try eventFD.read(into: efdBuf)
_ = try ring.blockingConsumeCompletion() //open
_ = try eventFD.read(into: efdBuf)
_ = try ring.blockingConsumeCompletion() //read
_ = try eventFD.read(into: efdBuf)
_ = try ring.blockingConsumeCompletion() //close
let result = String(cString: rawBuffer.assumingMemoryBound(to: CChar.self).baseAddress!)
XCTAssertEqual(result, "Hello, World!")
//Part 2: delete that file, then use the ring to write out a new one
let rmResult = path.withPlatformString {
remove($0)
}
XCTAssertEqual(rmResult, 0)
let enqueued2 = try ring.submit(linkedRequests:
.open(path, in: parent, into: ring.registeredFileSlots[0], mode: .readWrite, options: .create, permissions: .ownerReadWrite),
.write(ring.registeredBuffers[0], into: ring.registeredFileSlots[0]),
.close(ring.registeredFileSlots[0]))
XCTAssert(enqueued2)
_ = try eventFD.read(into: efdBuf)
_ = try ring.blockingConsumeCompletion() //open
_ = try eventFD.read(into: efdBuf)
_ = try ring.blockingConsumeCompletion() //write
_ = try eventFD.read(into: efdBuf)
_ = try ring.blockingConsumeCompletion() //close
memset(rawBuffer.baseAddress!, 0, rawBuffer.count)
//Verify using a non-ring IO method that what we wrote matches our expectations
print("about to open")
let nonRingFD = try FileDescriptor.open(path, .readOnly)
let bytesRead = try nonRingFD.read(into: rawBuffer)
XCTAssert(bytesRead == 13)
let result2 = String(cString: rawBuffer.assumingMemoryBound(to: CChar.self).baseAddress!)
XCTAssertEqual(result2, "Hello, World!")
try cleanUpHelloWorldFile(parent)
efdBuf.deallocate()
rawBuffer.deallocate()
}
}
#endif
#endif