Introduce parallel removal of items, instead of only doing a sequential
removal (and discovery) one-by-one
### Motivation:
#2933 mentioned this function call appears to be slow. If we attempt to
delete a directory with lots of subdirectories, we end up discovering
each subdirectory one by one on a single thread.
### Modifications:
This PR refactors the logic to remove items. Instead of sequentially
discovering and deleting items one by one, we traverse all directories
up to a maximum concurrency, and issue file deletions in parallel within
each directory.
### Result:
The new concurrent implementation was measured to be ~1.4x faster than
`FileManager`, ~1.5x faster than `rm -rf` and ~3x faster than the
current sequential implementation on `main`. This new concurrent
implementation does use the most CPU out of all the other approaches.
Results are broken down into two sections. We delete a copy of the
`swift-nio` repository, where the `.build/` folder has grown to a large
size, providing lots of additional files and subdirectories.
We produce a single CLI using `--configration=release` that can switch
between using the `.sequential` mode (what is currently available on the
`main` branch), `.parallel` (the new concurrent implementation in this
PR) and `filemanager` (uses `Foundation.FileManager` as a comparison).
Resolves https://github.com/apple/swift-nio/issues/2933
#### Normal size
**Description:** The `swift-nio` library as found when building the CLI
used for benchmarking.
**Benchmark run with:** 3579 directories, 11657 files
```
$ hyperfine --runs 10 --warmup 2 --prepare "rm -rf /tmp/swift-nio && cp -r $(pwd) /tmp/swift-nio" ".build/release/NIOTestCLI --sequential /tmp/swift-nio" ".build/release/NIOTestCLI --parallel /tmp/swift-nio" ".build/release/NIOTestCLI --filemanager /tmp/swift-nio" "rm -rf /tmp/swift-nio"
Benchmark 1: .build/release/NIOTestCLI --sequential /tmp/swift-nio
Time (mean ± σ): 2.712 s ± 0.053 s [User: 0.526 s, System: 8.632 s]
Range (min … max): 2.625 s … 2.781 s 10 runs
Benchmark 2: .build/release/NIOTestCLI --parallel /tmp/swift-nio
Time (mean ± σ): 896.7 ms ± 25.6 ms [User: 302.9 ms, System: 5313.5 ms]
Range (min … max): 853.2 ms … 942.1 ms 10 runs
Benchmark 3: .build/release/NIOTestCLI --filemanager /tmp/swift-nio
Time (mean ± σ): 1.337 s ± 0.076 s [User: 0.022 s, System: 1.253 s]
Range (min … max): 1.252 s … 1.508 s 10 runs
Benchmark 4: rm -rf /tmp/swift-nio
Time (mean ± σ): 1.483 s ± 0.037 s [User: 0.015 s, System: 1.416 s]
Range (min … max): 1.413 s … 1.541 s 10 runs
Summary
.build/release/NIOTestCLI --parallel /tmp/swift-nio ran
1.49 ± 0.10 times faster than .build/release/NIOTestCLI --filemanager /tmp/swift-nio
1.65 ± 0.06 times faster than rm -rf /tmp/swift-nio
3.02 ± 0.10 times faster than .build/release/NIOTestCLI --sequential /tmp/swift-nio
```
#### Large size
**Description:** The `swift-nio` library as found after re-running the
`.github` workflow many times locally. I'm unable to reproduce this
directory size, but have benchmarked against it.
**Benchmark run with:** 15260 directories, 46842 files
```
$ hyperfine --runs 10 --warmup 2 --prepare "rm -rf /tmp/swift-nio && cp -r $(pwd) /tmp/swift-nio" ".build/release/NIOTestCLI --sequential /tmp/swift-nio" ".build/release/NIOTestCLI --parallel /tmp/swift-nio" ".build/release/NIOTestCLI --filemanager /tmp/swift-nio" "rm -rf /tmp/swift-nio"
Benchmark 1: .build/release/NIOTestCLI --sequential /tmp/swift-nio
Time (mean ± σ): 12.082 s ± 0.354 s [User: 2.180 s, System: 41.547 s]
Range (min … max): 11.717 s … 12.923 s 10 runs
Benchmark 2: .build/release/NIOTestCLI --parallel /tmp/swift-nio
Time (mean ± σ): 4.770 s ± 0.102 s [User: 1.357 s, System: 25.785 s]
Range (min … max): 4.570 s … 4.883 s 10 runs
Benchmark 3: .build/release/NIOTestCLI --filemanager /tmp/swift-nio
Time (mean ± σ): 6.503 s ± 0.115 s [User: 0.081 s, System: 5.427 s]
Range (min … max): 6.262 s … 6.687 s 10 runs
Benchmark 4: rm -rf /tmp/swift-nio
Time (mean ± σ): 7.056 s ± 0.246 s [User: 0.064 s, System: 6.119 s]
Range (min … max): 6.435 s … 7.233 s 10 runs
Summary
.build/release/NIOTestCLI --parallel /tmp/swift-nio ran
1.36 ± 0.04 times faster than .build/release/NIOTestCLI --filemanager /tmp/swift-nio
1.48 ± 0.06 times faster than rm -rf /tmp/swift-nio
2.53 ± 0.09 times faster than .build/release/NIOTestCLI --sequential /tmp/swift-nio
```
---------
Co-authored-by: George Barnett <gbarnett@apple.com>
Updates to documentation comments in `NIOFileSystem`.
### Motivation:
I've been reviewing some parts of `NIOFileSystem` and came across a
variety of formatting of documentation comments.
### Modifications:
I've adopted a stricter 100-character limit (arbitrary, really; it could
be longer or shorter and is totally not consistent with the rest of the
repository). Also updated a bunch of grammar in comments. Fixed a few
typos along the way.
### Result:
- Updated comments have a better flow while reading, and aren't as
abrupt as before.
- Fewer typos in comments
---------
Co-authored-by: Cory Benfield <lukasa@apple.com>
### Motivation:
Documentation checking catches more issues in Swift 6.0.
### Modifications:
Adopt the Swift 6.0 image and fix the errors.
### Result:
More accurate docs.
Add Vision support
### Motivation:
https://github.com/apple/swift-nio/issues/2900 - close this
### Modifications:
- Bump SystemPackage to 1.4.0 and remove @preconcurrency
- remove `#if os(macOS) || os(iOS) || os(tvOS) || os(watchOS) ||
os(Linux) || os(Android)` to allow for visionOS support
- Remove conditional import for swift system
### Result:
Vision OS support
### Motivation:
Copying regular files on Darwin was taking more time than it needed to
because it was manually opening files and copying over bites rather than
making use of the system's ability to create CoW clones.
### Modifications:
The primary change is to switch `copyItem` to be backed by Darwin's
`copyfile` method rather than `fcopyfile`. `fcopyfile` ignores the flag
we were passing in to indicate that the file should be cloned if
possible, whereas `copyfile` is a higher-level API which takes this into
account.
This change exposes a `copyfile` wrapper method as a new public
function.
This change also adds a workaround to a seeming compiler bug on channel
options blocking building tests on more recent Xcodes.
### Result:
Copying files on Darwin will be faster.
Motivation:
Using some limited parallelism can improved the speed of the copy for very limited increase in cpu use.
In circumstances where the source and/or destination are on a high latency device, such as a network file system,
the resulting speed up can be even more significant.
Providing conservative defaults, while allowing users the ability to control it directly if desired, provides for a faster default and access to user targeted gains in performance.
The use of parallel by default is considered in keeping with the swift-nio model of concurrent capabilities.
The parallelism used is restricted because the underlying limited resource for this is highly likely to be limited system file descriptors, rather than the limits of the underlying executor itself, especially on embedded devices with significantly more limited concurrent descriptors.
Modifications:
All copyItem overloads except ```copyItem(sourcePath:destinationPath:shouldProceedAfterError:shouldCopyFile)```
can, and likely will, use concurrent copies when copying directories.
A Newly type ```CopyStrategy``` controls the behaviour, allowing:
* ```parallel(maxDescriptors: Int)```
* Parallel limited to a maximum number of concurrently open file descriptors.
* ```sequential```
* The previous behaviour, still asynchronous but only one operation at a time
* ```platformDefault```
* Whatever the API determines the default should be from the prior options
The cancellation semantics of copyItem were previously unclear/undocumented.
These are now stated and cancellation is tested.
There were some aspects of naming/types on this API which were inconsistent/unclear. Those have been altered to aid clarity.
* Less duplication with the protocol docs
* Consistent use of `item` not `file`
* Use DirectoryEntry for the source in both ```shouldXxx``` callbacks
NIOFileSystem package status remains unchanged, it is still _NIOFileSystem with the semantics that entails.
Result:
For any use of the following functions on FileSystemProtocol, when backed by FileSystem:
- ```copyItem(*)```:
The shouldCopyItem check happens before the internal FileSystem "do I know how to copy this sort of FileType" check. This makes the code a little simpler, but the desire is to allow consuming code to filter out anything unexpected if they like without needing to trap errors.
- ```copyItem(sourcePath:destinationPath)```:
Depending on the platform this will become parallel, users on MacOS, Linux and Windows will have up to 4 concurrent file system operations at once.
Empirical testing on a MacBook pro with a significant dense directory hierarchy of 12 GiB had performance improvements of ~50% reduction in elapsed time for CPU usage increases below the noise floor of the testing
- ```copyItem(sourcePath:destinationPath:shouldProceedAfterError:shouldCopyFile)```:
When migrating from this to ```copyItem(sourcePath:destinationPath:copyStrategy:shouldProceedAfterError:shouldCopyItem)```care should be taken to ensure the should callbacks are safe to use in different concurrency domains, or that CopyStrategy.sequential is explicitly stated.
The backward compatibility shim provided does this by default.
Motivation:
Get this repo building again for Android with the new overlay
Modifications:
- Import the new module or overlay wherever `Glibc` is used
- Keep this repo building with Swift 5 by duplicating some declarations
Result:
All the same tests keep passing on my Android CI, finagolfin/swift-android-sdk#158
* Apply formatting
* Apply no block comments rule
* Apply OmitExplicitReturns
* Apple OnlyOneTrailingClosureArgument
* Apply NoAssignmentInExpressions
* Fix up DontRepeatTypeInStaticProperties lint errors
* Apply `OrderedImports`
* Apply `ReplaceForEachWithForLoop`
* format file
* Enable the formatting pipeline
* Adopt `AmbiguousTrailingClosureOverload`
* Fix license header
* Fix format check
* Fix `EndOfLineComment`
* Fix CI
* Adapt CI script to check if changes when running formatting
* Separate lint and format into to steps
* Fix format
* Adopt `UseEarlyExits`
* Revert "Adopt `UseEarlyExits`"
This reverts commit d1ac5bbe12.
# Motivation
Next up replacing the documentation check.
# Modification
This moves the current approach of the script into a GH action. I tried just running the documentation check against all targets but that also results in warnings from downstream deps in their docs to show up.
I also fixed up some new errors that appeared since our current pipeline was running on 5.8.
# Result
Next CI job migrated to GHA
Motivation:
The name of the syscall used in errors for all rename calls is "rename".
However, this isn't always correct, in some cases "renamex_np" is used,
and in others "renameat2" is used.
Modifications:
- Allow the name of the syscall to be provided to the rename error
- Use the correct syscall name
Result:
Better errors
Motivation:
Single letter names are discouraged but are used in a handful of places
in NIOFileSystem.
Modifications:
- Replace 'R' with 'ReturnType' where appropriate
Result:
Clearer APIs
Motivation:
Many of the filesystem with methods have a requirement that the
result type is sendable. In most cases this is not required as
the result is not shared or sent.
Modifications:
Remove sendable requirement for result types where it is not required.
Result:
Users will not have to make so many types sendable.
Motivation:
There are two thread pools: one in NIOFileSystem and one in NIOPosix; we
should avoid the duplication.
Modifications:
- Switch to using NIOThreadPool
- Remove the old thread pool and associated code
- Add an init to FileSystem which takes a thread pool
Result:
Less duplication
Motivation:
Get the new module building and tested on my Android CI again
Modifications:
- Add force unwraps where needed
- Update C types and account for nullability annotations in NDK 26
- Use /data/local/tmp for the `temporaryDirectory` and invoke it in the tests, instead of /tmp directly
- Don't copy over extended attributes
- Android doesn't support hard-linking on most partitions, so don't use it for materializing temporary files
Result:
Everything works again on Android
Motivation:
SwiftNIO doesn't compile for visionOS because NIOFileSystem depends on
Swift System which doesn't yet support visionOS.
As NIOFileSystem isn't yet stable API we can define it out on some
platforms.
Modifications:
- Only define NIOFileSystem on macOS, iOS, tvOS, watchOS, Linux and
Android.
- Only include Swift System on the same platforms
Result:
SwiftNIO compiles on visionOS, albeit without NIOFileSystem.
Use direct syscall for `renameat2` because Musl doesn't include
a wrapper for it. Similarly, define the constants ourselves.
Add Musl imports in relevant places.
Motivation:
I/O is typically blocking, but options for interacting with the
filesystem using `async` APIs are limited. This change adds a new module
to NIO, `NIOFileSystem`. At the moment this is exposed as the
`_NIOFileSystem` product while users get a chance to try it out and
provide feedback.
During this time the API of `NIOFileSystem` is subject to change.
Breaking changes will come in _minor_ NIO releases and additive changes
may come in _patch_ releases of NIO.
Modifications:
- Add `NIOFileSystem`, `NIOFileSystemFoundationCompat`,
`NIOFileSystemTests`, `NIOFileSystemIntegrationTests`, and
`NIOFileSystemFoundationCompatTests`.
- Update docs
- Update CNIOLinux and CNIODarwin shims
Result:
File system can be accessed via `async` APIs