Commit Graph
19 Commits
Author SHA1 Message Date
Johannes Weiss 40ee44c6b9 always @preconcurrency import Glibc/Musl/Android/Bionic/WASILibc (#3153)
### Motivation:

The non-`Darwin` libcs don't have the correct concurrency annotations.
But due to these Swift bugs, it's important that the _first_ importer
uses `@preconcurrency`:

- https://github.com/swiftlang/swift/issues/79414
- https://github.com/swiftlang/swift/issues/77866

### Modifications:

Much like the Foundation (& corelibs) PRs such as
https://github.com/swiftlang/swift-foundation/pull/1175 , use
`@preconcurrency import` for the non-`Darwin` libcs.

### Result:

Fewer bad warnings/errors in user code.
2025-03-21 15:35:05 +00:00
1e900b4736 NIOFileSystem: Try ${TMPDIR} first for temporary directory (#3067)
### Motivation:

This PR aligns what temp directory NIOFileSystem will return first:
- First try `${TMPDIR}` as it's an expressed user preference
- On Darwin, try `_CS_DARWIN_USER_TMP_DIR` next. If it's set, return
that.
- Finally, fall back on `/tmp` on Darwin and Linux, and
`/data/local/tmp` on Android.

Closes #2861.

### Modifications:

- Reworks `NIOFileSystem.temporaryDirectory`.

### Result:

- NIOFileSystem will now try `${TMPDIR}` first for the temporary
directory.

### Caveats:

- We might want to align how Swift-NIO and FoundationEssentials resolve
temp directory, and [FoundationEssentials have a different
approach](https://github.com/swiftlang/swift-foundation/blob/9d57f36de757b3d5e3a2f7ffcf27aaec3033509f/Sources/FoundationEssentials/String/String%2BPath.swift#L484-L533).
But, I agree with [this
comment](https://github.com/apple/swift-nio/issues/2861#issuecomment-2327010713),
it feels like TMPDIR is an expected default.

---------

Co-authored-by: Cory Benfield <lukasa@apple.com>
Co-authored-by: George Barnett <gbarnett@apple.com>
2025-01-27 08:45:46 +00:00
Michael GechtandGeorge Barnett 29c65edb97 Add parallel removal of items (#3008)
Introduce parallel removal of items, instead of only doing a sequential
removal (and discovery) one-by-one

### Motivation:

#2933 mentioned this function call appears to be slow. If we attempt to
delete a directory with lots of subdirectories, we end up discovering
each subdirectory one by one on a single thread.

### Modifications:

This PR refactors the logic to remove items. Instead of sequentially
discovering and deleting items one by one, we traverse all directories
up to a maximum concurrency, and issue file deletions in parallel within
each directory.

### Result:

The new concurrent implementation was measured to be ~1.4x faster than
`FileManager`, ~1.5x faster than `rm -rf` and ~3x faster than the
current sequential implementation on `main`. This new concurrent
implementation does use the most CPU out of all the other approaches.

Results are broken down into two sections. We delete a copy of the
`swift-nio` repository, where the `.build/` folder has grown to a large
size, providing lots of additional files and subdirectories.

We produce a single CLI using `--configration=release` that can switch
between using the `.sequential` mode (what is currently available on the
`main` branch), `.parallel` (the new concurrent implementation in this
PR) and `filemanager` (uses `Foundation.FileManager` as a comparison).

Resolves https://github.com/apple/swift-nio/issues/2933

#### Normal size

**Description:** The `swift-nio` library as found when building the CLI
used for benchmarking.
**Benchmark run with:** 3579 directories, 11657 files

```
$ hyperfine --runs 10 --warmup 2 --prepare "rm -rf /tmp/swift-nio && cp -r $(pwd) /tmp/swift-nio" ".build/release/NIOTestCLI --sequential /tmp/swift-nio" ".build/release/NIOTestCLI --parallel /tmp/swift-nio" ".build/release/NIOTestCLI --filemanager /tmp/swift-nio" "rm -rf /tmp/swift-nio"
Benchmark 1: .build/release/NIOTestCLI --sequential /tmp/swift-nio
  Time (mean ± σ):      2.712 s ±  0.053 s    [User: 0.526 s, System: 8.632 s]
  Range (min … max):    2.625 s …  2.781 s    10 runs

Benchmark 2: .build/release/NIOTestCLI --parallel /tmp/swift-nio
  Time (mean ± σ):     896.7 ms ±  25.6 ms    [User: 302.9 ms, System: 5313.5 ms]
  Range (min … max):   853.2 ms … 942.1 ms    10 runs

Benchmark 3: .build/release/NIOTestCLI --filemanager /tmp/swift-nio
  Time (mean ± σ):      1.337 s ±  0.076 s    [User: 0.022 s, System: 1.253 s]
  Range (min … max):    1.252 s …  1.508 s    10 runs

Benchmark 4: rm -rf /tmp/swift-nio
  Time (mean ± σ):      1.483 s ±  0.037 s    [User: 0.015 s, System: 1.416 s]
  Range (min … max):    1.413 s …  1.541 s    10 runs

Summary
  .build/release/NIOTestCLI --parallel /tmp/swift-nio ran
    1.49 ± 0.10 times faster than .build/release/NIOTestCLI --filemanager /tmp/swift-nio
    1.65 ± 0.06 times faster than rm -rf /tmp/swift-nio
    3.02 ± 0.10 times faster than .build/release/NIOTestCLI --sequential /tmp/swift-nio
```

#### Large size

**Description:** The `swift-nio` library as found after re-running the
`.github` workflow many times locally. I'm unable to reproduce this
directory size, but have benchmarked against it.
**Benchmark run with:** 15260 directories, 46842 files

```
$ hyperfine --runs 10 --warmup 2 --prepare "rm -rf /tmp/swift-nio && cp -r $(pwd) /tmp/swift-nio" ".build/release/NIOTestCLI --sequential /tmp/swift-nio" ".build/release/NIOTestCLI --parallel /tmp/swift-nio" ".build/release/NIOTestCLI --filemanager /tmp/swift-nio" "rm -rf /tmp/swift-nio"
Benchmark 1: .build/release/NIOTestCLI --sequential /tmp/swift-nio
  Time (mean ± σ):     12.082 s ±  0.354 s    [User: 2.180 s, System: 41.547 s]
  Range (min … max):   11.717 s … 12.923 s    10 runs

Benchmark 2: .build/release/NIOTestCLI --parallel /tmp/swift-nio
  Time (mean ± σ):      4.770 s ±  0.102 s    [User: 1.357 s, System: 25.785 s]
  Range (min … max):    4.570 s …  4.883 s    10 runs

Benchmark 3: .build/release/NIOTestCLI --filemanager /tmp/swift-nio
  Time (mean ± σ):      6.503 s ±  0.115 s    [User: 0.081 s, System: 5.427 s]
  Range (min … max):    6.262 s …  6.687 s    10 runs

Benchmark 4: rm -rf /tmp/swift-nio
  Time (mean ± σ):      7.056 s ±  0.246 s    [User: 0.064 s, System: 6.119 s]
  Range (min … max):    6.435 s …  7.233 s    10 runs

Summary
  .build/release/NIOTestCLI --parallel /tmp/swift-nio ran
    1.36 ± 0.04 times faster than .build/release/NIOTestCLI --filemanager /tmp/swift-nio
    1.48 ± 0.06 times faster than rm -rf /tmp/swift-nio
    2.53 ± 0.09 times faster than .build/release/NIOTestCLI --sequential /tmp/swift-nio
```

---------

Co-authored-by: George Barnett <gbarnett@apple.com>
2024-12-10 17:43:29 +00:00
Michael GechtandCory Benfield 4d2fb577b9 Update documentation comments (#2998)
Updates to documentation comments in `NIOFileSystem`.

### Motivation:

I've been reviewing some parts of `NIOFileSystem` and came across a
variety of formatting of documentation comments.

### Modifications:

I've adopted a stricter 100-character limit (arbitrary, really; it could
be longer or shorter and is totally not consistent with the rest of the
repository). Also updated a bunch of grammar in comments. Fixed a few
typos along the way.

### Result:

- Updated comments have a better flow while reading, and aren't as
abrupt as before.
- Fewer typos in comments

---------

Co-authored-by: Cory Benfield <lukasa@apple.com>
2024-11-26 14:48:29 +00:00
Rick Newton-Rogers adfd61adc5 Use Swift 6.0 docs pipeline (#2966)
### Motivation:

Documentation checking catches more issues in Swift 6.0.

### Modifications:

Adopt the Swift 6.0 image and fix the errors.

### Result:

More accurate docs.
2024-11-07 13:03:47 +00:00
Ali Ali 9140817010 Issue-2900 - Add VisionOS support (#2947)
Add Vision support

### Motivation:
https://github.com/apple/swift-nio/issues/2900 - close this

### Modifications:
- Bump SystemPackage to 1.4.0 and remove @preconcurrency 
- remove `#if os(macOS) || os(iOS) || os(tvOS) || os(watchOS) ||
os(Linux) || os(Android)` to allow for visionOS support
- Remove conditional import for swift system

### Result:

Vision OS support
2024-10-24 10:26:32 +00:00
Rick Newton-Rogers bd12191411 Clone files on Darwin rather than copying them (#2823)
### Motivation:

Copying regular files on Darwin was taking more time than it needed to
because it was manually opening files and copying over bites rather than
making use of the system's ability to create CoW clones.

### Modifications:

The primary change is to switch `copyItem` to be backed by Darwin's
`copyfile` method rather than `fcopyfile`. `fcopyfile` ignores the flag
we were passing in to indicate that the file should be cloned if
possible, whereas `copyfile` is a higher-level API which takes this into
account.

This change exposes a `copyfile` wrapper method as a new public
function.

This change also adds a workaround to a seeming compiler bug on channel
options blocking building tests on more recent Xcodes.

### Result:

Copying files on Darwin will be faster.
2024-08-05 12:04:52 +01:00
Matt Hope 022ee1328b FileSystem.copyItem can parallelise directory copy (#2806)
Motivation:

Using some limited parallelism can improved the speed of the copy for very limited increase in cpu use.

In circumstances where the source and/or destination are on a high latency device, such as a network file system,
the resulting speed up can be even more significant.
Providing conservative defaults, while allowing users the ability to control it directly if desired, provides for a faster default and access to user targeted gains in performance. 
The use of parallel by default is considered in keeping with the swift-nio model of concurrent capabilities. 

The parallelism used is restricted because the underlying limited resource for this is highly likely to be limited system file descriptors, rather than the limits of the underlying executor itself, especially on embedded devices with significantly more limited concurrent descriptors.

Modifications:

All copyItem overloads except ```copyItem(sourcePath:destinationPath:shouldProceedAfterError:shouldCopyFile)```
can, and likely will, use concurrent copies when copying directories.

A Newly type ```CopyStrategy``` controls the behaviour, allowing:
* ```parallel(maxDescriptors: Int)```
   * Parallel limited to a maximum number of concurrently open file descriptors.
* ```sequential```
   * The previous behaviour, still asynchronous but only one operation at a time
* ```platformDefault``` 
    * Whatever the API determines the default should be from the prior options

The cancellation semantics of copyItem were previously unclear/undocumented. 
These are now stated and cancellation is tested.

There were some aspects of naming/types on this API which were inconsistent/unclear. Those have been altered to aid clarity.
* Less duplication with the protocol docs
* Consistent use of `item` not `file`
* Use DirectoryEntry for the source in both ```shouldXxx``` callbacks

NIOFileSystem package status remains unchanged, it is still _NIOFileSystem with the semantics that entails.

Result:

For any use of the following functions on FileSystemProtocol, when backed by FileSystem:

- ```copyItem(*)```:

The shouldCopyItem check happens before the internal FileSystem "do I know how to copy this sort of FileType" check. This makes the code a little simpler, but the desire is to allow consuming code to filter out anything unexpected if they like without needing to trap errors.

- ```copyItem(sourcePath:destinationPath)```:

Depending on the platform this will become parallel, users on MacOS, Linux and Windows will have up to 4 concurrent file system operations at once.
Empirical testing on a MacBook pro with a significant dense directory hierarchy of 12 GiB had performance improvements of ~50% reduction in elapsed time for CPU usage increases below the noise floor of the testing

- ```copyItem(sourcePath:destinationPath:shouldProceedAfterError:shouldCopyFile)```:
When migrating from this to ```copyItem(sourcePath:destinationPath:copyStrategy:shouldProceedAfterError:shouldCopyItem)```care should be taken to ensure the should callbacks are safe to use in different concurrency domains, or that CopyStrategy.sequential is explicitly stated.
The backward compatibility shim provided does this by default.
2024-07-29 11:14:10 +01:00
finagolfin 4feff3c2f6 Use the new Android overlay and Bionic module from Swift 6 (#2784)
Motivation:

Get this repo building again for Android with the new overlay

Modifications:

- Import the new module or overlay wherever `Glibc` is used
- Keep this repo building with Swift 5 by duplicating some declarations

Result:

All the same tests keep passing on my Android CI, finagolfin/swift-android-sdk#158
2024-07-22 09:58:09 +00:00
Franz Busch c9756e1083 Adopt swift-format (#2794)
* Apply formatting

* Apply no block comments rule

* Apply OmitExplicitReturns

* Apple OnlyOneTrailingClosureArgument

* Apply NoAssignmentInExpressions

* Fix up DontRepeatTypeInStaticProperties lint errors

* Apply `OrderedImports`

* Apply `ReplaceForEachWithForLoop`

* format file

* Enable the formatting pipeline

* Adopt `AmbiguousTrailingClosureOverload`

* Fix license header

* Fix format check

* Fix `EndOfLineComment`

* Fix CI

* Adapt CI script to check if changes when running formatting

* Separate lint and format into to steps

* Fix format

* Adopt `UseEarlyExits`

* Revert "Adopt `UseEarlyExits`"

This reverts commit d1ac5bbe12.
2024-07-19 11:48:17 +02:00
Franz Busch 9730938793 [GHA] Documentation check (#2764)
# Motivation

Next up replacing the documentation check.

# Modification

This moves the current approach of the script into a GH action. I tried just running the documentation check against all targets but that also results in warnings from downstream deps in their docs to show up.

I also fixed up some new errors that appeared since our current pipeline was running on 5.8.

# Result

Next CI job migrated to GHA
2024-07-04 16:21:18 +01:00
George Barnett 9f63b12a05 Imrprove rename error (#2731)
Motivation:

The name of the syscall used in errors for all rename calls is "rename".
However, this isn't always correct, in some cases "renamex_np" is used,
and in others "renameat2" is used.

Modifications:

- Allow the name of the syscall to be provided to the rename error
- Use the correct syscall name

Result:

Better errors
2024-05-29 14:17:52 +01:00
George Barnett 29e832adda Replace 'R' with 'Result' (#2709)
Motivation:

Single letter names are discouraged but are used in a handful of places
in NIOFileSystem.

Modifications:

- Replace 'R' with 'ReturnType' where appropriate

Result:

Clearer APIs
2024-04-29 11:01:17 +01:00
Peter Adams d3bdffd5e4 Remove surplus Sendable requirements from FileSystem with methods (#2706)
Motivation:

Many of the filesystem with methods have a requirement that the
result type is sendable.  In most cases this is not required as
the result is not shared or sent.

Modifications:

Remove sendable requirement for result types where it is not required.

Result:

Users will not have to make so many types sendable.
2024-04-24 10:36:03 +01:00
George Barnett 2d840c586f Use NIOThreadPool in NIOFileSystem (#2692)
Motivation:

There are two thread pools: one in NIOFileSystem and one in NIOPosix; we
should avoid the duplication.

Modifications:

- Switch to using NIOThreadPool
- Remove the old thread pool and associated code
- Add an init to FileSystem which takes a thread pool

Result:

Less duplication
2024-03-26 13:15:27 +00:00
finagolfin 53be20532e Build the new FileSystem module for Android (#2660)
Motivation:

Get the new module building and tested on my Android CI again

Modifications:

- Add force unwraps where needed
- Update C types and account for nullability annotations in NDK 26
- Use /data/local/tmp for the `temporaryDirectory` and invoke it in the tests, instead of /tmp directly
- Don't copy over extended attributes
- Android doesn't support hard-linking on most partitions, so don't use it for materializing temporary files

Result:

Everything works again on Android
2024-03-05 14:46:54 +00:00
George Barnett 53965cc348 Only compile FileSystem on some platforms (#2636)
Motivation:

SwiftNIO doesn't compile for visionOS because NIOFileSystem depends on
Swift System which doesn't yet support visionOS.

As NIOFileSystem isn't yet stable API we can define it out on some
platforms.

Modifications:

- Only define NIOFileSystem on macOS, iOS, tvOS, watchOS, Linux and
  Android.
- Only include Swift System on the same platforms

Result:

SwiftNIO compiles on visionOS, albeit without NIOFileSystem.
2024-02-01 13:24:21 +00:00
Alastair Houghton c14d7d6c9e More changes to support building with Musl. (#2628)
Use direct syscall for `renameat2` because Musl doesn't include
a wrapper for it.  Similarly, define the constants ourselves.

Add Musl imports in relevant places.
2024-01-18 07:44:33 -08:00
George Barnett f80c69f84b Add NIOFilesystem (#2615)
Motivation:

I/O is typically blocking, but options for interacting with the
filesystem using `async` APIs are limited. This change adds a new module
to NIO, `NIOFileSystem`. At the moment this is exposed as the
`_NIOFileSystem` product while users get a chance to try it out and
provide feedback.

During this time the API of `NIOFileSystem` is subject to change.
Breaking changes will come in _minor_ NIO releases and additive changes
may come in _patch_ releases of NIO.

Modifications:

- Add `NIOFileSystem`, `NIOFileSystemFoundationCompat`,
  `NIOFileSystemTests`, `NIOFileSystemIntegrationTests`, and
  `NIOFileSystemFoundationCompatTests`.
- Update docs
- Update CNIOLinux and CNIODarwin shims

Result:

File system can be accessed via `async` APIs
2024-01-16 03:30:26 -08:00