feat: merge directory uploads without replacing sibling artifacts

CHANGELOG.md

@@ -7,16 +7,16 @@ All notable changes to this package are documented here.
 ## 0.8.0
 
 - **New opt-in replacement of Swarm-managed artifacts: `--allow-replace [selector]` (repeatable) and `ServerOptions.allowReplace`.** Off by default. A bare flag (or `true`) lets an upload replace any managed artifact that holds its name; a selector (or each entry of an iterable) names one exact artifact name or one release template in the grammar of `artifactPatterns`, matched against the final basename: `beta-{version}`, `latest.json`, a bare `{version}` (the names that are SemVer versions, not every name), `{series}-nightly`. A `/` (a source parent is client-asserted and not stored) is refused, every value is validated even beside a bare flag, `history-` names, duplicates, more than 64 selectors and non-strings throw `PROTOCOL_INVALID` with a fixed message that echoes nothing, and a selector that is 64 lowercase hexadecimal characters is still refused by the CLI's secret guard. `server.allowReplace` is `false`, `true` or a frozen copy.
-- **Only managed artifacts, even with every name allowed.** Replacement needs a managed artifact of the same kind whose content still matches its record. An unmanaged file or directory, a managed directory with a child added, changed or removed (an empty directory included), a symlink, and a file where a directory is (or the reverse) are left exactly as they are and the upload gets `FILE_EXISTS`. The replaced artifact stays as `history-<its transfer id>`.
+- **File-level directory merges.** Directory uploads add missing leaves to existing containers, preserve omitted files and other platforms, and track ownership per file. Differing intact owned leaves need replacement permission; differing untracked or edited files, symlinks and kind conflicts reject the whole input. Identical untracked content remains untracked. Empty directory entries create nothing; zero-byte files are retained. Standalone file replacement still preserves public history.
 - **Allowing replacement pins nothing.** Retention is unchanged: `--allow-replace` never pins an artifact against age, count, version, quota or scoped retention. Only the legacy `--replace-name` / `replaceNames` keeps its standing pin on the current artifact. The artifact being replaced is held only for the commit's own pre-commit pass (a temporary pin under the root lease), and an unfinished replacement's journal protects what it owns from retention and from the scrub until recovery finishes.
-- **`--replace-name` / `replaceNames` now also replace a managed directory** (before, a directory was create-only and such an offer was `FILE_EXISTS`). File behavior is unchanged.
-- **Directories are replaced by an atomic exchange with a durable, recoverable transaction.** The new tree is parked at a private path, one exchange puts it at the public name and the old tree at the private path, and the old tree then becomes `history-<id>`: the public path is never missing and never half published, a continuous reader never sees either, and the old tree is never deleted. The journal (version 4) names every path with the inode identity it must hold, and recovery decides from what it observes: a durable new current record means the replacement is completed, otherwise it is undone exactly (the exchange repeated, the new tree back in staging, the session resumable); an unexpected inode, kind, symlink or occupant stops recovery without touching it. When the returning content already has a history copy (A, B, A) that copy is parked privately, with its record, and restored exactly if the replacement does not complete; it is removed only after the new current record is durable. Every move is preceded by a proof of the inode and of the free destination. New `StorageAdapter.atomicSwap(first, second)` (optional; an adapter must exchange atomically or change nothing, never emulate). The default adapter provides it on Linux and macOS through the new dependency `fs-native-extensions` (Holepunch), **loaded only when a directory is actually replaced**: importing the package, a `Client`, or running a server that never replaces a directory does not load the native addon. Windows has no atomic exchange. A storage without it, or a filesystem that refuses it (tested once per layout on scratch directories in the private state), refuses the replacement with `UNSUPPORTED_STORAGE` before anything published changes. File replacement needs none of this.
+- **Replacement permissions apply inside directory commits.** `--replace-name`, scoped `--allow-replace` and global `--allow-replace` authorize changed owned leaves under the matching top-level name; adding missing files needs no flag.
+- **Recoverable whole-input commits.** Receive and verify the complete input privately and check every conflict before publication. Version 5 merge journals and private per-file preimages provide whole-commit rollback/restart recovery; the durable owned-file catalog is the commit point. No new public directory history or whole-directory exchange. Concurrent readers can see a mixture of complete old/new files during installation. A pending cleanup journal must converge before a later same-name commit or deletion; otherwise return retryable `FILE_BUSY`. Scrub preserves its catalog generation until cleanup finishes.
 - **A busy artifact is retried, not replaced.** While an awaited upload hook or an explicit link request (under `rollback` or `keep`) is using an artifact's public path, and while a queued or running detached hook or an open link rollback journal names it, replacing that name (a file or a directory, `--replace-name` included) is refused with the retryable `FILE_BUSY`, at admission and again under the root lease before anything changes. A hook that already failed, or a request that settled, holds nothing. An automatic link's `afterLink` runs again for the replacement (`LINKED`); an explicit `link` request for the same target runs it (`UNCHANGED`) for the bytes that are there now.
 - **An interrupted A, B, A on a file no longer loses the returning content's history.** Through 0.7 a file replacement that returned content already held as history (A, B, then A again) deleted that history file and its record before the new current record was durable, so a crash or abort in between left neither. The history file and its record now stay exactly as they were until the new current record replaces the history record in one atomic step, and the superseded duplicate file is removed only afterwards, once proven to be a copy of the returning content (by inode within the attempt, by bytes from a journal alone); an abort or a failed or ambiguous write puts the history record back. The returning copy is looked for after the commit's own retention pass, so a pass that legitimately evicts it (quota, age, count) leaves an ordinary replacement instead of failing the commit. No persisted format changed (the version 2 journal and the records are the same), so this adds no state barrier, and a journal an earlier release left, after it had already removed the duplicate, still recovers. But the history record now stays where it was until the new current record is durable, and **0.7.x refuses to start on a pending version 2 journal of such an interrupted replacement** (`Commit sidecar does not match journal`, before it changes anything); it does not recover it. Start 0.8 once to recover, stop it cleanly, and only then roll back to 0.7.x; see the rollback note in `docs/operations.md`. A 0.7.x that meets the journal after the new current record became durable completes it and may leave the superseded duplicate file, which only blocks a later replacement of that history name with `FILE_EXISTS`. Full behavioral parity with 0.7.x on this journal is not claimed.
-- **Downgrade barrier: the first directory replacement makes older releases refuse the state directory.** Releases through 0.7 would quarantine a version 4 journal as corrupt and delete a version 4 record as invalid, and they read only the string `root` of the state directory's record. So before any version 4 journal, record or deduplication step, the state directory's record `.swarm-deploy-state.json` is durably rewritten (a synced private file renamed over the old one, then the directory synced) to `{ "version": 2, "root": { "path": "<root>", "requires": ["directory-replacement-v1"] } }`; an older reader sees a `root` that is not a string and refuses to start, before it reads a journal, a record or a session. This release reads both shapes (and `reset-storage` keeps the record), and refuses a record that requires a feature it does not know. Nothing writes the strict form unless a directory replacement actually begins, and nothing ever removes it. **Once a directory has been replaced, keep this release or restore a backup of the state directory taken before.**
+- **Per-file catalogs and downgrade barrier.** A private catalog per name owns at most 10,000 leaves (8 MiB, 16 recent receipts). Retention groups versions but removes only unchanged owned leaves and empty containers, preserving foreign siblings. Old whole-tree ownership imports only on exact proof. Before the first new-format journal/catalog, state durably requires `file-catalog-v1`; older readers refuse it before cleanup. Reset archives catalogs and retains required features. Restore public files and private state together from a pre-feature backup to downgrade. Legacy whole-tree journal recovery remains supported.
 - **Client `idleTimeout` is 10 minutes by default and configurable up to 2147483647 ms** (`ClientOptions.idleTimeout`, `swarm-deploy upload|link --idle-timeout`). It was 60 seconds and could not exceed 5 minutes. Any invalid value (zero, negative, fractional, above 2147483647) is rejected, never shortened. It bounds one wait, not a transfer: each protocol read, each backpressured write, and the single wait for the final answer (which covers verification, commit and awaited hooks, a directory replacement included) restart with every chunk that moves. The connect timeout is unchanged (30 seconds, at most 5 minutes), and so is the server's own `idleTimeout`. `MAX_IDLE_TIMEOUT` is exported from the client module.
 - **Bare: the no-follow and exclusive-create flags are right on every supported platform and architecture.** Bare's `fs` provides neither `O_NOFOLLOW` nor `O_EXCL`, so every safe open of a regular file (private state, journals, staging, published files, upload sources, and the CLI's seed file reads and `keygen` writes) used fallback numbers, and two were wrong. On Linux ARM (arm64, arm) the x86 `O_NOFOLLOW` value (0x20000) is `O_LARGEFILE`, so a symlink at the final path was followed; on macOS the Linux `O_EXCL` value (0x80) is `O_FSYNC`, so a private-state exclusive create opened an existing file instead of refusing it. 0.7.0 carries the same fallbacks. They now come from one table shared by the storage guards and the CLI, keyed by platform and architecture (Darwin: `O_NOFOLLOW` 0x100, `O_CREAT` 0x200, `O_EXCL` 0x800; Linux: `O_NOFOLLOW` 0x20000 on x64 and ia32 and 0x8000 on arm64 and arm, `O_CREAT` 0x40, `O_EXCL` 0x80). The runtime's own constants still win where it has them, a provided value that is not a positive integer is ignored, and any other platform or architecture refuses to open with the same error as before instead of guessing. Node is unaffected (its constants are complete), and no option, API or persisted format changed.
-- **Wire protocol and persisted formats.** No wire field or version changes: a 0.6.3 (or earlier) client uploads, and replaces, against a 0.8.0 server that allows it. New persisted forms are the version 4 history directory record, the version 4 journal, and the strict state record above; none is written unless a directory is replaced.
+- **Wire protocol and persisted formats.** No wire field/version change. Old clients can upload and merge with this server. Upload results keep incoming identity; links and link hooks use the aggregate owned-file snapshot, including exact already-committed retries. New private catalogs and version 5 journals are guarded before writing; standalone file formats remain unchanged.
 
 ## 0.7.0
 

README.md

@@ -30,10 +30,10 @@ protocol for stored artifacts.
   the client verifies the stored prefix.
 - **Verified:** files and directories are sent as deterministic TAR and checked
   with SHA-256 before an explicit terminal `COMMITTED` result.
-- **Safe storage:** names are create-only by default, managed files and whole
-  directories can opt in to replacement with history (`--allow-replace`,
-  `--replace-name`; a directory is exchanged atomically, 0.8.0), and unmanaged
-  files and paths are never overwritten or deleted.
+- **File merges:** directory uploads add files recursively, preserving omitted
+  files and other platforms. Replacing changed, Swarm-managed files requires
+  `--allow-replace` or `--replace-name`. Untracked files are never overwritten.
+  The whole input is verified privately before a recoverable commit begins.
 - **Retention and rotation:** age and storage-quota limits, plus count and
   SemVer rotation driven by artifact patterns.
 - **Managed symlinks:** automatic newest-match aliases and explicit client

docs/api.md

@@ -140,26 +140,24 @@ Optional operational limits and policies:
   peak of a fresh upload, which still needs its TAR and its extracted payload
   at once; `maxStagingBytes` and the reservations themselves are unchanged.
 - `maxAge`, `maxStorageBytes`
-- `replaceNames?: Iterable<string>`: exact names an upload may replace, files and
-  (from 0.8.0) managed directories. The current artifact of each name is pinned
-  against every retention stage (the legacy, standing pin). Readable as
-  `server.replaceNames`.
+- `replaceNames?: Iterable<string>`: exact artifact names under which an upload
+  may replace managed files, including files within directory inputs (0.8.0).
+  The current artifact of each name is pinned against retention (the legacy,
+  standing pin). Readable as `server.replaceNames`.
 - `allowReplace?: boolean | Iterable<string>` (0.8.0, off by default; readable as
-  `server.allowReplace`: `false`, `true`, or a frozen copy of the selectors): the
-  opt-in permission to replace Swarm-managed artifacts. `true` allows every managed
-  name; a list holds selectors, each an exact name or a release template in the
-  grammar of `artifactPatterns` (`beta-{version}`, a bare `{version}` for
-  SemVer-named artifacts only) matched against the final basename; a `/`, a
-  `history-` name, a duplicate, more than 64 selectors or anything that is not a
-  string throws `PROTOCOL_INVALID` ("Invalid allowReplace"). Only a managed artifact
-  of the same kind whose content still matches its record is ever replaced: an
-  unmanaged path, a modified directory, a symlink or the other kind of artifact is
-  left untouched (`FILE_EXISTS`). It pins nothing against retention. A directory is
-  replaced by an atomic exchange and needs a storage adapter whose `atomicSwap`
-  provides one (below); without it the replacement is refused with
-  `UNSUPPORTED_STORAGE` before anything published changes. While a hook or link
-  request is using the artifact's path the upload is refused with the retryable
-  `FILE_BUSY`. See [Replacing managed artifacts](operations.md#replacing-managed-artifacts).
+  `server.allowReplace`: `false`, `true`, or a frozen selector list): `true`
+  permits replacement under every artifact name. Each selector is an exact name
+  or a release template in the grammar of `artifactPatterns`, matched against
+  the final basename (`beta-{version}`, a bare `{version}` for SemVer names).
+  A `/`, `history-` name, duplicate, more than 64 selectors or a non-string
+  throws `PROTOCOL_INVALID` ("Invalid allowReplace"). Within an existing
+  directory, missing files can be added without this permission; omitted files
+  remain. A differing file is replaced only if Swarm owns it and its current
+  bytes still match its record. A differing untracked or edited file, a symlink
+  or a kind conflict rejects the whole input with `FILE_EXISTS` before
+  publication. It pins nothing against retention. Active hooks, link requests
+  and unresolved commit cleanup can return retryable `FILE_BUSY`. See
+  [Replacing managed artifacts](operations.md#replacing-managed-artifacts).
 - `artifactPatterns?: Iterable<string>`, `maxCount?: number`,
   `maxVersions?: number`, and `versionGranularity?: 'major' | 'minor'`; see
   [Artifact patterns and rotation](artifact-patterns.md).
@@ -185,20 +183,18 @@ Optional operational limits and policies:
   [Response and detached hooks](hooks.md#response-and-detached-hooks).
 
 `ServerOptions.storage` replaces the runtime filesystem with a `StorageAdapter`.
-From 0.8.0 an adapter may offer one optional capability, `atomicSwap(first, second)`:
-exchange two existing paths on one filesystem in a single atomic step, so that
-afterwards each name designates the other's former inode with every entry intact
-and no observer ever sees either name missing or half populated. An adapter must do
-exactly that or throw having changed nothing; a copy, or several renames, is not an
-exchange and must not be offered under this name. A server only calls it to replace
-a managed directory (and once per layout, on scratch directories, to learn whether
-the filesystem supports it); an adapter without it, or one that reports
-`ENOTSUP`/`EINVAL`, makes a directory replacement fail with `UNSUPPORTED_STORAGE`
-before anything published changes, and file replacement is unaffected. The default
-adapter provides it on Linux and macOS from `fs-native-extensions`, loaded only when
-a directory is first replaced: importing the package, a `Client`, or a server that
-never replaces a directory does not load the native addon. Windows has no atomic
-exchange, so the default adapter offers none there.
+Directory uploads use private staging, a whole-commit journal and per-file
+installation, without exchanging the containing directory. The entire input is
+verified and planned before public changes; failure rolls back the commit, and
+startup resolves interrupted commits. Concurrent readers can see a mixture of
+old and new files during installation; there is no instantaneous multi-file
+namespace switch. Each replaced file remains a complete old or new file. See
+[File-level directory commits](operations.md#replacing-managed-artifacts).
+
+The optional `atomicSwap(first, second)` capability remains for recovery of older
+whole-directory replacement journals. It must exchange two existing paths in one
+atomic step or throw without changing them. The default implementation uses
+`fs-native-extensions` on Linux and macOS; new file merges do not require it.
 
 The root package exports the `ServerHooks`, `HookModes`, `HookResponse`,
 `OperationOutcome`, `FailedOutcome`, `BeforeCommitContext`,
@@ -384,9 +380,11 @@ interface UploadHookResult {
 }
 ```
 
-Files and directories share the same result shape. For a directory, `size` is
-aggregate payload bytes and `entryCount` counts stored entries (files and
-directories).
+Files and directories share the same result shape. For a directory upload,
+`size`, `entryCount`, digest and transfer ID describe its input (files and
+directory entries), not everything already in the destination. Links and their
+hooks use the committed aggregate of owned files instead: its own snapshot ID,
+owned bytes and file count. Untracked siblings are excluded from that aggregate.
 
 `Client.upload(path, { hookResult: true })` asks the server to send back the data
 its `response`-declared hooks returned: `afterCommit`'s object and one `afterLink`