feat: add managed replacement and configurable idle waits (0.8.0)

CHANGELOG.md

@@ -4,6 +4,20 @@ All notable changes to this package are documented here.
 
 ## Unreleased
 
+## 0.8.0
+
+- **New opt-in replacement of Swarm-managed artifacts: `--allow-replace [selector]` (repeatable) and `ServerOptions.allowReplace`.** Off by default. A bare flag (or `true`) lets an upload replace any managed artifact that holds its name; a selector (or each entry of an iterable) names one exact artifact name or one release template in the grammar of `artifactPatterns`, matched against the final basename: `beta-{version}`, `latest.json`, a bare `{version}` (the names that are SemVer versions, not every name), `{series}-nightly`. A `/` (a source parent is client-asserted and not stored) is refused, every value is validated even beside a bare flag, `history-` names, duplicates, more than 64 selectors and non-strings throw `PROTOCOL_INVALID` with a fixed message that echoes nothing, and a selector that is 64 lowercase hexadecimal characters is still refused by the CLI's secret guard. `server.allowReplace` is `false`, `true` or a frozen copy.
+- **Only managed artifacts, even with every name allowed.** Replacement needs a managed artifact of the same kind whose content still matches its record. An unmanaged file or directory, a managed directory with a child added, changed or removed (an empty directory included), a symlink, and a file where a directory is (or the reverse) are left exactly as they are and the upload gets `FILE_EXISTS`. The replaced artifact stays as `history-<its transfer id>`.
+- **Allowing replacement pins nothing.** Retention is unchanged: `--allow-replace` never pins an artifact against age, count, version, quota or scoped retention. Only the legacy `--replace-name` / `replaceNames` keeps its standing pin on the current artifact. The artifact being replaced is held only for the commit's own pre-commit pass (a temporary pin under the root lease), and an unfinished replacement's journal protects what it owns from retention and from the scrub until recovery finishes.
+- **`--replace-name` / `replaceNames` now also replace a managed directory** (before, a directory was create-only and such an offer was `FILE_EXISTS`). File behavior is unchanged.
+- **Directories are replaced by an atomic exchange with a durable, recoverable transaction.** The new tree is parked at a private path, one exchange puts it at the public name and the old tree at the private path, and the old tree then becomes `history-<id>`: the public path is never missing and never half published, a continuous reader never sees either, and the old tree is never deleted. The journal (version 4) names every path with the inode identity it must hold, and recovery decides from what it observes: a durable new current record means the replacement is completed, otherwise it is undone exactly (the exchange repeated, the new tree back in staging, the session resumable); an unexpected inode, kind, symlink or occupant stops recovery without touching it. When the returning content already has a history copy (A, B, A) that copy is parked privately, with its record, and restored exactly if the replacement does not complete; it is removed only after the new current record is durable. Every move is preceded by a proof of the inode and of the free destination. New `StorageAdapter.atomicSwap(first, second)` (optional; an adapter must exchange atomically or change nothing, never emulate). The default adapter provides it on Linux and macOS through the new dependency `fs-native-extensions` (Holepunch), **loaded only when a directory is actually replaced**: importing the package, a `Client`, or running a server that never replaces a directory does not load the native addon. Windows has no atomic exchange. A storage without it, or a filesystem that refuses it (tested once per layout on scratch directories in the private state), refuses the replacement with `UNSUPPORTED_STORAGE` before anything published changes. File replacement needs none of this.
+- **A busy artifact is retried, not replaced.** While an awaited upload hook or an explicit link request (under `rollback` or `keep`) is using an artifact's public path, and while a queued or running detached hook or an open link rollback journal names it, replacing that name (a file or a directory, `--replace-name` included) is refused with the retryable `FILE_BUSY`, at admission and again under the root lease before anything changes. A hook that already failed, or a request that settled, holds nothing. An automatic link's `afterLink` runs again for the replacement (`LINKED`); an explicit `link` request for the same target runs it (`UNCHANGED`) for the bytes that are there now.
+- **An interrupted A, B, A on a file no longer loses the returning content's history.** Through 0.7 a file replacement that returned content already held as history (A, B, then A again) deleted that history file and its record before the new current record was durable, so a crash or abort in between left neither. The history file and its record now stay exactly as they were until the new current record replaces the history record in one atomic step, and the superseded duplicate file is removed only afterwards, once proven to be a copy of the returning content (by inode within the attempt, by bytes from a journal alone); an abort or a failed or ambiguous write puts the history record back. The returning copy is looked for after the commit's own retention pass, so a pass that legitimately evicts it (quota, age, count) leaves an ordinary replacement instead of failing the commit. No persisted format changed (the version 2 journal and the records are the same), so this adds no state barrier, and a journal an earlier release left, after it had already removed the duplicate, still recovers. But the history record now stays where it was until the new current record is durable, and **0.7.x refuses to start on a pending version 2 journal of such an interrupted replacement** (`Commit sidecar does not match journal`, before it changes anything); it does not recover it. Start 0.8 once to recover, stop it cleanly, and only then roll back to 0.7.x; see the rollback note in `docs/operations.md`. A 0.7.x that meets the journal after the new current record became durable completes it and may leave the superseded duplicate file, which only blocks a later replacement of that history name with `FILE_EXISTS`. Full behavioral parity with 0.7.x on this journal is not claimed.
+- **Downgrade barrier: the first directory replacement makes older releases refuse the state directory.** Releases through 0.7 would quarantine a version 4 journal as corrupt and delete a version 4 record as invalid, and they read only the string `root` of the state directory's record. So before any version 4 journal, record or deduplication step, the state directory's record `.swarm-deploy-state.json` is durably rewritten (a synced private file renamed over the old one, then the directory synced) to `{ "version": 2, "root": { "path": "<root>", "requires": ["directory-replacement-v1"] } }`; an older reader sees a `root` that is not a string and refuses to start, before it reads a journal, a record or a session. This release reads both shapes (and `reset-storage` keeps the record), and refuses a record that requires a feature it does not know. Nothing writes the strict form unless a directory replacement actually begins, and nothing ever removes it. **Once a directory has been replaced, keep this release or restore a backup of the state directory taken before.**
+- **Client `idleTimeout` is 10 minutes by default and configurable up to 2147483647 ms** (`ClientOptions.idleTimeout`, `swarm-deploy upload|link --idle-timeout`). It was 60 seconds and could not exceed 5 minutes. Any invalid value (zero, negative, fractional, above 2147483647) is rejected, never shortened. It bounds one wait, not a transfer: each protocol read, each backpressured write, and the single wait for the final answer (which covers verification, commit and awaited hooks, a directory replacement included) restart with every chunk that moves. The connect timeout is unchanged (30 seconds, at most 5 minutes), and so is the server's own `idleTimeout`. `MAX_IDLE_TIMEOUT` is exported from the client module.
+- **Bare: the no-follow and exclusive-create flags are right on every supported platform and architecture.** Bare's `fs` provides neither `O_NOFOLLOW` nor `O_EXCL`, so every safe open of a regular file (private state, journals, staging, published files, upload sources, and the CLI's seed file reads and `keygen` writes) used fallback numbers, and two were wrong. On Linux ARM (arm64, arm) the x86 `O_NOFOLLOW` value (0x20000) is `O_LARGEFILE`, so a symlink at the final path was followed; on macOS the Linux `O_EXCL` value (0x80) is `O_FSYNC`, so a private-state exclusive create opened an existing file instead of refusing it. 0.7.0 carries the same fallbacks. They now come from one table shared by the storage guards and the CLI, keyed by platform and architecture (Darwin: `O_NOFOLLOW` 0x100, `O_CREAT` 0x200, `O_EXCL` 0x800; Linux: `O_NOFOLLOW` 0x20000 on x64 and ia32 and 0x8000 on arm64 and arm, `O_CREAT` 0x40, `O_EXCL` 0x80). The runtime's own constants still win where it has them, a provided value that is not a positive integer is ignored, and any other platform or architecture refuses to open with the same error as before instead of guessing. Node is unaffected (its constants are complete), and no option, API or persisted format changed.
+- **Wire protocol and persisted formats.** No wire field or version changes: a 0.6.3 (or earlier) client uploads, and replaces, against a 0.8.0 server that allows it. New persisted forms are the version 4 history directory record, the version 4 journal, and the strict state record above; none is written unless a directory is replaced.
+
 ## 0.7.0
 
 - **The free-disk reserve no longer counts bytes twice.** Admission compared the free space with the reserve plus the full reservation (TAR plus extracted payload) of every open session plus the new upload, so a session whose TAR bytes were already on disk was counted both in the free space and in its reservation. It now adds up only what open sessions can still write: a receiving session's remaining durable TAR plus its full payload (the payload stays owed through extraction), and nothing for a verified or deleting session. `reservedBytes`, `maxStagingBytes`, the 1 GiB default, the staging cap, restart behavior, and the retention of the TAR until commit are unchanged. **This does not reduce the peak of a fresh upload**, which still needs its TAR and its extracted payload at once, and it does not free any space.

README.md

@@ -30,8 +30,10 @@ protocol for stored artifacts.
   the client verifies the stored prefix.
 - **Verified:** files and directories are sent as deterministic TAR and checked
   with SHA-256 before an explicit terminal `COMMITTED` result.
-- **Safe storage:** names are create-only by default, named files can opt in to
-  replacement with history, and unmanaged files are never overwritten or deleted.
+- **Safe storage:** names are create-only by default, managed files and whole
+  directories can opt in to replacement with history (`--allow-replace`,
+  `--replace-name`; a directory is exchanged atomically, 0.8.0), and unmanaged
+  files and paths are never overwritten or deleted.
 - **Retention and rotation:** age and storage-quota limits, plus count and
   SemVer rotation driven by artifact patterns.
 - **Managed symlinks:** automatic newest-match aliases and explicit client
@@ -285,9 +287,14 @@ Upload a file:
 
 ```sh
 swarm-deploy upload --seed-file client.seed --server-key "$SERVER_KEY" \
-  --idle-timeout 60000 ./artifact.bin
+  --idle-timeout 600000 ./artifact.bin
 ```
 
+`--idle-timeout` is how long one wait for the server may stay silent, in
+milliseconds: 600000 (ten minutes) by default from 0.8.0, and up to 2147483647. It is
+not a deadline for the whole transfer, and the single wait for the final answer
+covers verification, commit and awaited hooks.
+
 Successful output is:
 
 ```text

SECURITY.md

@@ -51,6 +51,15 @@ and disclosure may use a private GitHub Security Advisory.
   Stop the server cleanly before backup or restore, preserve the output root
   together with its state directory, and let the same or newer package version
   complete recovery before serving traffic.
+- Replacing managed artifacts is opt-in (`--allow-replace`, `--replace-name`) and
+  only ever replaces what the server itself published and can still prove
+  unchanged: unmanaged paths, symlinks, modified directories and artifacts of the
+  other kind are left untouched, whatever is allowed. The first time a directory is
+  replaced, the state directory's record is durably rewritten in a form that 0.7.x
+  and older refuse at startup, because those releases would quarantine or delete
+  the new journal and records. Do not hand such a state directory to an older
+  release: use this release, or restore a backup taken before the first directory
+  replacement.
 - Run every instance and `reset-storage` for one output root as the same user, with
   the same `HOME` and the same `--state-dir`. The storage lock lives in the private
   state directory, because nothing private may sit in the output root, so it only

docs/api.md

@@ -139,7 +139,27 @@ Optional operational limits and policies:
   that are already on disk are no longer counted twice. It does not reduce the
   peak of a fresh upload, which still needs its TAR and its extracted payload
   at once; `maxStagingBytes` and the reservations themselves are unchanged.
-- `maxAge`, `maxStorageBytes`, `replaceNames`
+- `maxAge`, `maxStorageBytes`
+- `replaceNames?: Iterable<string>`: exact names an upload may replace, files and
+  (from 0.8.0) managed directories. The current artifact of each name is pinned
+  against every retention stage (the legacy, standing pin). Readable as
+  `server.replaceNames`.
+- `allowReplace?: boolean | Iterable<string>` (0.8.0, off by default; readable as
+  `server.allowReplace`: `false`, `true`, or a frozen copy of the selectors): the
+  opt-in permission to replace Swarm-managed artifacts. `true` allows every managed
+  name; a list holds selectors, each an exact name or a release template in the
+  grammar of `artifactPatterns` (`beta-{version}`, a bare `{version}` for
+  SemVer-named artifacts only) matched against the final basename; a `/`, a
+  `history-` name, a duplicate, more than 64 selectors or anything that is not a
+  string throws `PROTOCOL_INVALID` ("Invalid allowReplace"). Only a managed artifact
+  of the same kind whose content still matches its record is ever replaced: an
+  unmanaged path, a modified directory, a symlink or the other kind of artifact is
+  left untouched (`FILE_EXISTS`). It pins nothing against retention. A directory is
+  replaced by an atomic exchange and needs a storage adapter whose `atomicSwap`
+  provides one (below); without it the replacement is refused with
+  `UNSUPPORTED_STORAGE` before anything published changes. While a hook or link
+  request is using the artifact's path the upload is refused with the retryable
+  `FILE_BUSY`. See [Replacing managed artifacts](operations.md#replacing-managed-artifacts).
 - `artifactPatterns?: Iterable<string>`, `maxCount?: number`,
   `maxVersions?: number`, and `versionGranularity?: 'major' | 'minor'`; see
   [Artifact patterns and rotation](artifact-patterns.md).
@@ -164,6 +184,22 @@ Optional operational limits and policies:
   waits for it. See
   [Response and detached hooks](hooks.md#response-and-detached-hooks).
 
+`ServerOptions.storage` replaces the runtime filesystem with a `StorageAdapter`.
+From 0.8.0 an adapter may offer one optional capability, `atomicSwap(first, second)`:
+exchange two existing paths on one filesystem in a single atomic step, so that
+afterwards each name designates the other's former inode with every entry intact
+and no observer ever sees either name missing or half populated. An adapter must do
+exactly that or throw having changed nothing; a copy, or several renames, is not an
+exchange and must not be offered under this name. A server only calls it to replace
+a managed directory (and once per layout, on scratch directories, to learn whether
+the filesystem supports it); an adapter without it, or one that reports
+`ENOTSUP`/`EINVAL`, makes a directory replacement fail with `UNSUPPORTED_STORAGE`
+before anything published changes, and file replacement is unaffected. The default
+adapter provides it on Linux and macOS from `fs-native-extensions`, loaded only when
+a directory is first replaced: importing the package, a `Client`, or a server that
+never replaces a directory does not load the native addon. Windows has no atomic
+exchange, so the default adapter offers none there.
+
 The root package exports the `ServerHooks`, `HookModes`, `HookResponse`,
 `OperationOutcome`, `FailedOutcome`, `BeforeCommitContext`,
 `AfterCommitContext`, `AfterLinkContext`, `HookFailureContext`,
@@ -226,7 +262,7 @@ const client = new Client({
   seed: process.env.SWARM_DEPLOY_CLIENT_SEED!,
   serverPublicKey: parsePublicKey(process.env.SWARM_DEPLOY_SERVER_KEY!),
   connectTimeout: 30_000,
-  idleTimeout: 60_000
+  idleTimeout: 600_000
 })
 
 const result = await client.upload('./dist/release.tar.gz')
@@ -240,7 +276,16 @@ string. `serverPublicKey` is a Buffer produced by `parsePublicKey`. Optional
 values are `connectTimeout`, `idleTimeout`, `includeSourceParent`, `dht`,
 `dhtFactory`, and `logger`.
 
-`connectTimeout` defaults to 30 seconds. `idleTimeout` defaults to 60 seconds.
+`connectTimeout` defaults to 30 seconds and is at most 5 minutes. `idleTimeout`
+defaults to 10 minutes (600000 from 0.8.0, 60000 before) and takes a positive
+integer up to 2147483647 milliseconds, the longest delay a timer accepts; any other
+value is rejected with `PROTOCOL_INVALID` rather than clamped (a longer delay would
+fire at once). It bounds one wait for the server, not the whole transfer: each
+protocol read, each backpressured chunk write, and the single wait for the final
+answer while the server verifies, commits and runs awaited hooks, and every chunk
+that moves restarts it. A client that gives up first (`UPLOAD_IDLE_TIMEOUT`) leaves
+the server to finish; repeating the identical upload returns `ALREADY_COMMITTED` if
+it did. The server's own `idleTimeout` is separate and unchanged.
 `includeSourceParent` defaults to `true`; set it to `false` to suppress the
 [source parent](artifact-patterns.md#source-parent) for every upload the client
 makes. Calling `close()` aborts pending work, closes active sockets, and is