fix: deliver actionable deployment failures and reduce repeated warnings

CHANGELOG.md

@@ -4,6 +4,16 @@ All notable changes to this package are documented here.
 
 ## Unreleased
 
+## 0.6.3
+
+- **Failures now reach the client with the server's own reason.** Through 0.6.2 the server destroyed the connection right behind every failure answer except `HOOK_FAILED`, so a client could see only `connection reset by peer` for an ordinary failure such as `DISK_RESERVE` or a failure while receiving. Now every terminal failure answer that was written (admission `REJECTED`, `FAILED` final, `FAILED` link result) is followed by ending the stream and a bounded 5-second grace, as `HOOK_FAILED` already was; a rejected offer's connection is no longer left open for the peer to close. The lingering connection holds only its connection slot, for at most that long. An answer that could not be written, an unauthorized or over-capacity peer, and a closing server still destroy the connection at once. No wire field, version, authentication, resume, checksum, publication, retention, or hook behavior changes. This does not create disk space or change any limit; it only reports them.
+- **Unwrapped raw upload errors are no longer reported as `PROTOCOL_INVALID`.** Through 0.6.2 any raw (non-`SwarmDeployError`) error in an upload was reported as `PROTOCOL_INVALID`, so a server fault could look like an invalid request. Now an explicit `SwarmDeployError` always keeps its stable code: a malformed record or mismatched transfer ID stays `PROTOCOL_INVALID`, as does a peer's own stream failing, and so does a failure that the storage layer wraps in its own `PROTOCOL_INVALID` guard (for example opening or checking a staging file or directory, rolling back a failed append, or legacy state in the output root), even when a known errno lies under it. Only a raw error that reaches the server unwrapped is classified: a known errno maps to an existing code (`ENOSPC`/`EDQUOT` to `DISK_RESERVE`; `EXDEV`/`ENOSYS`/`ENOTSUP` to `UNSUPPORTED_STORAGE`; `EACCES`/`EPERM`/`EROFS`/`EIO` to `COMMIT_FAILED`, whose explanation is now "could not write or commit"), and anything else is the one new stable code `SERVER_ERROR` with a fixed explanation (`ERRORS.SERVER_ERROR`). The fixed explanation of `PROTOCOL_INVALID` now says it can be a protocol or a storage-state check failure, so it no longer implies that the request or the client version was at fault; the server log's `phase` and `systemCode` say more. Link requests are unchanged (an unclassified failure is `LINK_FAILED`). No record field or version changes and the answer still carries only the code. **A client older than 0.6.3 does not know `SERVER_ERROR` and fails closed**: it rejects that answer as an invalid record and cannot show the code, so read the server log for it; requests that succeed are unaffected, and the mapped codes already existed. `Direct upload failed` and `Direct link failed` now also log the `phase` and, from a short fixed list, the errno name as `systemCode`, never an error message, path, stack, or hook text. `Client` still reports a local raw exception as before.
+- **A client reads a failure answer sent while it is still sending.** A server that fails an upload while receiving now reaches the client as that code even under backpressure: the client keeps reading while it sends, stops sending when the server answers, and reports the answer. Local failures (changed source, an abort) and answers that are unexpected or unreadable (including a completion before the payload is in) are never replaced or taken for success, and the wait for an answer after a failed send is bounded.
+- **Errors say where they came from.** `Client` errors carry `remote` (the server's own answer), `transport` (the connection failed or closed with no reason), `operation`, and `phase` (`connect`, `request`, `admission`, `streaming`, `final`), set on the error that was thrown, so types, codes, causes, abort and timeout distinctions are unchanged. Decide on `remote` and `transport`: a local failure (a changed source, an unreadable answer) has neither but may still carry `operation` and `phase`, and an abort carries none of the four. A server answer's message now names the code and a fixed explanation (`Server rejected upload (FILE_EXISTS): ...`), never server text. `RequestOperation` and `RequestPhase` are exported.
+- **CLI.** `upload` prints that message, or for a connection that gave no reason says where it failed and that no reason was received (and that the server-side outcome is unknown), never the stream's own text as a reason; `link` still prints only its stable code for server answers. With `--json`, a failed `upload` or `link` now always prints one object, `{ "code", "origin"?, "phase"?, "hookResult"? }` (`link` adds `target` and `name` first), where it printed nothing unless the server returned `onFailure` data: **scripts that treated empty stdout as the failure signal must use the exit status, which stays `1`.** Nothing from a message, cause, or stack is printed; usage and configuration errors still exit `2` with no JSON. See [Failure output](https://github.com/tetherto/swarm-deploy/blob/main/docs/cli.md#failure-output) and [Troubleshooting failed uploads](https://github.com/tetherto/swarm-deploy/blob/main/docs/operations.md#troubleshooting-failed-uploads).
+- **Limits.** A client key the server does not allow, an over-capacity server, a crash, and a network failure still end the connection with no answer; they are reported as unexplained transport failures, not as a server reason. A server older than 0.6.3 still resets the connection behind most failure answers whatever the client version, so upgrade servers first.
+- **Quieter logs.** The retention scan no longer repeats `Ignoring unknown committed path` for every unmanaged entry on every pass: a name is logged when it first appears (and again if it went away and returned), at most 32 per scan, with the rest counted once in `Ignoring more unknown committed paths`. Invalid managed state and failed scans or operations are still logged every time; unmanaged files are still never adopted or deleted.
+
 ## 0.6.2
 
 - **New `ServerHooks.beforeClose(): void | Promise<void>`, exported by hooks modules as `beforeClose` (minimum 0.6.2).** A general teardown callback so a hook that changes something outside the server can cancel and fully drain that work before the server lets go of its storage. Before, closing aborted the waits on operation hooks and abandoned a callback that was still running, so a publisher could keep mutating an external store after the server had rolled back locally, released the storage lock, and let another server start. It is not an operation hook: it takes no `--hook-mode` or `hookModes` entry (refused as an unknown key), is always awaited, and changes no wire record or the point at which ordinary post-hooks respond. There is no Snap-specific code in the package.

README.md

@@ -310,9 +310,10 @@ the npm package, so these links point to GitHub.
   identity helpers, `Server`, `Client`, link and upload results, events,
   logging, and errors.
 - [Operations, storage, and rollback](https://github.com/tetherto/swarm-deploy/blob/main/docs/operations.md):
-  the production checklist, transfer and resume behavior, storage, replacement,
-  and retention, resetting a storage root, and rollout, compatibility, and
-  rollback.
+  the production checklist, troubleshooting failed uploads (a server's answer
+  versus a connection that gave no reason), transfer and resume behavior,
+  storage, replacement, and retention, resetting a storage root, and rollout,
+  compatibility, and rollback.
 - [Artifact patterns and rotation](https://github.com/tetherto/swarm-deploy/blob/main/docs/artifact-patterns.md):
   pattern templates, source parent, mandatory matching, and count and version
   rotation.

docs/api.md

@@ -475,7 +475,8 @@ link request restores its previous link first unless it asked to keep it (see
 Stable codes include authentication and server-key rejection, invalid
 configuration and protocol records, file and staging limits, disk reserve,
 filename and replacement conflicts, checksum failures, connection or upload
-timeouts, aborts, commit failures, cleanup failures, and managed-link failures
+timeouts, aborts, commit failures, cleanup failures, unexpected server errors
+(`SERVER_ERROR`, 0.6.3), and managed-link failures
 (`LINK_NOT_ALLOWED`, `LINK_TARGET_NOT_FOUND`, `LINK_CONFLICT`, `LINK_FAILED`,
 `UNSUPPORTED_STORAGE`). Manual requests use `LINK_NOT_ALLOWED` for policy,
 self-link, and automatic-name rejection; `LINK_TARGET_NOT_FOUND` for an absent
@@ -485,3 +486,85 @@ reconciliation failure after a durable commit behaves like other post-commit
 deployment failures: the artifact stays committed, the client receives a
 stable failure, and an already-committed retry reruns reconciliation before
 succeeding. Import `ERRORS` rather than matching exception messages.
+
+### How a server reports its own failures (0.6.3)
+
+Through 0.6.2 a server reported every upload error that was not a
+`SwarmDeployError` as `PROTOCOL_INVALID`, so a raw server fault could look like
+an invalid request. From 0.6.3, for **uploads**:
+
+- An explicit `SwarmDeployError` always keeps its stable code. That includes
+  `PROTOCOL_INVALID`, which can mean an invalid request or answer (a malformed
+  record, a transfer ID that does not match, a peer's own stream failing) **or**
+  a check of the server's own storage state: the storage layer can wrap a raw
+  failure in its own `PROTOCOL_INVALID` (for example when it cannot open or check
+  a staging file or directory, or when rolling back a failed append), and the
+  server refuses new work while legacy private state is in the output root. So
+  `PROTOCOL_INVALID` alone does not say that the request, a field, or the client
+  version was at fault; the server log's `phase` and `systemCode` say more.
+- A raw error that reaches the server **unwrapped** and has a known errno is
+  reported as the code that names it: `ENOSPC` and `EDQUOT` as `DISK_RESERVE`;
+  `EXDEV`, `ENOSYS` and `ENOTSUP` as `UNSUPPORTED_STORAGE`; `EACCES`, `EPERM`,
+  `EROFS` and `EIO` as `COMMIT_FAILED` (a write or commit that could not be
+  done). The errno is read from the error or from one level of its `cause`. An
+  errno inside a storage guard's `PROTOCOL_INVALID` is not remapped.
+- Any other unwrapped raw error is the new stable code `SERVER_ERROR`
+  (`ERRORS.SERVER_ERROR`): the server hit an error it did not expect. Its
+  explanation is fixed; no message or cause is sent.
+- Link requests are unchanged: a failure of a link request that has no stable
+  link code is reported as `LINK_FAILED`, so the errno mapping and `SERVER_ERROR`
+  do not apply to link answers.
+- The answer carries only the code, as before: no record field and no version
+  changed. The server log line `Direct upload failed` (and `Direct link failed`)
+  carries `publicKeyId`, `reason`, the `phase` the server was in (`offer`,
+  `transfer`, `verification`, `beforeCommit`, `commit`, `afterCommit`,
+  `afterLink`; `link` or `afterLink` for a link request) and, when the failure
+  has one, `systemCode`: the errno name, from a short fixed list. It never
+  carries the error's message, a path, a stack, or hook text.
+
+A client older than 0.6.3 does not know `SERVER_ERROR`: it rejects the answer as
+an invalid record (`PROTOCOL_INVALID`, raised by the client while decoding) and
+cannot show the code, so look at the server log, which says `SERVER_ERROR` and
+the phase. This fails closed: no request that succeeds is affected, and the
+other codes above already existed. A client of 0.6.3 or newer shows the code.
+
+### Where a client failure came from (0.6.3)
+
+A `Client.upload` or `Client.link` failure says whether the server explained it:
+
+- `error.remote === true`: the server answered and `error.code` is its own
+  stable code (for example `FILE_EXISTS`, `FILE_TOO_LARGE`, `DISK_RESERVE`,
+  `LINK_NOT_ALLOWED`). The message names the code and a fixed explanation of it
+  (`Server rejected upload (DISK_RESERVE): the server does not have enough free
+disk space or quota`); text the server wrote, paths, and hook causes never
+  appear in it.
+  `error.hookResult` is unchanged. This includes a failure the server hit while
+  the client was still sending: the client reads the answer, stops sending, and
+  reports that code, not a local write error.
+- `error.transport === true`: the connection failed, closed, or timed out and
+  **no server reason was received**. The code then says nothing about the
+  server's cause, and nothing says whether the server rolled back or finished:
+  an abrupt close after the whole payload was sent may follow a commit. The
+  error object is the one that was thrown (a raw stream error such as a reset
+  stays a raw error with its own message and code; a `SwarmDeployError` keeps its
+  code and `cause`), with these fields added on it.
+- `error.operation` (`'upload' | 'link'`) and `error.phase`
+  (`'connect' | 'request' | 'admission' | 'streaming' | 'final'`) say where it
+  surfaced, when known: `connect` covers the DHT lookup, the server-key check and
+  the handshake (a client key the server does not allow is dropped before any
+  answer, so it looks the same as an unreachable server); `admission` is waiting
+  for the decision on an upload offer; `streaming` is sending the artifact;
+  `final` is waiting for the terminal answer.
+- An error with neither `remote` nor `transport` (an invalid input, a local
+  checksum or changed-source failure, an answer that could not be read, an abort)
+  is the client's own: it keeps its type and code. It may still carry `operation`
+  and `phase` saying where the client was when it failed, which does not mean the
+  server or the connection was involved, so decide on `remote` and `transport`,
+  not on the presence of `phase`. An abort is never relabelled and carries no
+  `operation` or `phase`.
+
+`RequestOperation` and `RequestPhase` are exported types. The server can only
+answer a client it has authenticated, and only while it can write: a client
+key the server does not allow, an over-capacity server (`maxConnections`), a
+crash, or a network failure end the connection with no answer, and are
+`transport` failures.

docs/cli.md

@@ -193,10 +193,55 @@ seed inline.
   [Returning data in the answer](hooks.md#link-publication-hooks)) and is present
   only when a `response` hook returned something. `--json` only asks for the data
   and formats the output: whether the server waits for a hook is its own
-  declaration. When the request fails and the server's `response` `onFailure`
-  returned data, it is printed as `{ "code", "hookResult": { "onFailure" } }`
-  before the usual stderr diagnostic and exit status `1`. It needs a server of
-  version 0.6.0 or later; an older one rejects the opted-in offer.
+  declaration. It needs a server of version 0.6.0 or later; an older one rejects
+  the opted-in offer.
+- A failed upload exits `1` and prints a diagnostic on stderr; see
+  [Failure output](#failure-output). With `--json` it also prints one JSON
+  failure object on stdout.
+
+### Failure output
+
+A failed `upload` or `link` exits `1`. What it prints depends on whether the
+server explained the failure (see [Errors](api.md#where-a-client-failure-came-from-063)):
+
+- **The server answered** (an authoritative rejection). `upload` prints its
+  stable code and a fixed explanation on stderr, never text the server wrote:
+
+  ```text
+  Server rejected upload (DISK_RESERVE): the server does not have enough free disk space or quota
+  ```
+
+  `link` keeps printing only the stable code (`LINK_CONFLICT`, `HOOK_FAILED`, ...).
+
+- **The connection failed or closed with no answer** (a reset, a close, a
+  timeout, an unauthorized key, a crash). Both commands say where and that no
+  reason was received, and never print the stream's own text as if it were one:
+
+  ```text
+  Upload failed while sending the artifact: the connection failed or closed. No server reason was received, so the outcome on the server is unknown.
+  ```
+
+  Where the artifact may already be committed (`waiting for the final answer`),
+  it says so and that repeating an identical upload reports `ALREADY_COMMITTED`
+  if it was. Nothing is implied about a rollback.
+
+With `--json` a failure also prints one object on one line on stdout (the stderr
+diagnostic and exit status are unchanged):
+
+```json
+{ "code": "DISK_RESERVE", "origin": "server", "phase": "admission" }
+{ "code": null, "origin": "transport", "phase": "streaming" }
+```
+
+`link` adds `target` and `name` first. `code` is the stable code, or `null` when
+the failure carries none (a raw connection reset, or a local failure such as an
+unreadable input). `origin` is `server` for the server's own answer and
+`transport` for a connection that gave no reason; it is absent for a local
+failure. `phase` is `connect`, `request`, `admission`, `streaming`, or `final`
+when known. `hookResult` is added, as before, only when the server's `response`
+`onFailure` returned data. Nothing from an error message, `cause`, stack, or the
+server is included. Configuration and usage errors still exit `2` and print no
+JSON.
 
 Accepted names start with an ASCII letter or digit, contain only letters,
 digits, `.`, `_`, and `-`, and occupy at most 100 UTF-8 bytes. `history-` is
@@ -289,9 +334,10 @@ object has just `target`, `name`, and `status`. It is data the hook chose to
 report, such as an accepted submission's ID, not proof that anything was
 published; see [Link publication hooks](hooks.md#link-publication-hooks). A
 failure prints a stable code on stderr and exits `1`, as without the flag, and
-diagnostics stay on stderr; stdout stays empty unless the server's `response`
-`onFailure` returned data, in which case it is
-`{ "target", "name", "code", "hookResult" }` with the primary code.
+diagnostics stay on stderr; stdout gets one failure object,
+`{ "target", "name", "code", "origin"?, "phase"?, "hookResult"? }`, described in
+[Failure output](#failure-output) (`hookResult` only when the server's `response`
+`onFailure` returned data).
 `--json` only asks for the data and formats the output; it never decides whether
 the server waits for a hook. `--idle-timeout` is still the
 whole time budget. Without `--json` the command asks for nothing and prints

docs/hooks.md

@@ -487,12 +487,14 @@ seeds, and TAR bytes:
   finalized or restored as the hook decides; the artifact stays durable), so a
   retry is safe. Set the client idle timeout above the worst-case total, and
   make publishers fail on their own well before it.
-- **The answer is delivered.** After writing a `HOOK_FAILED` answer the server
-  ends its side of the stream and keeps the connection for up to 5 seconds, so
-  the client reads the code instead of a connection reset. This applies to every
-  `HOOK_FAILED` answer, including `beforeCommit` and `afterCommit`. Other
-  failure answers are still followed by an immediate close, as before, so on a
-  real network a client may see a reset rather than their stable code.
+- **The answer is delivered.** After writing a failure answer the server ends
+  its side of the stream and keeps the connection for up to 5 seconds, so the
+  client reads the code instead of a connection reset. This applies to every
+  terminal failure answer that was written, `HOOK_FAILED` (including
+  `beforeCommit` and `afterCommit`) and every other code alike (0.6.3; through
+  0.6.2 only `HOOK_FAILED` was delivered this way and the others could reach a
+  client as a reset). An answer that could not be written, and a server that is
+  closing, still end the connection at once.
 - **A failed link request is rolled back by default.** The link is made
   durable before the hook runs, so the hook sees it. When the hook then fails,
   returns a result that cannot be sent, or is cut off because the server is