Skip to content

Latest commit

 

History

History
159 lines (111 loc) · 7.39 KB

testing.md

File metadata and controls

159 lines (111 loc) · 7.39 KB
title Write & run tests
permalink /how-to/testing
diataxis how-to
redirect_from
/testing
/testing.md

Write & run tests

This guide shows how to write a *.test.jh file with mocked prompts and stubbed dependencies, and then run it with jaiph test. Each test block runs the def under test with the same interpreter that jaiph run uses, and then checks the output the def produced.

jaiph test runs on the host in-process. It does not run the credential pre-flight, and it does not run hooks.

Mock every prompt step, and stub imported defs or scripts when you need to. If you leave a prompt unmocked, or a queued list of mock prompt "…" responses runs out, the runtime falls through to a real, live prompt call against the configured agent backend, the same way jaiph run would. One difference is that jaiph test turns off prompt retries by default by setting JAIPH_PROMPT_RETRY=0, so a prompt that falls through and errors fails right away instead of retrying on the production schedule. Set JAIPH_PROMPT_RETRY yourself when you want to test the retry behavior.

A pattern-based mock prompt { … } block does not fall through. An unmatched prompt fails the test unless a _ default arm catches it.

Prerequisites

  • The def under test lives in a separate .jh file you can import (recommended, and it keeps test files small).
  • You know the def's parameters and what prompt calls it makes.

1. Create the test file

Test files end in .test.jh. By convention, keep them next to the module under test, or under a top-level tests/ or e2e/ directory.

import "workflow_greeting.jh" as w

test "runs happy path and prints PASS" {
  mock prompt "e2e-greeting-mock"
  const response = w.main()
  expect_contain response "e2e-greeting-mock"
  expect_contain response "done"
}

A test file can have any top-level construct (import, config, def, …), but the CLI only executes test "..." { ... } blocks. # line comments and blank lines are allowed between steps inside a test block.

2. Queue mock prompt responses

mock prompt "first response"
mock prompt "second response"

Multiple mock prompt lines queue in order, and one response is consumed per prompt call. Strings use double quotes. \" is the only escape, and every other backslash stays as written. For multi-line text, declare a test-block const with a """…""" block and refer to it by name. A bare identifier refers to a test-block const declared earlier:

const greeting = "hi"
mock prompt greeting

A queued mock prompt line is the keyword followed by exactly one response, either a "…" string or a bare const name. Any text after the response is E_PARSE (mock prompt must be: …) at the mock keyword's column, like every other test-block statement, so mock prompt "a" x is rejected.

To pick a response based on the content of the prompt, use the pattern form. Do not mix queued mock prompt "…" or mock prompt <const> lines with a mock prompt { … } block in one test, because the compiler rejects that with an E_VALIDATE error. Separate tests in the same file may use different styles:

mock prompt {
  /greeting/ => "hello"
  /farewell/ => "goodbye"
  _ => "default response"
}

Arms are checked from top to bottom, and the first match wins. A /regex/ arm matches when its pattern is found anywhere in the prompt text. A "string" arm matches only when the whole prompt text equals it exactly. Without a _ wildcard arm, an unmatched prompt fails the test. The same block can also be written on one line, with the arms separated by commas, as in mock prompt { "a" => "x", _ => "z" }. A comma inside a "…" response or inside a call's arguments (f(x, y)) does not split the arms. The arms follow the same rules as a compact match. jaiph format keeps a one-line block on one line and a multiline block on multiple lines.

3. (Optional) Stub defs or scripts

Replace a def or script body for this test case. Parentheses are required:

mock def w.greet() {
  return "stubbed greeting"
}

mock def w.validate() {
  return "stubbed validation"
}

mock script w.helper() {
  echo "stubbed script"
}

mock def uses Jaiph steps in the body. mock script uses raw shell, the same as a real script.

The parentheses can also declare parameters. A parameter receives the matching call argument by position, so a mock that reads its arguments should list the same parameters as the real def:

mock def w.greet(name) {
  return "hi, ${name}"
}

An empty () ignores any arguments passed at the call site.

4. Run the def and capture output

const response = w.main()
const response = w.main("my input")            # with one argument
const response = w.main("first", "second")     # comma-separated arguments
const response = w.main() allow_failure        # accept non-zero exit

The call captures the def's return value when the exit code is 0 and the return value is not empty. When the exit code is non-zero, it captures the runtime error string instead. In any other case, it captures the def's *.out files, read in sorted order and joined together.

5. Assert on the captured value

expect_contain     response "expected substring"
expect_not_contain response "unwanted text"
expect_equal       response "exact expected value"

The second argument is either a double-quoted literal or a test-block const name (bare identifier, not quoted).

6. Run the tests

jaiph test                            # discover *.test.jh under the workspace

The runner discovers *.test.jh files recursively. A bare jaiph test walks every subdirectory under the workspace root, which includes installed libraries under .jaiph/libs/, so any test files that ship inside a library run too. Pass a directory (jaiph test ./e2e) or a single *.test.jh file to narrow the run, and see CLI — jaiph test for every invocation form and the exit-0-on-no-match behavior.

Grant use keys with --env

Unlike jaiph run, jaiph test runs no use pre-flight, so a script's declared use keys are absent from the spawned subprocess unless you grant them with --env. See Environment variables — Script subprocess environment for the --env KEY / --env KEY=VALUE grammar and the E_ENV_* codes.

Verification

A passing run prints one block per case, then ✓ N test(s) passed, and exits 0:

testing workflow_greeting.test.jh
  ▸ runs happy path and prints PASS
  ✓ 0s
✓ 1 test(s) passed

A failure prints the failing assertion and exits non-zero:

  ▸ handles error case
  ✗ expect_contain failed: "response" (42 chars) does not contain "expected" 1s

✗ 1 / 2 test(s) failed
  - handles error case

Related