This document defines the explicit Foundry behavior contract targeted by this repo.
The goal is not to emulate all of Foundry.
The goal is to emulate the specific Foundry runtime, dataset, and stream behavior required so that:
- developers can iterate locally,
- run the same Go binary locally and in Foundry,
- and have high confidence that local behavior matches deployed behavior for the surfaces this template depends on.
This document is the source of truth for:
- emulator scope,
- runtime assumptions,
- integration tests,
- cleanup/de-duplication work,
- future parity investigations,
- future migration decisions.
Each claim in this document should be treated as one of:
- Confirmed — supported by official Palantir docs, official Palantir SDK code, or direct captured evidence.
- Inferred — strongly suggested, but not yet fully proven on the exact target stack(s).
- Deferred — intentionally out of scope for parity v1.
If code and docs disagree, use this order:
- Official Palantir documentation
- Palantir-owned public SDKs / official examples
- Direct captured evidence (HAR / traces / responses)
- Repo-local contract
- Current implementation
This repo’s parity target includes:
- Pipeline-mode compute module runtime
- Dataset view / read semantics
- Dataset transaction + file upload + commit semantics
- Legacy compute-module stream-proxy semantics
- Stream-backed dataset-view semantics
- Minimal internal runtime keepalive semantics
This repo’s parity target does not include:
- full Foundry UI behavior
- full GraphQL gateway behavior
- full catalog / metadata service parity
- full row-level policy enforcement behavior
- full authz parity
- exact parity for every stream API family
- exact retention / cursor / compaction behavior
- complete infra provisioning parity
Official compute-module SDKs and docs confirm that pipeline-mode modules use mounted/runtime-provided values including:
BUILD2_TOKENRESOURCE_ALIAS_MAP
Additional service discovery/runtime variables supported by official SDKs include:
FOUNDRY_SERVICE_DISCOVERY_V2DEFAULT_CA_PATH
For this repo:
BUILD2_TOKENis a file path to a bearer tokenRESOURCE_ALIAS_MAPis a file path to JSON alias metadata- alias entries are shaped as:
rid- optional
branch
- if
branchis absent, default branch behavior is assumed
The local harness must provide file-backed equivalents of these vars exactly as Foundry does.
Official TypeScript and Python compute-module SDKs both model pipeline resources as:
- alias ->
{ rid, branch? }
This repo assumes:
- aliases are the stable application-facing identifiers
- actual resource RIDs may vary by environment
- runtime code should resolve resources by alias, not hardcoded RID
Alias parsing should remain centralized and schema-stable.
Official Python and TypeScript compute-module SDKs both read BUILD2_TOKEN from a mounted file.
This repo assumes:
- pipeline-mode Foundry API calls use the mounted build token
- the token is read from disk, not directly from env value contents
This should remain a file-backed invariant across runtime, tests, and emulator setup.
Official compute-module SDKs read FOUNDRY_SERVICE_DISCOVERY_V2 and expose at least:
api_gatewaystream_proxy
Official Python platform SDK routing also distinguishes:
- generic API endpoint
- auth endpoint
- high-scale endpoint
This repo’s runtime contract is:
api_gatewayresolves dataset/general Foundry API callsstream_proxyresolves stream/high-scale callsFOUNDRY_URLmay be supported locally as a development shorthand, but it is not the primary platform contract
The parity doc and code should make clear that service discovery is the primary Foundry-native contract.
Official TypeScript compute-module SDK uses:
GET_JOB_URIPOST_RESULT_URIPOST_SCHEMA_URIMODULE_AUTH_TOKENDEFAULT_CA_PATH
For this repo:
- these endpoints are treated as internal runtime infrastructure
- not as the primary business/data API surface
- keepalive/query-ack behavior is allowed to exist even when the repo is operating in pipeline-oriented mode
Some pipeline-mode deployments may still expect these endpoints to be exercised so the container is considered responsive.
Parity v1 only needs minimal support for:
- GET next job
- POST result
- auth via
Module-Auth-Token - optional CA-based HTTPS behavior if tested
This section is stronger than the earlier draft and should be treated as a first-class concept.
Official dataset SDK comments describe a dataset view as the effective file contents of a dataset for a branch and/or transaction context.
This repo adopts the following dataset-view model:
A dataset view may be resolved from:
- a branch
- an end transaction
- a transaction range
- a specific transaction (by setting start and end transaction equal)
Code should conceptually operate on dataset views, not just raw endpoint calls.
Branch-aware reads and file listings are officially supported.
A branch view should represent:
- the latest resolved dataset state for that branch
- usually defaulting to
masterif no branch is provided
The emulator must support branch-specific logical state.
Official dataset SDKs support:
endTransactionRidstartTransactionRid- transaction-equality for exact-transaction views
For this repo:
endTransactionRidmay resolve the latest effective view at that transactionstartTransactionRid + endTransactionRidmay resolve a range-constrained viewstart == endshould be interpreted as a specific transaction view
Parity v1 does not need a perfect full historical engine, but it must preserve enough transaction/view semantics to support the repo’s real behaviors.
Official dataset SDK comments explicitly state that:
- an intermediate snapshot transaction resets/removes prior files from the resolved file view
This repo’s parity model must explicitly acknowledge:
- dataset/file views are not simple append-only histories
- snapshot boundaries matter
- range semantics differ from naive cumulative append semantics
Parity v1 may simplify this internally, but the contract must not pretend dataset history is append-only.
Official dataset SDK error types include a DatasetViewNotFound concept.
A dataset view may be absent if:
- the dataset has no transactions
- the dataset has no files in the resolved view
- the branch is invalid
- the caller lacks access
The emulator should be able to represent “no dataset view available” distinctly from unrelated generic failures.
The repo relies on readTable.
Official SDKs confirm parameters including:
branchNamestartTransactionRidendTransactionRidformatcolumnsrowLimit
For this repo:
- reads are transaction/view-aware
- CSV is the primary format used
- branch defaults to
masterwhen omitted - transaction pinning should be used where deterministic reads matter
Official docs note that row ordering should not be treated as deterministic.
This repo must not rely on row order for correctness.
The emulator does not need to randomize rows, but tests and merge logic must assume ordering instability.
Official SDKs distinguish errors such as:
ReadTablePermissionDenied- schema/view not found cases
For this repo’s incremental logic:
404/view-missing => treat as no prior readable output where appropriate403=> treat as unreadable prior output where appropriate- other failures => surface unless explicitly classified otherwise
Official SDKs confirm:
POST /v2/datasets/{datasetRid}/transactions- optional
branchName - one open transaction per dataset branch
OpenTransactionAlreadyExists
This repo assumes:
- transaction creation is branch-scoped
- open-transaction conflicts are real platform behavior
- branch defaults to
masterfor most enrollments when omitted
Must support:
- create
- branch-scoped conflict
OpenTransactionAlreadyExistsstructured error
Official docs and SDKs support listing transaction history and document reverse chronological behavior.
This repo assumes:
- reverse chronological transaction history
- OPEN transactions are discoverable for fallback/reuse logic
- preview-gated/list-history behavior may exist depending on endpoint flavor
Must support enough list behavior to find the most recent OPEN transaction.
This repo’s current fallback strategy aligns with official transaction constraints:
- try create
- on
OpenTransactionAlreadyExists, list history - reuse latest OPEN transaction
- upload to that transaction
- avoid committing if ownership is external/Foundry-managed
- Confirmed that open transaction conflict exists
- Inferred that pipeline builds may pre-open and externally manage output transaction lifecycle in the relevant cases
Must support this flow explicitly.
Official SDKs document that commit:
- preserves file modifications
- updates the branch to point to the transaction
This repo assumes commit causes:
- logical branch-head advancement
- read-after-write visibility
- transaction status transition from OPEN to COMMITTED
Must model branch head updates.
This section is important because the platform SDKs give stronger semantics than the earlier draft captured.
Official SDKs document:
POST /v2/datasets/{datasetRid}/files/{filePath}/upload- upload to:
- a new branch-scoped transaction via
branchName - or a manually opened transaction via
transactionRid
- a new branch-scoped transaction via
- if
branchNameis used directly, transaction creation/commit may happen implicitly - default transaction type for branch-based upload is
UPDATE
This repo currently uses the manual transaction path.
The parity contract should therefore distinguish:
- upload into manually opened transaction via
transactionRid
- upload directly to branch with implicit transaction lifecycle
The contract and code should not conflate these two modes.
Official SDKs document file listing/content/metadata resolution by:
- branch
- transaction
- transaction range
- exact transaction
The parity contract should explicitly say:
- dataset file visibility is view-derived
- the repo’s dataset semantics are really dataset-view semantics
This makes the contract much more precise.
This section is the biggest place where the plan is now sharper.
From docs, HAR, and local platform SDK repos, there are at least two relevant stream surfaces:
Observed in compute-module docs and current repo:
GET /stream-proxy/api/streams/{rid}/branches/{branch}/recordsPOST /stream-proxy/api/streams/{rid}/branches/{branch}/jsonRecord
Observed in official platform SDKs:
POST /v2/highScale/streams/datasets/{datasetRid}/streams/{streamBranchName}/publishRecordPOST /v2/highScale/streams/datasets/{datasetRid}/streams/{streamBranchName}/publishRecordsGET /v2/highScale/streams/datasets/{datasetRid}/streams/{streamBranchName}/getRecordsGET /v2/highScale/streams/datasets/{datasetRid}/streams/{streamBranchName}/getEndOffsets
These high-scale calls are routed through the stream/high-scale endpoint.
This repo currently targets the legacy compute-module-compatible stream-proxy surface.
Parity v1 for this repo should remain defined in terms of the legacy surface:
- probe by reading
records - publish one row at a time via
jsonRecord - use stream records as best-effort incremental cache input
The parity doc should explicitly state that this is a deliberate implementation stance, not a claim that it is the only Foundry stream API.
Official platform SDKs show the newer streams model includes:
- schema-validated record publishing
- partitions
- sparse offsets
viewRid- end offsets
- subscriber APIs
- record size limits
This repo does not currently model the full newer high-scale streams contract.
Stream I/O is isolated behind foundryio.StreamBackend, so a future high-scale backend can be added without rewriting app orchestration.
Docs and current repo tests already reflect that records responses may vary in shape.
This repo should tolerate at least:
- top-level array of records
- object-wrapped list responses
- per-record wrappers such as:
recordvaluedata
The emulator may return one canonical shape, but tests should preserve parser tolerance.
This remains central.
Official stream docs plus HAR evidence indicate:
- stream data has low-latency streaming behavior
- and also dataset-style archived/view behavior
This repo’s parity contract is:
- stream-backed outputs are not only “push endpoints”
- they also eventually participate in dataset-view semantics
HAR evidence shows stream-backed datasets with:
- transaction history
- file listings
- archived files like
part=0_start=...avro - stream metadata referencing views and start transaction RIDs
The parity model should explicitly include:
- append-style transactional history for archived stream data
- dataset-view readability of archived stream data
- schema continuity between streaming and dataset-view representations
Official platform SDKs for newer stream APIs expose viewRid.
HAR evidence also shows stream latest-view metadata.
Parity v1 does not need to fully implement view RID control, but the contract should acknowledge that:
- streams have view identity
- the “latest stream on a branch” is a real concept
- stream reset can create a new view
Official SDK error types and observed behavior support structured errors with:
errorCodeerrorNameerrorInstanceId- typed parameters
The emulator should return structured errors for key paths, including:
OpenTransactionAlreadyExists- permission denied cases
- transaction not found / not open
- invalid dataset/file path cases
- stream permission/validation failures where modeled
Dataset side:
OpenTransactionAlreadyExistsTransactionNotFoundTransactionNotOpenReadTablePermissionDeniedDatasetViewNotFoundFileAlreadyExists
Stream side:
PublishRecordToStreamPermissionDeniedPublishRecordsToStreamPermissionDeniedRecordDoesNotMatchStreamSchemaRecordTooLarge
Error handling in the repo should become more explicit and centralized around these named classes of failure.
There are distinct auth/header modes for:
- pipeline build token / bearer-style calls
- internal runtime
Module-Auth-Token
Parity v1 must support:
- bearer auth for dataset/stream-facing calls
- module auth for internal runtime endpoints
Full Foundry authz is out of scope.
Parity v1 only needs to support meaningful distinctions between:
- success
- not found
- permission denied
for the endpoints this repo depends on.
Foundry dataset and legacy stream-proxy I/O use foundryio.DefaultRetryPolicy and foundryio.RetryTransient. Retryable failures include:
- network timeouts
- connection resets/refusals
4295xx
Permission and not-found responses are not classified as transient; callers handle them as contract-level outcomes where appropriate.
Worker/enricher retry remains separate from Foundry I/O retry because it handles per-row provider behavior rather than Foundry transport behavior.
The following are intentionally out of scope for parity v1:
- full public high-scale streams implementation
- subscriber APIs
- exact partition offset semantics
- exact
viewRidlifecycle semantics - exact stream retention/compaction behavior
- full row-level policy service behavior
- full UI service graph parity
- exact catalog/metadata service parity
- full Terraform/provider parity
The parity contract is now reflected in the code through these boundaries:
| Contract area | Current owner | Status |
|---|---|---|
| Email-enricher output columns and CSV codec | examples/email_enricher/pipeline/rows.go, csv.go |
Implemented |
| Legacy stream JSON row codec and stream table projection | examples/email_enricher/pipeline/stream.go |
Implemented |
| Foundry stream API family boundary | pkg/pipeline/io/foundry/stream_backend.go |
Implemented for legacy stream-proxy |
| Foundry retry policy for dataset and stream I/O | pkg/pipeline/io/foundry/retry.go |
Implemented |
| Incremental cache planning and row selection | internal/app/incremental.go |
Implemented, app-local |
| Mock stream readTable header projection | pkg/mockfoundry/server.go with caller-supplied header |
Implemented |
| Mock dataset branch heads, exact transaction reads, and open transaction branch reuse | pkg/mockfoundry/server.go, pkg/foundry/client.go |
Implemented for template scope |
The app runner should stay focused on orchestration. Stream codecs, retry policy, and incremental merge helpers should not be reintroduced into internal/app/enricher.go.
These are the intentionally unresolved areas after the current cleanup slice:
Official platform SDKs expose the newer high-scale streams API, but this repo still runs on the legacy compute-module-compatible stream-proxy surface. The StreamBackend boundary exists so a future HighScaleStreamsBackend can be added without rewriting RunFoundry.
Do not switch the default backend without evidence that the target compute-module deployment path should prefer the high-scale surface.
The mock server supports the dataset transaction behavior this template uses, including branch-scoped committed heads, exact committed-transaction reads, open transaction visibility rules, and branch-aware reuse of existing open transactions. It still does not implement a full historical Foundry dataset-view engine. In particular, transaction-range behavior, snapshot reset handling across arbitrary histories, and pagination remain simplified.
The local mock exposes stream records through a dataset-style CSV projection for local read-after-write and incremental-cache tests. It does not model real hot-buffer-to-cold-storage archive lag.
The mock can distinguish success, not found, and permission denied in tests, but it does not model Foundry row-level policy or full authorization behavior.
Use Tabex only if local docs, SDKs, HARs, and code are insufficient to answer one of these stack-specific questions:
- Which stream surface should this template prefer on the target compute-module stack: legacy stream-proxy or high-scale streams?
- Which
stream-proxy /recordsresponse envelopes still appear on the target stack? - Does
viewRidmatter for this template's stream output and incremental-cache behavior? - How much stream archival lag should the local emulator model for meaningful confidence?
- Compute modules getting started
- Compute modules containers/env
- dataset read/transaction/file docs
- streams concepts docs
typescript-compute-modulepython-compute-module
foundry-platform-typescriptfoundry-platform-python
.memory/data-tmp/23dimethyl.usw-3.palantirfoundry.com-logs-0.0.18-clean-stream.har.memory/data-tmp/part=0_start=8.avro