Skip to content

Latest commit

 

History

History
566 lines (434 loc) · 21.4 KB

development.md

File metadata and controls

566 lines (434 loc) · 21.4 KB

Development Guide

This guide describes the current source-development workflow for TICO. Repository commands are routed through the ./ccex helper in the project root. It owns environment setup, test and runtime commands, diagnostics, formatting, and CI reproduction. The System Test Guide owns test strategy, harness behavior, and correctness criteria.

Contents

Prerequisites

  • Linux development environment
  • Python 3.10 or newer
  • git
  • A Python virtual environment is strongly recommended

Neither conversion nor the test suite requires ONE (one-compiler) or onert. Converted Circle models are executed by the built-in reference runtime (tico.circle.runtime). ONE's circle-interpreter and the onert package are optional compatibility runtimes that are used only when selected explicitly.

Create a development environment

git clone https://github.com/Samsung/TICO.git
cd TICO

python3 -m venv .venv
source .venv/bin/activate

# Install a supported Torch build and TICO in editable mode.
./ccex install

# Install formatter/static-check dependencies and test-only dependencies.
./ccex configure

The order matters. ./ccex configure test validates an already installed Torch package and installs the matching TorchVision build; it does not install TICO or Torch. Run ./ccex install first. Test configuration also installs test/requirements.txt and validates the package environment. test/requirements_pre.txt is reserved for pre-release test dependencies and is currently empty. Optional external runtimes are installed separately; see Runtime selection.

Set up only one part of the environment when needed:

./ccex configure format
./ccex configure test

Torch selection

The source installer accepts a stable family, an exact version, a qualification candidate, a repository-pinned nightly, or the latest published nightly pair:

./ccex install --torch_ver 2.13
./ccex install --torch_ver 2.7    # legacy best-effort
./ccex install --torch_ver 2.11
./ccex install --torch_ver 2.13.0+cu132
./ccex install --torch_ver 2.14
./ccex install --torch_ver nightly          # repository-pinned Torch/TorchVision
./ccex install --torch_ver nightly-latest   # latest published nightly pair

TICO keeps Torch 2.5 through 2.10 as legacy best-effort source-install choices, qualifies 2.11, 2.12, and 2.13, and uses 2.13 as the default. Torch 2.14 is installable as a qualification candidate but is not part of the release-support window. Family requests resolve to a project-pinned patch rather than allowing pip to select an arbitrary patch release. The package metadata itself keeps a bare torch dependency, so a normal pip install does not reject a user-managed older version. nightly uses the versions pinned under infra/dependency/; nightly-latest resolves Torch and TorchVision together from one moving nightly index.

Compute-platform options:

# Force a CPU wheel.
./ccex install --cpu_only

# Override detected host CUDA capability when selecting a compatible wheel.
./ccex install --cuda_ver 12.8

--cpu_only and --cuda_ver are mutually exclusive.

When no Torch version is explicitly requested and a configured stable Torch package is already installed, ./ccex install preserves that installation if its compute platform is compatible. This includes legacy best-effort families. ./ccex configure test then installs the matching TorchVision package and verifies the final package pair with pip check. Nightly selectors are deliberately re-resolved: nightly restores the repository pin, while nightly-latest upgrades the Torch/TorchVision pair together before test configuration validates it.

The source of truth for families, exact patches, CUDA wheel variants, and CI matrices is tico/utils/compat/torch_version_policy.py. See the PyTorch Version Policy for qualification, promotion, and release-branch rules.

Build and install a wheel

Build the distribution artifacts:

./ccex build

Install from dist/ instead of editable source:

./ccex install --dist

A clean wheel workflow similar to CI is:

./ccex build
./ccex install --dist --torch_ver 2.13
./ccex configure test --torch_ver 2.13
pt2-to-circle -h

Run tests

Default suite

./ccex test

With no filter, ccex runs unittest discovery under test/. Large model tests are selected separately with -m and are not part of the normal model-independent suite. ./ccex test supports repository paths containing spaces.

Keyword filtering

./ccex test -k add
./ccex test -k ConvertMatmulToLinear
./ccex test -k quantization
./ccex test -k passes
./ccex test -k test_quantizer_registry

Quote keyword patterns containing spaces to pass them to unittest as one argument.

The shorthand keywords op and net are expanded to the corresponding generated module-test namespaces:

./ccex test -k op
./ccex test -k net

Use the narrowest relevant test first, then expand to the owning subsystem and finally to the complete suite for cross-cutting changes.

Other test-runner options

./ccex test --all                 # Explicit full suite
./ccex test -i                    # Include internal-only tests
./ccex test -v                    # Set TICO_LOG=4 for this run
./ccex test -r reference          # Built-in reference runtime (default)
./ccex test -r onert              # Optional: execute with the onert package
./ccex test -p                    # Run performance benchmarks

--all and --keyword cannot be combined. --model also cannot be combined with --all or --keyword. -i sets RUN_INTERNAL_TESTS=1 for discovery. -p / --perf selects the performance entry point instead of normal discovery. For example, ./ccex test -v -k add combines verbose diagnostics with a focused test; setting TICO_LOG=4 explicitly has the same effect (see Log levels).

Test duration reporting

./ccex test --durations 10         # Collection, suite time, and slowest 10 full test IDs
./ccex test -k SimpleAdd --durations 0  # Report every executed test duration

--durations N is opt-in, accepts a nonnegative count (0 means all), and works with the default suite, -k, and -m. It does not apply to --perf. It uses the standard unittest duration support on Python 3.12 and newer, with a small result adapter on Python 3.10/3.11; selection, outcomes, and exit behavior stay with unittest.

Collection time starts in the timing worker before importing unittest and ends after discovery (including discovered imports and generated test construction). It excludes interpreter startup and the parent ccex script. Suite time surrounds unittest's runner, including class/module fixtures and result output. Individual durations include test setup, body, teardown, and cleanups; they exclude class/module fixtures and discovery. Thus suite time minus individual durations is not a discovery measurement. For complete command wall time, measure a fresh process externally, for example with time ./ccex test -k SimpleAdd --durations 10; compare identical environment, selection, and cache conditions across repeated fresh processes.

Opt-in test requirements

Small local CPU wrapper tests run in the default suite; their Transformers version or model-class availability guards still apply. --include-internal is reserved for external dependencies and costly CPU integration checks. It does not install their prerequisites. Prefer selecting one test rather than enabling every internal test:

# Downloads Maykeye/TinyLLama-v0 and Salesforce/wikitext (wikitext-2-raw-v1).
# Requires the configured Transformers/datasets packages and Hugging Face access
# or an explicitly prepared local cache.
./ccex test --include-internal -k SmoothQuantTest.test_value

# Local CPU integration: large calibration tensors and Hessians; allow extra
# CPU time and RAM. This does not need a checkpoint or dataset download.
./ccex test --include-internal -k GPTQTest.test_net_on_zero_inputs

# Requires triv24-compile and triv24-ssinfer from the TRIV24 toolchain under
# /usr/share/one/backends/triv24/bin (separate from the reference runtime).
./ccex test --include-internal -k EvaluateTest.test_evaluate_simple_linear

The GPTQ test_model and introspection TestSmoothQuantPTQDiff checks use the same remote model and dataset as the SmoothQuant example above. Other retained GPTQ/FPIGPTQ CPU integration skips describe their calibration/Hessian costs in the test reason. Model-directory tests keep their separate model selection path. Local Gemma4 assistant integration requires GEMMA4_ASSISTANT_PATH (and GEMMA4_TARGET_PATH for verification); run python3 -m unittest -v test.quantization.recipes.integration.test_gemma4_assistant_integration with those checkpoint directories set. CUDA observer checks require a CUDA-enabled Torch build and a working GPU. The large Circle tests document their independent memory/disk and ONE opt-ins. External runtimes keep the explicit runtime selection requirements.

Performance benchmarks

./ccex test -p runs the configured conversion-time and serialized-size threshold benchmark:

./ccex test -p

The full Circle O1 scheduler comparison needs a caller-provided .circle artifact and is invoked directly:

python3 -m test.performance.benchmark_circle_optimizer \
  model.circle \
  --repeat 3

It compares the legacy RESTART scheduler with O1's round-based UNTIL_NO_CHANGE scheduler, verifies both outputs, and requires byte-identical Circle binaries. It is an opt-in diagnostic benchmark rather than a repository threshold. See test/performance/README.md for all options.

Test organization

Use focused unit tests for local behavior and module tests for PyTorch-to-Circle parity. Model tests and performance benchmarks are opt-in workflows. The test documentation index links the subsystem guides.

The System Test Guide owns the test layers, directory layout, and end-to-end harness flow. Consult its correctness criteria when deciding what a new regression must assert and its change-specific checklist when choosing coverage.

Runtime selection

The default runtime for module parity tests, CircleModel.__call__(), and tico.interpreter.infer() is the built-in reference runtime. It executes the serialized Circle bytes with NumPy/CPU PyTorch kernels, supports dynamic shapes, and needs no external package:

./ccex test -k add
./ccex test -r reference -k add

Two optional compatibility runtimes can be selected from the command line or the environment. They are never chosen implicitly, even when installed:

# ONE luci-interpreter through /usr/share/one/lib/libcircle_interpreter_cffi.so
pip install cffi              # plus a ONE release providing the shared library
./ccex test -r circle-interpreter -k add

# onert Python package (pre-release wheels)
pip install --pre onert==0.2.0.dev250922
./ccex test -r onert -k add
CCEX_RUNTIME=onert ./ccex test -k add

circle-interpreter cannot execute dynamic-shape models; the harness runs a test tagged use_onert with onert when that legacy runtime is selected. A negative test whose expected error is runtime-specific (@test_negative(..., runtime="onert")) runs as a normal parity test under the other runtimes.

See Circle artifact tools for the supported operator set, quantized execution modes, and debugging options.

Model tests

Model tests are selected by directory name or shell-style pattern:

pip install -r test/modules/model/<model_name>/requirements.txt
./ccex test -m <model_name>

# Quote wildcard patterns so the shell does not expand them.
./ccex test -m "Llama*"

The -m option sets CCEX_TEST_MODEL and runs the model-test loader. Model tests may have additional dependencies and can be significantly more expensive than synthetic unit or module tests. Do not use a full model test when a small deterministic graph can cover the behavior.

Debug conversion

Log levels

Set TICO_LOG before the Python process imports TICO:

TICO_LOG=4 ./ccex test -k add
TICO_LOG=4 python examples/my_conversion.py
Value Level
1 fatal
2 warning
3 info
4 debug

Debug mode includes instrumented graph and constant-size diffs around passes and conversion phases.

Intermediate graph images

TICO_GRAPH_DUMP=1 ./ccex test -k add

The main conversion pipeline writes these stage snapshots when applicable:

.tico_tmp/session_<timestamp>/1_after_decompose.png
.tico_tmp/session_<timestamp>/2_after_legalize.png
.tico_tmp/session_<timestamp>/3_after_quantfold.png

Graph rendering uses pydot and Graphviz. The output directory is session-scoped; do not commit generated images.

Circle artifact inspection

tico-circle inspect model.circle --tensors --operators
tico-circle verify model.circle
tico-circle extract model.circle --ops 20-64 -o region.circle
tico-circle optimize model.circle --preset o1 -o model.o1.circle

Use tico-circle verify for static Circle consistency. Use an end-to-end runtime test for numerical parity and backend execution. CircleReferenceRuntime(...).run(inputs, trace=True) retains every materialized tensor value for debugging. Virtual full-table dequantization values used by selected-row execution are omitted from the trace.

Formatting and static checks

Before opening a pull request, apply formatting and run all static checks with one command:

./ccex format

This runs UFMT (Black formatting and import sorting), Pylint, and mypy together. To check without applying formatter patches:

./ccex format --no-apply-patches

Use Python 3.12 and the development environment described above. Pylint and mypy need the project and test dependencies: missing PyTorch or different installed library versions can change inferred types without reporting an import error.

To reproduce CI in a fresh Python 3.12 virtual environment, install the policy-selected CPU Torch build and configure the same dependencies:

TORCH_FAMILY=$(python3 tico/utils/compat/torch_version_policy.py default-family)
./ccex install --torch_ver "$TORCH_FAMILY" --cpu_only
./ccex configure test --torch_ver "$TORCH_FAMILY" --cpu_only
./ccex configure format
python3 -m pip check
python3 -m pip list --format=freeze
./ccex format --no-apply-patches

Use a separate environment when developing against another Torch version or optional model dependencies. The static-analysis CI job records installed package versions to help compare environments.

To check tracked files changed from the local main branch, use --diff-only. Add new files to Git before checking them:

./ccex format --diff-only --no-apply-patches

CI runs formatting and static analysis as separate jobs so formatting can finish without installing PyTorch. Both jobs use the same lintrunner configuration as ./ccex format; developers do not need to select individual tools.

The formatter's transitive dependencies are pinned in requirements-lintrunner.txt, including stdlibs and libcst. Keep those pins when updating formatter packages: their versions can change import classification and formatting even when Black, UFMT, and usort are unchanged.

Coverage

Run the discovery suite under coverage and print a terminal report:

./ccex coverage

Write a report under test/reports/cov/:

./ccex coverage -f txt
./ccex coverage -f xml

If coverage is not installed, the current helper requests version 7.6.1. Failed tests return a nonzero exit status while still generating the requested report; report failures also return nonzero when the tests pass. Coverage discovery and report generation support checkout paths containing spaces and treat shell pattern characters in those paths literally.

Pull-request CI

The PR workflow targets main and rel/*. PyTorch versions are resolved from the central policy module rather than duplicated in workflow YAML.

  1. Commit-message check

    • Runs when the pull request is ready for review.

    • Requires at least one commit-body line in the form:

      TICO-DCO-1.0-Signed-off-by: <NAME> <<EMAIL>>
      
  2. Format check (check-format)

    • Runs on Ubuntu 24.04 with Python 3.12.
    • Executes ./ccex configure format, then runs lintrunner with --take UFMT without applying patches.
    • Does not depend on the package build or install PyTorch.
  3. Package build

    • Builds the TICO wheel once.
    • Uploads one short-lived artifact reused by static analysis and versioned tests.
  4. Static analysis (check-static)

    • Runs on Ubuntu 24.04 with Python 3.12.
    • Reuses the test-environment setup action with the default Torch family from the central policy, including CPU Torch, matching TorchVision, TICO, and test dependencies.
    • Configures the analysis tools, checks package dependency consistency, records installed versions, runs the Transformers/Pylint infrastructure regression tests (python3 -m unittest -v infra.style.tests.test_transformers_config_pylint), and runs lintrunner with --take PYLINT,MYPY without applying patches.
    • The existing Check style status (check-style) succeeds only when both formatting and static analysis succeed, preserving existing required-check settings.
  5. Versioned tests

    • Runs the complete suite on the default qualified family, currently 2.13.
    • Runs blocking export and quantization smoke tests on the oldest supported family, currently 2.11.
    • Runs the same smoke tests non-blockingly on the qualification candidate, currently 2.14.

The smoke path covers a small torch.export/PT2/Circle conversion and a quantized CNN Circle export. These focused tests detect version-contract breakage without running the complete suite on every family for each pull request.

A separate compatibility workflow runs nightly-latest smoke tests on six days each week and the complete supported/candidate/nightly-latest matrix on the remaining day. Manual dispatch also runs the complete matrix. It provides early warning without participating in PR branch protection. Official package publication builds the wheel once and runs the full suite on every qualified stable family before publishing; candidate and nightly selectors are outside that release-support gate. See the PyTorch Version Policy for the exact tiers.

The conversion/size threshold benchmark is available through ./ccex test -p. The full Circle O1 scheduler benchmark is invoked directly with a caller-provided artifact. Neither benchmark is part of the current PR matrix.

Change-specific guidance

Add or change a PyTorch-IR pass

  • Implement PassBase.call() and return PassResult(modified=...) accurately.
  • Preserve ExportedProgram graph-signature mappings while replacing nodes.
  • Update metadata for newly created or rewritten tensors.
  • Add focused coverage under test/unit_test/passes/.
  • Include both matching and structurally similar non-matching cases.
  • Add an end-to-end module test when serialization or runtime behavior changes.

Add a serialized operator

  • Add or update a NodeVisitor under tico/serialize/operators/.
  • Register every supported ATen overload with register_node_visitor.
  • Add serializer/operator unit coverage and a module conversion/parity test.
  • Do not leave an unsupported operator in the final graph and rely on a runtime to reject it; TICO validates supported targets before serialization.

Change quantization

  • Preserve the prepare -> calibration/statistics -> convert lifecycle.
  • Put generic algorithms and infrastructure in their owning quantization packages.
  • Put model-family behavior in recipe adapters rather than generic stages.
  • Prefer deterministic synthetic tests before full model evaluation.
  • Update the relevant quantization README or configuration reference when behavior is user visible.

Change Circle artifact tools

  • Keep PyTorch-IR passes in tico/passes/ and serialized Circle-to-Circle passes in tico/circle/passes/.
  • Verify index remapping, graph boundaries, buffers, signatures, multi-subgraph behavior, and cleanup contracts as applicable.
  • Follow the dedicated pass guidance in Circle artifact tools.

See also