Testing

Unit tests, live language-engine and embedding-model suites, and the corpus, artifact, agent, and cold-start checks all run on every pull request and push to main. The corpus checks run against pinned Next.js, Bun, and FastAPI repositories. Each release target other than x86-64 Linux builds every workspace target once and runs the unit suite from that build, so every test runs on the OS it targets. Windows runs the server and proxy regressions under the same ci profile. x86-64 Linux runs the unit suite from one instrumented archive, and the corpus checks there reuse one optimized archive rather than compiling their own. The gate job waits for every other ci.yml job and fails unless each one succeeded, and main accepts a pull request only once gate passes.

Local index regressions compare complete incremental read-service facts with a cold build after replacement, deletion, restoration, and case changes. The comparison keeps file digests, map values, index documents, documentation metadata, symbol pages, and relationship rows.

Live MCP cases validate partial maps and preparation warnings against the served schema. Worker and database cases force queue exhaustion, cancellation, publication ordering, and shutdown; resource cases verify published paging without recapturing every source file.

Background validation tests force elapsed deadlines under continuous events and recover edits without watcher reports. Supervisor tests hold one capture under a paused clock, supersede it, and read the order from the paths each later capture names: the owed rebuild runs before a due check, and a check one interval late runs first.

Git lock cases verify bounded waits, retained-lock expiration, linked worktree control paths, and publication equal to a cold build.

A failed rebuild followed by byte-identical restoration must recover without a watcher event, retain its read service, and equal the cold digests and derived facts.

Global resolution cases verify repeated bounded reads share their context and request bytes. Changed entry bounds, published dependency contexts, and requested packages refill the latest cache entry; replacement releases the preceding context and request.

CI compiles through sccache, which keeps compiled crates in the rift-oss-build-cache R2 bucket, so a job compiles only the crates whose inputs changed since a cached build. On Windows the native jobs link with /Brepro: without it, every proc-macro build carries a new timestamp, and each crate that uses the macro misses the cache. The release workflow starts the same build cache and restores the Cargo registry main saved, so a tag recompiles only what changed since an earlier tag. rift-dev build-cache starts it; a run without the bucket's secrets, such as a pull request from a fork, compiles everything.

Unit testsCopy Anchor Link

Install the pinned Rust toolchain, uv, just, cargo-audit, cargo-deny, cargo-llvm-cov, cargo-nextest, and oas3-gen 0.28.0. Run the unit suite from the repository root:

just test

The unit suite runs without external language servers, model downloads, or corpus checkouts. The test selection keeps those dependencies in the integration suite.

Some suites start a server that binds the loopback port range, which is machine-global. Nextest runs each test in its own process, so those suites share one election test group that admits a single test at a time.

just test runs one instrumented nextest invocation through rift-dev. The appliance sets OTEL_SDK_DISABLED=false and starts a local OTLP HTTP/protobuf collector before execution. Test processes and their children inherit the same logs, traces, and metrics endpoints; the collector stays open through process cleanup and exporter shutdown.

Nextest captures test output and emits JSON+ results. Python validates those results and the saved telemetry JSONL through Pydantic models. Discovery supplies an exact lookup from each JSON+ full name to its binary ID and test name; OTLP resources carry those identities with the run ID and opaque attempt ID. Retries remain zero, and tests keep nextest's concurrent scheduling.

A successful run prints its start time followed by:

{n} tests launched, all succeeded in under {x} minutes and {y} seconds.

A failed run groups output by test, with the error first, then timestamped logs and metrics. Its report keeps original multiline records, trace and span IDs, span duration and status, and metric units and temporality. Rust exports OTLP through the standard SDK Tokio batch processors and periodic reader, using async HTTP. Python receives the exported records. SDK providers stay open until test process exit; synchronous SDK shutdown runs on Tokio's blocking pool within the existing deadline. The upstream tracing-opentelemetry crate is used without a vendor patch. ScopedRecorderBuilder::clock names the clock span timings and measure_elapsed! read on the recorder's thread, so a test advances time instead of sleeping.

Each invocation retains artifacts under target/integration/nextest/nextest-<profile>-<uuid>: run.json separates execution results from collection errors, telemetry.jsonl keeps original OTLP records and collector request timing, and recording.zip exports that invocation's nextest recording. RIFT_TEST_REPORT_DIRECTORY selects another parent directory. The appliance bounds console evidence at 64 KiB, each captured stream at 16 MiB, and telemetry at 256 MiB; it continues draining after retention limits and reports omissions. Cache evictions have separate counters because original records remain on disk. JSON+ exposes no source-output truncation count, so the report names the recording as the source for nextest's captured output.

A failed binary test that opened a FailureWindow prints it to stderr. Its heading names the test with NEXTEST_BINARY_ID, NEXTEST_TEST_NAME, and NEXTEST_ATTEMPT_ID, then the window's bounds. A T: line states when the window ended, and a machine: line states the host's logical CPUs, system, and architecture; the harness reads no physical memory, so memory_bytes says so. When the window covers one workspace, each line a registered rift mcp child wrote to stderr merges into that workspace's records by timestamp, opening with rift mcp <pid> | . A stderr copy keeps at most 16 MiB and states the bytes it read and dropped past that bound. A server stderr file that holds rift: the log store refused a batch opens with a line saying the window falls back to that file for the records the store lacks.

just doctest runs Rust documentation examples through cargo test --doc, because nextest does not support doctests on stable Rust. The appliance retains their captured output and suite-scoped telemetry under target/integration/doctest. Stable batched rustdoc execution supplies no supported per-example OTLP identity; failed example names come from rustdoc output.

The remaining local checks cover formatting, generated files, crate boundaries, Clippy, Rust documentation, dependency policy, and developer tooling. The crate boundary check, rift-dev rust-architecture, also counts the crates whose shipped source declares an instrumentation SCOPE or opens a traced! operation, and fails when they outnumber SCOPES_MAX in crates/rift-tracing/src/metrics.rs:

just rust-gate

The repository's Git hook runs the quick gate before each commit. The gate checks formatting, dashes, generated files, locked workspace metadata, all-target and all-feature compilation, crate boundaries, and strict Clippy. Install the hook after syncing the development environment:

uv sync --locked --project dev --group dev
git config --local core.hooksPath .githooks

The local Git setting keeps this hook inside the repository. The hook runs pre-commit, which applies just quick-gate from .pre-commit-config.yaml. CI runs the same hook over all files with uv run --locked --project dev pre-commit run --all-files. just rust-gate adds Rust documentation, doctests, dependency audit, unit tests, release tests, installer tests, and developer tooling checks. CI keeps native, live, corpus, artifact, agent, and cold-start suites in their own jobs.

The hook sets CARGO_PROFILE_DEV_DEBUG=0 when that variable is unset and keeps any value already supplied.

CI retains compiled test archives and the CLI as artifacts. Test jobs reuse those artifacts; dependency caches avoid compiling unchanged dependencies on each run.

Integration testsCopy Anchor Link

Live suites require rust-analyzer and Bun, plus network access for model weights; the corpus suites also need the pinned corpus source. Install the language tool with rustup component add rust-analyzer; the TypeScript suite starts its language server through Bun. Model and package caches are reused between runs. Run the heavy suites from the repository root:

just integration-test

just live-test selects the live language-engine and model suites. They read the same build the unit suites do, so CI runs them on every pull request from the same archive just fast-archive writes: what they exercise is the engine integration, which any change can break.

CI also runs them on macOS, from an archive the Apple Silicon native job writes from the build its unit suite runs from. Windows runs none of them, for the reason issue #388 records.

The corpus suites need the optimized build and run from their own archive: just integration-archive writes an instrumented archive of the corpus suites alone, and just integration-cli the plain CLI the artifact checks serve. Each is cached on the exact Rust inputs that produce it, so a pull request that leaves Rust source and configuration untouched reuses the build instead of compiling it again.

For one corpus repository, fetch its pinned source and run its suite:

just corpus-sync
just corpus-test fastapi
just corpus-test bun
just corpus-test nextjs

Corpus suites use the optimized Cargo corpus profile with debug assertions, overflow checks, and line tables. Nextest runs one corpus test at a time. CI reuses one compiled archive across the corpus jobs.

Each corpus case stops its own actions thirty seconds before the deadline nextest ends it at, so removing the served checkout and writing the JSON report land inside that same deadline. A case that crosses the bound fails naming the action it was running, rather than being killed with no evidence of where it stood.

The cache lives under ~/.cache/rift-corpus. Each suite creates a disposable checkout at the commit in crates/rift/tests/corpus/pins.toml. Fetches retain 50 commits of history; a separate depth-one checkout tests the shallow-history response. Every run compares Git tree counts and sizes with the pin and requires reachable history objects before serving the tree. A repin must update the recorded measurements.

Corpus checks cover search, symbol lookup, syntax reads, history, and shutdown. They also create and remove source files outside Rift, compare later reads with those bytes, and verify that unrelated lexical rows in the index database at .rift/index remain unchanged. The report records the size of each database under .rift and its sidecar files before and after.

The corpus checks serve the same source and dependency configuration used by a workspace. Embedding model downloads stay disabled for corpus runs, so repository checks measure source and lexical indexing.

The Next.js churn case writes a probe declaration every two seconds while search, get_symbol, and nodes read it. It runs at least 50 calls across those tools and requires at least two writes during pending reads. Every call must return the same symbol identity, a complete written source revision, and its matching byte range. Source older than the revision present when the call started must carry stale_index.

Each budget the case applies stands strictly inside the one outside it, so a breach fails naming the action that ran long instead of tearing down whatever the next budget out was waiting on:

A tool's first read waits for the index the workspace is still building. Every read after it answers from a published index and never enters the readiness wait, so a steady read that does start waiting fails the case rather than hiding inside the first read's room. After writes stop, the check repeats only reads that returned older source with stale_index; older source without that warning fails. The report records source revisions, warnings, each call's duration and its own deadline, its overlapping writes, and each tool's first and maximum duration.

Artifact, agent, and conformance checks also retain original OTLP records and collection accounting in their persistent report directories through server cleanup.

Reports record the commit, seed, case, selected identities and languages, actions, elapsed time, and failure, including the complete Python traceback. RIFT_CORPUS_REPORT selects the JSON report location. Nextest displays the foreground server and MCP proxy diagnostics when each corpus test completes. Action records stay in the JSON report. Logs and reports stay outside the served checkout.

Each server a case starts keeps its standard error in a log file beside the report, cut at 8 MiB. When a case stops a server, the runner reads that server's persisted records with rift server logs --tail 5000 after the stop and before the served checkout is removed, and keeps the newest 1 MiB beside the report as <report>.server-<n>.records.log, led by a line that counts the bytes left out. The stops list in the report holds one entry per stop with the log path, the records path, and the size in bytes of index, metrics, vectors and their -wal files. An absent file reads "absent"; vectors exists only after the first vector operation. A records read that fails sets records to null and states the error in records_error; the case continues. One case keeps at most three records files, 3 MiB.

Each stops entry also carries the values the stop recorded, read from the records of that server alone. database_close lists the busy, log, and checkpointed values of each database.close record, one per database. stop_stages lists the stage, remaining, and outcome of each stop stage ended record. lacks names each of the two record kinds the records did not hold, and records_lines counts the records read; a count of 5000 means older records may be missing.

The report also carries a machine section: logical_cpus, cpu_model, memory_bytes, system, architecture, and the runner values runner_os, runner_arch, image_os, image_version read from RUNNER_OS, RUNNER_ARCH, ImageOS, and ImageVersion. A fact the host does not give is null. The runner prints the same facts on one machine: line when the case starts.

Each case starts an in-memory OTLP collector on 127.0.0.1 before its first server and stops it once the served tree is removed. Every server the case starts exports to it: OTEL_EXPORTER_OTLP_ENDPOINT names the collector, and OTEL_METRIC_EXPORT_INTERVAL and OTEL_BSP_SCHEDULE_DELAY set an export every 1000 ms. The report's collector entry counts the metric points and spans it received and what its bounds dropped.

A failed case prints, for each server it started, the server's standard error, the standard error of each rift mcp proxy, the persisted records, and the failure window. The failure window is the records that rift server logs --tail 5000 --since <start> --until <end> selects. The output carries the newest 256 KiB of it, and the newest 1 MiB is kept beside the report as <report>.server-<n>.window.log. Its start is the end of the last action the report recorded, or the start of the server when that is later; the failing action began at or after it. Its end is the moment the runner began collecting evidence.

The newest operations in flight record before the window's end follows the window. Then come what the collector received inside the same window, oldest first:

• the newest 200 metric points, each with its time, kind, name, the service.instance.id of the server that sent it, its instrumentation scope as otel.scope.name and otel.scope.version when the point carries one, its attributes, and its value
• the newest 100 spans, each at its end time with its name, attributes, and elapsed time; a span that carries request_id adds records=, the count of the window's log records whose req is the same, so the span and the records of its request read together

A point carries no request, so the window's time bounds are the one key that joins a point to the log records. A collector that received nothing prints no metric points received or no spans received, and a bound that dropped anything is stated.

Each note states a cut or a missing record instead of omitting it. The report names the files and the window bounds under evidence. A server stderr that holds rift: the log store refused a batch opens with the same fallback line the Rust window prints, and a machine: line with the host facts closes the notes. A pass prints none of this.

Artifact and agent checksCopy Anchor Link

Artifact checks run the native release binary from a temporary workspace. Agent checks use the Python MCP SDK and JSON Schema validator over stdio, exercise search, get_symbol, and nodes, and read the workspace, map, and log resources:

just artifact-test
just agent-test

Both commands build the native release binary unless --binary <path> supplies one. --target <triple> selects a Cargo target when building. A supplied binary is tested unchanged, so CI can reuse its build artifact.

The artifact and agent checks require reads to observe external filesystem writes. The agent check also reads incoming references through the embedded Python engine and verifies the source stays unchanged. The integration suite runs upstream MCP conformance checks against the same binary.

Both checks start the same in-memory OTLP collector and point their server at it. When an artifact or agent check fails, the runner attaches the same notes to the error: the server's standard error, the standard error of each proxy, the persisted records, and the failure window from the start of the server to the failure, with the collector's points and spans. Pass or fail, the runner prints one collector: line with the points and spans received and what the bounds dropped.

Cold first useCopy Anchor Link

The cold check needs Docker and a Linux binary matching the container architecture:

just coldstart-test --binary target/release/rift

The container has no network, language servers, or existing Rift cache. It starts with an empty workspace, then reads a newly created source file. After configuration selects a missing language server, an incoming-reference request must report the launch failure while syntax reads remain available. CI runs this check with its compiled Linux artifact; a local macOS run needs a Linux binary.

When the cold check fails, it prints the container's standard error, the persisted records, the proxy's standard error, and the failure window read through docker exec from just before the container was created. The container and its files are removed afterwards, so nothing is kept. The cold check starts no collector: the container runs with --network none, so the server inside cannot reach one.

The export reports what it loses on the server's stderr, as opentelemetry_sdk warnings and errors, whatever RUST_LOG names: the first span a full batch queue drops, the dropped total at shutdown, and each failed export.

Developer toolingCopy Anchor Link

The rift-dev Python package lives in dev/. uv installs its locked dependencies, Typer provides the CLI, and Pydantic validates configuration. Run a command through the package:

uv run --locked --project dev rift-dev --help
uv run --locked --project dev rift-dev corpus sync
uv run --locked --project dev rift-dev agent --binary target/release/rift

The justfile wraps the same commands for local work and CI: a recipe that is more than a few plain commands is a rift-dev command, and the recipe calls it.