Evidence-backed memory pilot

Maintainer-only evaluation: This runbook is not part of Cortexa installation or normal product use. Cortexa has no Docker runtime dependency. Docker is used only to isolate untrusted generated code during the optional coding experiment; synthetic container checks run in the dedicated Linux CI job.

This runbook evaluates whether Cortexa helps technical teams reuse reviewed project knowledge. It separates three questions:

1. Can a new user capture a decision, retrieve it in another session or client, and inspect its source?
2. Do teams voluntarily keep using that workflow?
3. Does frozen memory improve independently scored coding-task correctness?

Implementation tests answer none of those questions. They show that the harness and its safety boundaries operate as specified. This document authorizes neither partner outreach nor publication; both require a separate decision.

Technical gate

Complete this gate before inviting a participant or running a held-out suite:

• The product, documentation, and offline benchmark CI jobs are green.
• Traceable context is explicitly enabled and the vault is unlocked.
• The project contains substantive, locally reviewed decisions with inspectable evidence. Generated reflections and build output are not treated as reviewed source decisions.
• The operator has an existing local Ollama model. Record its exact name and digest. Do not download a model or select a paid provider as part of the pilot.
• A Linux-container Docker engine is available. Build the tracked non-root image from bench/fixtures/code-runner/Dockerfile; record its immutable image ID and the resolved node:24-alpine repository digest embedded in its label.
• CORTEXA_API_TOKEN is set in the process environment. Never put the token in a suite, command argument, report, screenshot, support record, or publication.
• Raw reports and trial directories remain under gitignored bench/results/. They can contain private context and generated code.

Run the offline checks and smoke validation first:

npm ci
npm run test:bench
npm run bench:coding -- --suite bench/coding-tasks.example.json --model unused --dry-run

The smoke suite is synthetic. A green smoke run proves harness operation only.

Build and identify the runner

Skip this section unless you are a maintainer running the coding experiment locally. Use an already available Docker engine; do not install Docker merely to run Cortexa. These commands can pull the declared container base image, but they never download a language model:

docker pull node:24-alpine
BASE_IMAGE_DIGEST=$(docker image inspect --format '{{index .RepoDigests 0}}' node:24-alpine)
docker build --pull=false --build-arg BASE_IMAGE_DIGEST="$BASE_IMAGE_DIGEST" -t cortexa-coding-runner:local bench/fixtures/code-runner
docker image inspect --format '{{.Id}} {{.Config.User}} {{index .Config.Labels "org.cortexa.base-image-digest"}}' cortexa-coding-runner:local

Confirm that the user is non-root, the label equals BASE_IMAGE_DIGEST, and the image ID begins with sha256:. Windows live runs require Linux-container Docker; there is no unrestricted host-Node fallback.

Prepare the held-out suite

Maintain two suite families:

• Development suites are disposable and can be used while changing prompts, retrieval, adapters, or scoring.
• The held-out suite is frozen before the evaluated run. Never move a task from development into held-out after observing arm results.

The held-out suite must contain at least 30 independent tasks across at least three project domains. Use three repeats per arm. A task is independent when it tests one source decision and one independently scored behavior; prompt variants or several cases for the same rule belong in one task rather than being counted as separate evidence.

The suite must cover all of these classes:

• current project rules;
• exceptions to a general rule;
• superseded rules where the current replacement must win;
• irrelevant-memory distractors;
• conflicting evidence that requires the reviewed current decision;
• no-answer cases where the correct behavior is explicit uncertainty or null.

Before freezing, two human reviewers should inspect every task's source decision, project scope, prompt, starting files, allowed paths, hidden cases, and exact expected values. Resolve disagreements before the run. Record reviewer roles and review date outside the suite without adding personal data to shareable reports.

Freeze and record:

• the full suite SHA-256, including hidden cases;
• every input-file hash and allowed path;
• the repository commit and dirty state;
• exact model name and digest, decoding parameters, seed policy and budgets;
• embedding and strict-packet settings;
• runner image ID, base-image digest and runtime identity. The immutable runner image binds the trusted verifier code used for the run.

Do not edit the held-out suite, vault snapshot, model, runner, prompts, budgets, or retrieval configuration after examining a result. A necessary change creates a new preregistered experiment version; retain the invalidated run and reason.

Fixed coding experiment

The default evaluated protocol is fixed:

• at least 30 independent tasks;
• three paired repeats per task;
• one generation per arm and no repair loop;
• alternating arm order with paired seeds;
• 4,096 output tokens;
• 60-second generation timeout;
• 10-second verification timeout;
• no more than 200 total model calls in one invocation.

The harness fetches one complete strict project packet per task before either arm starts. The assisted arm reuses its exact frozen bytes for every repeat; control receives empty context. Generation never receives hidden cases or expected values. The trusted parent applies allowlisted files and independently scores container outputs with exact typed equality.

Run --dry-run with the final suite and operator-supplied model name first. Check the task, repeat, arm, and call counts. Then run locally with the recorded image and base digest:

npm run bench:coding -- \
  --suite bench/held-out-coding.local.json \
  --model YOUR_LOCAL_MODEL \
  --runner-image cortexa-coding-runner:local \
  --base-image-digest node:24-alpine@sha256:RESOLVED_DIGEST

Do not run Sources sync, Foundry learning, capture, reflection, prompt evolution, or background retrieval changes during the trials. Do not publish raw packets, answers, generated files, trial directories, or reports.

Analysis rules

A run is valid only when all paired attempts use the frozen inputs and identities, all strict packets are complete, and there are no model, API, runner, or verifier infrastructure errors. Incorrect code, invalid generated files, syntax errors, timeouts, and resource-limit failures remain valid negative outcomes.

Report the entire suite, including losses and failures. A task-performance claim requires all of the following preregistered conditions:

• paired success lift is positive;
• the task-level 95% bootstrap interval is entirely above zero;
• scope violations are zero;
• token coverage is reported honestly, with partial usage left unknown.

A scope violation means the control arm received nonempty retrieved context, a frozen packet's declared project scope differs from its task, or packet inspection finds memory from outside that task's project boundary. Inspect the frozen packet manifests before unblinding aggregate results and retain the audit record privately.

If the interval crosses zero, call the result inconclusive. Do not increase repeats, remove losses, change tasks, or alter the success rule after seeing the result. If both arms are at a correctness ceiling, preregister a separate efficiency experiment before evaluating time or tokens, and require correctness to remain preserved.

Blank report

Copy this block before a run. Keep every unmeasured value null; do not insert illustrative numbers that could be mistaken for observations.

{
  "state": "not_run",
  "execution_kind": "single_pass_code",
  "independent_tasks": null,
  "paired_success_lift_pp": null,
  "task_bootstrap_95_percent": null,
  "scope_violations": null,
  "token_coverage": null,
  "conclusion": "No performance conclusion before a valid held-out run."
}

Five-partner workflow

Recruit exactly five consenting design partners only after outreach is separately authorized. Observe each partner's current multi-agent workflow before proposing a replacement. Do not enable telemetry by default.

Session 1: first use

1. Record consent and the existing agents, clients, projects, and handoff pain in a private research record.
2. Ask the partner to choose one real project and several substantive decisions.
3. Help them capture or import those decisions and review the source evidence.
4. Connect one agent they already use; do not require replacing their stack.
5. Have them request a strict project packet from another session or client, inspect the source and review state, and identify any retrieval warning.
6. Record elapsed time from setup start until they complete the workflow without operator intervention.

Success target: at least four of five partners complete capture, cross-session or cross-client retrieval, and source inspection within 15 minutes using the guide.

Weeks 1–4: voluntary use

• Let partners choose when to use Cortexa. Do not schedule synthetic engagement.
• With consent, use a weekly interview or partner-maintained log to record which workflow they used, what they bypassed, and whether a decision was reused.
• Record setup minutes, support contacts, operator interventions, failed attempts, confusing states, scope mistakes, and time spent maintaining memory.
• Keep drop-off, negative feedback, workarounds, and weeks with no use visible.

Retention target: at least three of five voluntarily use the workflow each week for four weeks. Qualitative value target: at least three of five can identify one concrete reused decision or avoided repeated investigation. These accounts are qualitative evidence, not coding-task lift.

Closeout

Give each partner a review of the private notes and allow corrections or deletion. Summarize adoption, retention, support burden, scope failures, positive examples, negative outcomes, and abandoned workflows. Public reporting requires separate approval and uses sanitized aggregates only. Never publish names, tokens, private source URIs, raw memory, prompts, answers, files, or partner-specific identifiers.

If the workflow does not show benefit, narrow the product to the supported use cases. Do not respond to weak results by changing the held-out set or adding more autonomous agents.

Multi-tool agent evaluation remains separate

The implemented experiment measures single-pass coding correctness. It does not measure real coding-agent productivity, tool use, or autonomous delivery.

A later multi-tool adapter needs a separate design that freezes starting commits, CLI and tool permissions, model and kit versions, hidden verifier hashes, budgets, and environment images. It must disable all trial-time capture and learning, use fresh disposable isolation per arm, and record every tool call, retry, edit, token, duration, and verifier result. It receives no performance claim from this single-pass experiment.

Operator closeout checklist

• [ ] Raw reports and trials remain private under bench/results/.
• [ ] Every invalid, negative, and incomplete outcome remains in the record.
• [ ] The blank report stays not_run until a valid held-out run finishes.
• [ ] Partner consent, corrections, deletion requests, and support burden are recorded.
• [ ] Any public statement has separate approval and uses sanitized aggregates.
• [ ] The conclusion matches the preregistered decision rule.