#Testing / Evals

Sero uses repository tests and promptfoo evals as separate quality signals. Not every suite runs for each pull request. Run real LLM evals only when their provider cost and credentials are intentional.

#Current root command surface

pnpm typecheck
pnpm build
pnpm test
pnpm test:ci
pnpm eval:snapshot
pnpm eval:search
pnpm eval:file-tools
pnpm eval:memory-save
pnpm eval:memory-search
pnpm eval
pnpm eval:view

pnpm test runs turbo run test. Use the root commands above as the supported entry points.

#CI workflows

The Test workflow runs for pushes to main, pull requests to main, and manual dispatch. It classifies changed paths. The affected jobs run pnpm typecheck, pnpm build, and pnpm test.

The root pnpm test:ci command also runs pnpm e2e:contract, but the Test workflow does not call that root command. The separate E2E Contract and E2E Agent workflows run each day and by manual dispatch. E2E Workflow is manual-only. No workflow runs pnpm eval or pnpm eval:snapshot.

#Evals command reference

#Snapshot evals

Snapshot evals use eval/promptfoo-snapshot.yaml and eval/snapshotProvider.ts. They assemble an approximation of the full Sero session prompt from real prompt-building functions and check:

• SDK/base prompt block presence
• CLI prompt block presence
• container/subagent prompt guidance where applicable
• prompt block ordering for cache stability
• full prompt size against baseline
• metadata completeness

Run snapshot evals before committing changes to prompt assembly, CLI instructions, container prompt blocks, subagent guidance, or session setup.

#Real LLM evals

Real evals use promptfooconfig.yaml and eval/seroProvider.ts. They run through promptfoo with actual model calls. The default config uses the Sero provider with a 120s timeout and an Anthropic grading provider for rubric assertions.

The search eval uses eval/promptfoo-search.yaml. Its Bash, FFF, Graphify, and combined arms receive the same seeded repository and tasks. It covers ranked lookup, exhaustive lookup, dependency tracing, architecture explanation, and a profile-wide lookup in a second workspace. The provider records tool names, arguments, result text estimates, tool latency, total token usage, and total latency.

The Graphify arms load the real plugin hooks and query engine. The fixture seeds known graph files and profile state, so this eval does not run the Python indexer. Graph commands use the bridged sero-cli model tool. The Bash and FFF controls must report that profile-wide search is unavailable rather than search outside the current workspace. Promptfoo runs cases serially so temporary profiles and native indexes do not overlap across arms.

The file-tool eval sets toolMode: runtime in eval/promptfoo-file-tools.yaml. The provider builds its session from Sero's host file-tool factory through eval/runtimeFileTools.ts, so recorded calls exercise the runtime edit and write tools rather than Pi's built-ins. eval/assertions/editBatching.ts reports replacements per call, same-file edit runs, calls by tool name, result tokens, latency, and failures. eval/seroProvider.test.ts covers extension-loader isolation in the default and runtime modes.

#Memory evals and metrics report

The memory checks measure memory; none of them gates a build.

• Save-recall (eval/promptfoo-memory-save.yaml, eval/memorySaveProvider.ts, eval/assertions/memorySave.ts). The provider plays each test's conversation in a real Pi session with the memory plugin and a memory-only sero-cli tool. Turns are separated by a line that holds only ---. Each test lists moments, and each moment has marker words. A saved entry that holds all the markers of a moment counts for that moment, one entry per moment. Other entries, including a second save of the same fact, count as noise. The assertion reason holds the report line: saved: … | missed: … | noise saves: …. The model comes from the provider model config. Use deepseek/deepseek-v4-flash with DEEPSEEK_API_KEY, or openai-codex/gpt-5.6-luna on the OpenAI subscription.

• Offline search (eval/promptfoo-memory-search.yaml, eval/memorySearchProvider.ts, eval/memory-search/). The provider writes the fixed entries in eval/memory-search/fixture.json to a temporary profile and scores each fixture query with the plugin's own search. It reports the hit rate and false-hit rate for a range of thresholds, in keyword mode and hybrid mode. The test passes when the default threshold keeps the hit rate at 0.8 or more and the false-hit rate at 0.1 or less. The defaults, 0.6 for hybrid and 0.5 for keyword search, come from this report.

• Live metrics report (scripts/memory-metrics-report.mjs). The memory plugin writes one metrics-YYYY-MM-DD.jsonl file per day to <SERO_HOME>/debug/memory/. The script summarises them:

  node scripts/memory-metrics-report.mjs [--dir <path>] [--from YYYY-MM-DD] [--to YYYY-MM-DD] [--json]

  A missing folder gives an empty report. Any other read error fails the script.

The memory plugin also has a test of the real search index under Electron's Node: run pnpm --filter @sero-ai/plugin-memory test:qmd. Plain pnpm test skips it.

Set SERO_EVAL_MODEL to use a specific model in all arms. Use a canonical provider/model value, for example:

SERO_EVAL_MODEL=anthropic/claude-haiku-4-5 pnpm eval:search

If you do not set it, the eval uses the default model in the Sero agent settings.

Auth/cost notes:

• pnpm eval can consume paid provider tokens.
• It expects provider credentials such as ANTHROPIC_API_KEY from the shell or eval environment handling.
• The eval provider can apply env credentials as runtime API-key overrides before falling back to ~/.sero-ui/agent/auth.json.
• Do not run live evals in CI or on PRs unless budget and credentials are explicitly intended.

#Scenario matrix

To add scenarios, create/edit a YAML file under eval/scenarios/ and add it to the relevant promptfoo config.

#Failure interpretation

#Relationship to other tests

#Related docs

• Running Evals
• Development Setup
• Troubleshooting