LedgeLM Docs
Dashboard

LedgeLM

LedgeLM tells you whether an application change made your AI evaluations worse. It receives results from the evals your team already runs, preserves the context behind those results, and compares pull requests with a rolling mainline baseline.

LedgeLM is an aggregator, not an eval framework. Keep using notebooks, promptfoo, pandas, pytest, Vitest, custom judges, or whatever already works.

What LedgeLM does

One CI run sends many results to LedgeLM. Each result belongs to a commit, branch, and optionally a pull request. LedgeLM then:

  1. stores the raw result and its provenance,
  2. groups results by eval name,
  3. compares the run with recent completed runs on the baseline branch,
  4. publishes one GitHub Check, and
  5. keeps the raw data available for debugging and trend analysis.

The mission is deliberately narrow: make “did this change regress our AI behavior?” quick to answer and easy to investigate.

Eval philosophy

Start lazy

There is no registration step. Report any stable eval name and it appears as an informational eval. Once the signal proves useful, promote it to blocking and add thresholds in .ledgelm.yml.

Compare with a distribution

A single judge call can be noisy. Pull requests are compared with rolling mean, standard deviation, and pass-rate statistics from recent baseline-branch runs, not only with the previous commit.

Preserve provenance

Keep the eval name stable while its meaning stays stable. Record changing dimensions separately:

  • target: model or system being evaluated
  • judge: model scoring the target
  • prompt: prompt name and version
  • dataset: dataset name, version, and split
  • test_case_id: durable test-case identity
  • target_id: stable comparison lane for a model or system variant
  • tags: queryable dimensions such as locale or product area
  • metadata: rationale, traces, and other debugging context

This is what lets future investigations separate a code regression from a model, prompt, judge, or dataset change.

Never make observability an outage

If LedgeLM is unavailable, reporters warn and return an unknown delivery verdict. They do not throw and should not fail the eval job.

The dashboard

The dashboard stays close to the regression workflow:

  • Projects lists repositories connected through the NexiHealth GitHub App.
  • PR comparison shows current scores and pass rates beside rolling baselines.
  • Eval trend shows mainline drift over time, including model provenance.
  • Run detail preserves every test case, rationale, cost, and latency.
  • Commit history helps identify when a regression was introduced.
  • Project settings controls baselines, configuration, archiving, and project API tokens.
  • Manual runs groups local experiment attempts without changing CI baselines.

Archived evals are hidden from default comparisons and baseline calculations, but their historical data remains available.

GitHub integration

The GitHub App reads repository configuration at the reported commit, writes one Check Run for each PR eval run, and upserts one summary comment on the PR. It requires Checks and Pull requests write permissions. It does not authenticate CI uploads.

CI uses a LedgeLM-issued token scoped to one project. From Project → Settings, select Configure GitHub Actions to create:

  • repository secret LEDGELM_API_TOKEN
  • repository variable LEDGELM_API_URL

The token is encrypted for GitHub on the server and is never returned to the browser. Reconfiguring rotates the managed token.

For local runs, issue a named token from Project → Settings → Project API tokens and copy it when shown. The plaintext is displayed once. Export it as LEDGELM_API_TOKEN alongside LEDGELM_API_URL; do not commit it or use GitHub OAuth/App credentials for uploads.

TypeScript

Install

bun add @ledgelm/reporter
# or: npm install @ledgelm/reporter

Report results

import { flush, report } from '@ledgelm/reporter';

report({
	name: 'response_grounding',
	type: 'judge',
	score: 0.82,
	passed: true,
	cost: 0.004,
	latency_ms: 940,
	usage: {
		target: { input_tokens: 812, output_tokens: 146, cached_input_tokens: 500 },
		judge: { input_tokens: 1042, output_tokens: 87, reasoning_tokens: 32 }
	},
	input: { messages: [{ role: 'user', content: 'Can I return this purchase?' }] },
	output: 'Yes. Refunds are available within 30 days with a receipt.',
	expected_output: 'Explain the 30-day refund policy and receipt requirement.',
	provenance: {
		test_case_id: 'refund-42',
		target_id: 'support-claude-sonnet',
		target: {
			provider: 'anthropic',
			model: 'claude-sonnet',
			version: '2026-07'
		},
		judge: { provider: 'openai', model: 'gpt-5' },
		prompt: { name: 'support-answer', version: 'v8' },
		dataset: { name: 'support-golden', version: '2026-07', split: 'ci' },
		tags: { locale: 'en', product: 'support' }
	},
	metadata: {
		rationale: 'The response cites the supplied refund policy.'
	}
});

// Call once after all report() calls.
const result = await flush();
console.log(`LedgeLM delivery: ${result.verdict}`);

report() validates and buffers locally. flush() discovers GitHub Actions context, uploads the buffer in chunks, and finalizes the run.

Python

Install

pip install ledgelm
# or: uv add ledgelm

Report results

from ledgelm import flush, report

report(
    name="response_grounding",
    type="judge",
    score=0.82,
    passed=True,
    cost=0.004,
    latency_ms=940,
	usage={
		"target": {"input_tokens": 812, "output_tokens": 146, "cached_input_tokens": 500},
		"judge": {"input_tokens": 1042, "output_tokens": 87, "reasoning_tokens": 32},
	},
	input={"messages": [{"role": "user", "content": "Can I return this purchase?"}]},
	output="Yes. Refunds are available within 30 days with a receipt.",
	expected_output="Explain the 30-day refund policy and receipt requirement.",
    provenance={
        "test_case_id": "refund-42",
        "target_id": "support-claude-sonnet",
        "target": {
            "provider": "anthropic",
            "model": "claude-sonnet",
            "version": "2026-07",
        },
        "judge": {"provider": "openai", "model": "gpt-5"},
        "prompt": {"name": "support-answer", "version": "v8"},
        "dataset": {
            "name": "support-golden",
            "version": "2026-07",
            "split": "ci",
        },
        "tags": {"locale": "en", "product": "support"},
    },
    metadata={"rationale": "The response cites the supplied refund policy."},
)

result = flush()
print(f"LedgeLM delivery: {result.verdict}")

Async applications can use await async_flush(). Both variants retain buffered results and return unknown when delivery fails.

BYOK evaluation adapters

Evaluation adapters invoke models and framework objects already configured by your code. LedgeLM does not provide models, hold provider keys, manage prompts, or choose how an answer passes.

TypeScript frameworks

bun add @ledgelm/evals @ledgelm/langchain @ledgelm/langgraph @ledgelm/ai-sdk
import { evaluateRunnable } from '@ledgelm/langchain';
import { evaluateGraph, evaluateNode } from '@ledgelm/langgraph';
import { evaluateGenerateText } from '@ledgelm/ai-sdk';

await evaluateRunnable({ name: 'model-answer', runnable: model, input, assess });
await evaluateGraph({ name: 'agent-graph', graph, input: testState, assess });
await evaluateNode({ name: 'router-node', node: router, input: customState, assess });
await evaluateGenerateText({ name: 'sdk-answer', generation: { model, prompt }, assess });

@ledgelm/evals exposes the generic evaluate() primitive used by every adapter. The assessment callback can be deterministic or call any BYOK judge model. Framework telemetry is captured when available, and explicit reporter provenance always wins over inferred values.

Python frameworks

uv add 'ledgelm[langchain,langgraph]'
from ledgelm.integrations.langchain import evaluate_runnable
from ledgelm.integrations.langgraph import evaluate_graph, evaluate_node

evaluate_runnable(name="model-answer", runnable=model, input=case, assess=assess)
evaluate_graph(name="agent-graph", graph=graph, input=test_state, assess=assess)
evaluate_node(name="router-node", node=router, input=custom_state, assess=assess)

Use the async variants with asynchronous runnables. Every helper buffers one result and leaves the single final flush() or async_flush() call to the suite. Execution and assessment exceptions become failed results by default so later dataset cases still run; set on_error="throw" when a test runner should stop.

GitHub Actions

First use Configure GitHub Actions in the project settings. Then expose the repository secret and variable to the eval process.

Choose a CI topology

  • Single job: preferred when the full suite fits on one runner. Report each result and flush once at the end.
  • Matrix with artifact fan-in: supported and recommended for parallel suites. Matrix jobs upload typed JSON artifacts; the LedgeLM fan-in action validates everything and flushes once.
  • Multiple independent workflows: use only for intentionally separate eval runs. They cannot currently contribute safely to one logical result set.
  • Direct matrix shard uploads: planned, but not yet supported. The future protocol will group shards by project, branch, commit, optional PR, and eval group, then publish only after an explicit close operation.

TypeScript workflow

name: AI evals

on:
  pull_request:
  push:
    branches: [main]

jobs:
  evals:
    runs-on: ubuntu-latest
    env:
      LEDGELM_API_URL: ${{ vars.LEDGELM_API_URL }}
      LEDGELM_API_TOKEN: ${{ secrets.LEDGELM_API_TOKEN }}
    steps:
      - uses: actions/checkout@v4
      - uses: oven-sh/setup-bun@v2
      - run: bun install --frozen-lockfile
      - run: bun run evals

Python workflow

name: AI evals

on:
  pull_request:
  push:
    branches: [main]

jobs:
  evals:
    runs-on: ubuntu-latest
    env:
      LEDGELM_API_URL: ${{ vars.LEDGELM_API_URL }}
      LEDGELM_API_TOKEN: ${{ secrets.LEDGELM_API_TOKEN }}
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v6
      - run: uv sync --frozen
      - run: uv run python evals.py

The reporter derives the commit, branch, PR number, workflow, run ID, and attempt from the GitHub Actions environment. Explicit context overrides remain available for nonstandard workflows and tests.

Matrix workflows

Do not call flush() independently from parallel matrix jobs: every job in a workflow shares the same GitHub run ID, so independent uploads would race for the same LedgeLM run. Have each shard upload a typed, versioned result artifact, then use the LedgeLM fan-in action to validate and flush the complete matrix.

jobs:
  eval-shard:
    strategy:
      fail-fast: false
      matrix:
        suite: [grounding, safety, quality]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: bun run eval:${{ matrix.suite }} --output ledgelm-results.json
        env:
          LEDGELM_SHARD: ${{ matrix.suite }}
      - uses: actions/upload-artifact@v4
        with:
          name: ledgelm-results-${{ matrix.suite }}
          path: ledgelm-results.json

  report-evals:
    needs: eval-shard
    runs-on: ubuntu-latest
    steps:
      - uses: NexiHealth/ledgelm/actions/flush@main
        with:
          api-url: ${{ vars.LEDGELM_API_URL }}
          api-token: ${{ secrets.LEDGELM_API_TOKEN }}

Each file uses { schema_version: 1, shard, results }. TypeScript exports reportArtifactSchema and ReportArtifact; Python exports the generated ReportArtifact Pydantic model. The standalone JSON Schema is available at /schemas/latest/report-artifact.schema.json.

The action currently tracks main. Keep /actions/flush in the uses: path; NexiHealth/ledgelm@main is not valid because the action manifest is not at the repository root.

The action downloads ledgelm-results-*, validates every artifact, rejects duplicate shard keys, calls report() for each result, and calls flush() once. It outputs the verdict, run/dashboard IDs, artifact/result counts, and JSON arrays of failing and warning evals. Finalization publishes one Check and updates one PR summary comment for the complete commit-level result set.

If matrix shards represent different models or agent variants, give every result a stable provenance.target_id. The artifact shard remains a delivery identifier; summaries, trends, and rolling baselines are separated by target_id, not by GitHub job names.

Keep semantic eval failures in the artifact as passed: false, so matrix jobs still upload their result. Do not force the fan-in job to run after an infrastructure failure: publishing an incomplete matrix would create a false comparison.

The future distributed protocol will remove the artifact fan-in requirement: each matrix job will upload a stable shard directly, and a lightweight final job will close the branch-and-commit run. Until that protocol is implemented, independent matrix flush() calls are unsafe and must not be used.

Manual runs

Manual runs are for local model bakeoffs, prompt iterations, and exploratory dataset checks. They are shown separately and compare with the rolling CI baseline, but never contribute to that baseline or publish a Check or PR comment.

Track them with three identities:

  • a unique run_id for one execution attempt
  • a stable manual_group for iterations of the same experiment
  • a stable provenance.target_id for each model or system variant inside the run

Add an optional label for the changed condition and set dirty when the working tree is not clean. The commit remains the nearest reproducible anchor.

Create an entrypoint such as evals/run-manual.ts. Reuse the same suites as CI, run every selected case and target through @ledgelm/evals, and flush once:

import { execFileSync } from 'node:child_process';
import { randomUUID } from 'node:crypto';
import { evaluate } from '@ledgelm/evals';
import { flush } from '@ledgelm/reporter';

const git = (...args: string[]) => execFileSync('git', args, { encoding: 'utf8' }).trim();
const commit = git('rev-parse', 'HEAD');
const branch = git('branch', '--show-current') || `detached-${commit.slice(0, 8)}`;
const dirty = git('status', '--porcelain').length > 0;
const group = process.env.LEDGELM_MANUAL_GROUP ?? 'support-model-bakeoff';

for (const target of targets) {
	for (const testCase of cases) {
		await evaluate({
			name: 'support-answer-quality',
			input: testCase.input,
			expectedOutput: testCase.expected,
			run: (input) => target.invoke(input),
			assess: ({ output, expectedOutput }) => judge(output, expectedOutput),
			report: {
				type: 'judge',
				provenance: {
					test_case_id: testCase.id,
					target_id: target.id,
					target: { provider: target.provider, model: target.model }
				}
			}
		});
	}
}

const result = await flush({
	context: {
		run_kind: 'manual',
		run_id: `manual:${group}:${Date.now()}:${randomUUID()}`,
		manual_group: group,
		label: process.env.LEDGELM_MANUAL_LABEL,
		dirty,
		commit_sha: commit,
		branch
	}
});
console.log(result.dashboard_url ?? result.verdict);
LEDGELM_MANUAL_GROUP=support-model-bakeoff LEDGELM_MANUAL_LABEL='temperature 0.2' bun run evals/run-manual.ts

Local reporters default to run_kind: manual; setting it explicitly makes the entrypoint’s intent clear. GitHub workflow_dispatch still counts as CI. Treat manual results as exploratory and rerun the selected setup in CI before using it as merge evidence.

Organize the client eval project

Keep cases, targets, assessors, suites, and entrypoints separate:

evals/
  cases/
  targets/
  assessors/
  suites/
  run-ci.ts
  run-manual.ts

Cases own durable test IDs and optional expected outputs. Targets own stable variant IDs and model provenance. Assessors own scoring policy. Suites combine them and call evaluate() but never flush. Entrypoints select suites and flush once. Both entrypoints should reuse the same suites so local and CI semantics do not drift.

Project configuration

Configuration is optional. Unlisted evals still appear as lazy informational signals. Add .ledgelm.yml only when you want to interpret, promote, threshold, or archive an eval.

# yaml-language-server: $schema=https://YOUR_LEDGE_HOST/schemas/latest/ledgelm-config.schema.json
project: my-app
baseline_branch: main
rolling_window: 20

evals:
  response_grounding:
    type: judge
    blocking: true
    thresholds:
      min_score: 0.8
      max_score_drop: 0.05

  json_schema_valid:
    type: deterministic
    blocking: true

  tone_check:
    type: judge
    blocking: false

  old_relevance_check:
    archived: true

YAML is used because it is readable in code review and cannot execute repository code. JSON is also valid YAML. JavaScript and TypeScript configuration files are intentionally unsupported.

Agent skill

LedgeLM includes a Codex-compatible skill that teaches coding agents what LedgeLM does, how to integrate either reporter, and how to preserve useful eval provenance. The skill is guidance for the agent; it does not replace installing @ledgelm/reporter or ledgelm in the client project.

Install for one repository

Run this from the client repository:

npx skills add NexiHealth/ledgelm@ledgelm

The Skills CLI discovers .agents/skills/ledgelm in the LedgeLM repository and installs it for the current project. Commit the installed skill metadata when everyone working in that repository should use the same guidance.

Because LedgeLM is an internal repository, your local GitHub credentials must have access to NexiHealth/ledgelm.

The installed skill entry point is:

.agents/skills/ledgelm/SKILL.md

Install personally

To make the skill available across your Codex workspaces:

npx skills add NexiHealth/ledgelm@ledgelm -g -y

Start a new Codex session after installing it so the skill catalog is refreshed. Invoke it explicitly with $ledgelm, for example:

Use $ledgelm to add Python eval reporting to this repository.

The skill can also be selected automatically for tasks involving LedgeLM reporters, result provenance, CI setup, or .ledgelm.yml.

Use npx skills check to check for updates and npx skills update to update installed skills.

Operational model

LedgeLM stores raw results first, then computes versioned summaries. GitHub Check publication happens asynchronously through a retryable queue, so a temporary GitHub failure does not corrupt the run. Configuration is read at the reported commit and snapshotted to keep historical verdicts reproducible.