LedgeLM Docs
Dashboard

LedgeLM

LedgeLM tells you whether an application change made your AI evaluations worse. It receives results from the evals your team already runs, preserves the context behind those results, and compares pull requests with a rolling mainline baseline.

LedgeLM is an aggregator, not an eval framework. Keep using notebooks, promptfoo, pandas, pytest, Vitest, custom judges, or whatever already works.

What LedgeLM does

One CI run sends many results to LedgeLM. Each result belongs to a commit, branch, and optionally a pull request. LedgeLM then:

  1. stores the raw result and its provenance,
  2. groups results by eval name,
  3. compares the run with recent completed runs on the baseline branch,
  4. publishes one GitHub Check, and
  5. keeps the raw data available for debugging and trend analysis.

The mission is deliberately narrow: make “did this change regress our AI behavior?” quick to answer and easy to investigate.

Eval philosophy

Start lazy

There is no registration step. Report any stable eval name and it appears as an informational eval. Once the signal proves useful, promote it to blocking and add thresholds in .ledgelm.yml.

Compare with a distribution

A single judge call can be noisy. Pull requests are compared with rolling mean, standard deviation, and pass-rate statistics from recent baseline-branch runs, not only with the previous commit.

Preserve provenance

Keep the eval name stable while its meaning stays stable. Record changing dimensions separately:

  • target: model or system being evaluated
  • judge: model scoring the target
  • prompt: prompt name and version
  • dataset: dataset name, version, and split
  • test_case_id: durable test-case identity
  • tags: queryable dimensions such as locale or product area
  • metadata: rationale, traces, and other debugging context

This is what lets future investigations separate a code regression from a model, prompt, judge, or dataset change.

Never make observability an outage

If LedgeLM is unavailable, reporters warn and return an unknown delivery verdict. They do not throw and should not fail the eval job.

The dashboard

The dashboard stays close to the regression workflow:

  • Projects lists repositories connected through the NexiHealth GitHub App.
  • PR comparison shows current scores and pass rates beside rolling baselines.
  • Eval trend shows mainline drift over time, including model provenance.
  • Run detail preserves every test case, rationale, cost, and latency.
  • Commit history helps identify when a regression was introduced.
  • Project settings controls baselines, configuration, archiving, and CI tokens.

Archived evals are hidden from default comparisons and baseline calculations, but their historical data remains available.

GitHub integration

The GitHub App reads repository configuration at the reported commit and writes one Check Run for each PR eval run. It does not authenticate CI uploads.

CI uses a LedgeLM-issued token scoped to one project. From Project → Settings, select Configure GitHub Actions to create:

  • repository secret LEDGELM_API_TOKEN
  • repository variable LEDGELM_API_URL

The token is encrypted for GitHub on the server and is never returned to the browser. Reconfiguring rotates the managed token.

TypeScript

Install

bun add @ledgelm/reporter
# or: npm install @ledgelm/reporter

Report results

import { flush, report } from '@ledgelm/reporter';

report({
	name: 'response_grounding',
	type: 'judge',
	score: 0.82,
	passed: true,
	cost: 0.004,
	latency_ms: 940,
	provenance: {
		test_case_id: 'refund-42',
		target: {
			provider: 'anthropic',
			model: 'claude-sonnet',
			version: '2026-07'
		},
		judge: { provider: 'openai', model: 'gpt-5' },
		prompt: { name: 'support-answer', version: 'v8' },
		dataset: { name: 'support-golden', version: '2026-07', split: 'ci' },
		tags: { locale: 'en', product: 'support' }
	},
	metadata: {
		rationale: 'The response cites the supplied refund policy.'
	}
});

// Call once after all report() calls.
const result = await flush();
console.log(`LedgeLM delivery: ${result.verdict}`);

report() validates and buffers locally. flush() discovers GitHub Actions context, uploads the buffer in chunks, and finalizes the run.

Python

Install

pip install ledgelm
# or: uv add ledgelm

Report results

from ledgelm import flush, report

report(
    name="response_grounding",
    type="judge",
    score=0.82,
    passed=True,
    cost=0.004,
    latency_ms=940,
    provenance={
        "test_case_id": "refund-42",
        "target": {
            "provider": "anthropic",
            "model": "claude-sonnet",
            "version": "2026-07",
        },
        "judge": {"provider": "openai", "model": "gpt-5"},
        "prompt": {"name": "support-answer", "version": "v8"},
        "dataset": {
            "name": "support-golden",
            "version": "2026-07",
            "split": "ci",
        },
        "tags": {"locale": "en", "product": "support"},
    },
    metadata={"rationale": "The response cites the supplied refund policy."},
)

result = flush()
print(f"LedgeLM delivery: {result.verdict}")

Async applications can use await async_flush(). Both variants retain buffered results and return unknown when delivery fails.

GitHub Actions

First use Configure GitHub Actions in the project settings. Then expose the repository secret and variable to the eval process.

TypeScript workflow

name: AI evals

on:
  pull_request:
  push:
    branches: [main]

jobs:
  evals:
    runs-on: ubuntu-latest
    env:
      LEDGELM_API_URL: ${{ vars.LEDGELM_API_URL }}
      LEDGELM_API_TOKEN: ${{ secrets.LEDGELM_API_TOKEN }}
    steps:
      - uses: actions/checkout@v4
      - uses: oven-sh/setup-bun@v2
      - run: bun install --frozen-lockfile
      - run: bun run evals

Python workflow

name: AI evals

on:
  pull_request:
  push:
    branches: [main]

jobs:
  evals:
    runs-on: ubuntu-latest
    env:
      LEDGELM_API_URL: ${{ vars.LEDGELM_API_URL }}
      LEDGELM_API_TOKEN: ${{ secrets.LEDGELM_API_TOKEN }}
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v6
      - run: uv sync --frozen
      - run: uv run python evals.py

The reporter derives the commit, branch, PR number, workflow, run ID, and attempt from the GitHub Actions environment. Explicit context overrides remain available for nonstandard workflows and tests.

Project configuration

Configuration is optional. Unlisted evals still appear as lazy informational signals. Add .ledgelm.yml only when you want to interpret, promote, threshold, or archive an eval.

# yaml-language-server: $schema=https://YOUR_LEDGE_HOST/schemas/latest/ledgelm-config.schema.json
project: my-app
baseline_branch: main
rolling_window: 20

evals:
  response_grounding:
    type: judge
    blocking: true
    thresholds:
      min_score: 0.8
      max_score_drop: 0.05

  json_schema_valid:
    type: deterministic
    blocking: true

  tone_check:
    type: judge
    blocking: false

  old_relevance_check:
    archived: true

YAML is used because it is readable in code review and cannot execute repository code. JSON is also valid YAML. JavaScript and TypeScript configuration files are intentionally unsupported.

Agent skill

LedgeLM includes a Codex-compatible skill that teaches coding agents what LedgeLM does, how to integrate either reporter, and how to preserve useful eval provenance. The skill is guidance for the agent; it does not replace installing @ledgelm/reporter or ledgelm in the client project.

Install for one repository

Run this from the client repository:

npx skills add NexiHealth/ledgelm@ledgelm

The Skills CLI discovers .agents/skills/ledgelm in the LedgeLM repository and installs it for the current project. Commit the installed skill metadata when everyone working in that repository should use the same guidance.

Because LedgeLM is an internal repository, your local GitHub credentials must have access to NexiHealth/ledgelm.

The installed skill entry point is:

.agents/skills/ledgelm/SKILL.md

Install personally

To make the skill available across your Codex workspaces:

npx skills add NexiHealth/ledgelm@ledgelm -g -y

Start a new Codex session after installing it so the skill catalog is refreshed. Invoke it explicitly with $ledgelm, for example:

Use $ledgelm to add Python eval reporting to this repository.

The skill can also be selected automatically for tasks involving LedgeLM reporters, result provenance, CI setup, or .ledgelm.yml.

Use npx skills check to check for updates and npx skills update to update installed skills.

Operational model

LedgeLM stores raw results first, then computes versioned summaries. GitHub Check publication happens asynchronously through a retryable queue, so a temporary GitHub failure does not corrupt the run. Configuration is read at the reported commit and snapshotted to keep historical verdicts reproducible.