# LLM evaluation — planning worksheet

Reference guide: https://programmatic.llc/resources/llm-evaluation

Audience: AI product teams and reviewers defining acceptance evidence for language-model applications.

Intended output: A representative evaluation set, scoring rubric and release comparison process.

## Working context

- Workflow / system:
- Accountable owner:
- Date:
- Scope and constraints:

## Review checklist

- [ ] Is success defined per task?
  Separate source relevance, grounded response quality, tool behavior and final task completion where applicable.

  Evidence / open question:
  Owner / next action:

- [ ] Does the dataset represent difficult cases?
  Include ambiguous requests, access failures, missing context and known regressions alongside normal traffic patterns.

  Evidence / open question:
  Owner / next action:

- [ ] Are scoring methods calibrated?
  Compare automated judgments with reviewer decisions and document disagreements rather than treating one score as definitive.

  Evidence / open question:
  Owner / next action:

- [ ] Can releases be compared consistently?
  Version examples and configuration, record results and agree which regressions prevent release.

  Evidence / open question:
  Owner / next action:

## Reference sequence

### Define expected behavior

Specify task success, prohibited outputs and the evidence a reviewer needs for each case.

Expected artifact: A task-specific evaluation rubric

### Build representative cases

Combine normal requests, difficult examples, known failures and cases requiring refusal or escalation.

Expected artifact: A versioned evaluation dataset

### Score and investigate

Use appropriate deterministic and human checks; calibrate model-based scores against reviewed examples.

Expected artifact: An error analysis tied to system components

### Compare before release

Run the same evaluation across proposed changes and connect production findings to future test cases.

Expected artifact: A documented release decision and feedback loop

## Decision record

- Decision:
- Evidence considered:
- Remaining assumptions:
- Acceptance criteria:
- Reviewer:
- Next review date:

This worksheet supports discussion; completion is not a certification or a guarantee of project readiness.