Stateful Agent Evaluation: Datasets, Graders, and CI Gates

Overview

Stateful agent evals differ from stateless LLM evals. Agents accumulate memory, use tools, and behave differently across multi-turn conversations. The pipeline tests whether agents remember correctly, use tools appropriately, and produce quality responses.

Pipeline: Dataset → Target → Extractor → Grader → Gate → Results

The Six Concepts

1. Dataset

A list of observations to send to the agent. Formats: CSV (single-turn) or JSONL (multi-turn).

input,ground_truth
"What is the capital of France?",Paris
"What is the capital of Japan?",Tokyo

Multi-turn (JSONL):

{"input": ["Remember that my favorite city is Paris", "What's my favorite city?"], "ground_truth": "Paris"}
{"input": ["My dog is named Biscuit", "What is my dog's name?"], "ground_truth": "Biscuit"}

2. Target

The agent being evaluated. Three modes:

| Mode | Behavior | Use Case | |------|----------|----------| | Agent file | Clones one fresh agent per observation | Reproducible regression testing | | Agent ID | Sends all observations to the same live agent | Training/learning loops | | Agent factory | Function that constructs agents programmatically | Parameterized testing |

Agent file mode gives isolated, reproducible tests. Agent ID mode mutates the agent — useful for training via repeated evaluation.

3. Extractor

Pulls the specific piece of agent output to grade.

Built-in extractors:

• last_assistant — most recent assistant message
• memory block (by label) — contents of a named memory block (e.g., user_preferences)
• tool_calls — what tools the agent invoked
• tool_returns — what the tools returned
• Custom functions for anything else

4. Grader

Scores the extracted output. Two types:

Tool-based grading — deterministic, binary:

• contains — does extracted text contain the ground truth?
• Score: 1 if yes, 0 if no

Rubric-based grading — uses a second LLM as judge:

• Define weighted criteria (e.g., accuracy 50%, completeness 30%, format 20%)
• Judge reads rubric + extracted output + ground truth
• Returns 0–1 score with rationale

Anything you can articulate in a rubric can be scored.

5. Gate

Pass/fail threshold for CI integration. Throws an error when score falls below threshold.

gate = EvalGate(min_score=0.67)  # 67% of observations must pass

Designed for GitHub Actions integration — fails the pipeline if agent quality degrades.

6. Results

Metrics: average score, standard deviation, error count, per-observation breakdown with scores and rationale.

Workflow: Build an Eval Suite

Step 1: Create Dataset

# Single-turn CSV
dataset = [
    {"input": "What is my name?", "ground_truth": "Alice"},
    {"input": "What city do I live in?", "ground_truth": "Seattle"},
]

# Or multi-turn: teach then test
dataset = [
    {
        "input": [
            "My name is Alice and I live in Seattle",
            "What is my name?"
        ],
        "ground_truth": "Alice"
    }
]

Step 2: Define Target

from letta_evals import AgentEvalTarget

target = AgentEvalTarget(
    agent_file="agents/memory-agent.json",  # clones per observation
    # OR
    agent_id="agent-abc123",               # same live agent
)

Step 3: Configure Extractor

from letta_evals import extractors

extractor = extractors.last_assistant()           # default
# OR
extractor = extractors.memory_block("user_info")  # specific memory block
# OR
extractor = extractors.tool_calls()               # what tools were called

Step 4: Configure Grader

from letta_evals import graders

# Simple containment check
grader = graders.contains()

# LLM judge with rubric
grader = graders.rubric(
    criteria={
        "accuracy": {"weight": 0.5, "description": "Is the answer factually correct?"},
        "completeness": {"weight": 0.3, "description": "Does it cover all required info?"},
        "format": {"weight": 0.2, "description": "Is it properly formatted?"},
    },
    model="gpt-4o",
)

Step 5: Run and Gate

from letta_evals import EvalSuite, EvalGate

suite = EvalSuite(
    dataset=dataset,
    target=target,
    extractor=extractor,
    grader=grader,
    gate=EvalGate(min_score=0.75),
)

results = suite.run()
print(f"Score: {results.average_score:.2%}")
results.to_csv("eval-results.csv")

CI/CD Integration

# .github/workflows/agent-evals.yml
- name: Run Agent Evals
  run: python run_evals.py
  env:
    LETTA_API_KEY: ${{ secrets.LETTA_API_KEY }}
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

If EvalGate threshold is not met, the script exits non-zero and the workflow fails.

Memory Block Testing Pattern

Test that agents store information in the correct memory block:

dataset = [
    {
        "input": ["I prefer dark mode and always use vim keybindings", "check"],
        "ground_truth": "dark mode",  # should appear in user_preferences block
    }
]
extractor = extractors.memory_block("user_preferences")
grader = graders.contains()

Common Eval Mistakes

| Mistake | Fix | |---------|-----| | Single-turn only | Add multi-turn: teach then probe | | Ground truth too strict | Use rubric grading for fuzzy matches | | Testing on training data | Hold out 20% of examples for final eval | | No gate in CI | Always wire a min_score threshold | | Grading full response | Use extractors to isolate what you're scoring |