{"slug":"stateful-agent-evals","title":"Stateful Agent Evaluation: Datasets, Graders, and CI Gates","tags":["agents","evals","testing","llm","memory","letta","ci-cd"],"agent_summary":"Evaluation framework for memory-capable agents — dataset formats, target modes, extractor types, rubric grading, and CI gate patterns. Covers multi-turn testing and memory block verification.","trigger_phrases":["agent evals","stateful agent testing","memory verification","LLM evaluation","agent grader","eval dataset","agent CI gate"],"runnable":false,"markdown":"\n## Overview\n\nStateful agent evals differ from stateless LLM evals. Agents accumulate memory, use tools, and behave differently across multi-turn conversations. The pipeline tests whether agents remember correctly, use tools appropriately, and produce quality responses.\n\n**Pipeline:** Dataset → Target → Extractor → Grader → Gate → Results\n\n## The Six Concepts\n\n### 1. Dataset\n\nA list of observations to send to the agent. Formats: CSV (single-turn) or JSONL (multi-turn).\n\n```csv\ninput,ground_truth\n\"What is the capital of France?\",Paris\n\"What is the capital of Japan?\",Tokyo\n```\n\nMulti-turn (JSONL):\n```jsonl\n{\"input\": [\"Remember that my favorite city is Paris\", \"What's my favorite city?\"], \"ground_truth\": \"Paris\"}\n{\"input\": [\"My dog is named Biscuit\", \"What is my dog's name?\"], \"ground_truth\": \"Biscuit\"}\n```\n\n### 2. Target\n\nThe agent being evaluated. Three modes:\n\n| Mode | Behavior | Use Case |\n|------|----------|----------|\n| **Agent file** | Clones one fresh agent per observation | Reproducible regression testing |\n| **Agent ID** | Sends all observations to the same live agent | Training/learning loops |\n| **Agent factory** | Function that constructs agents programmatically | Parameterized testing |\n\nAgent file mode gives isolated, reproducible tests. Agent ID mode mutates the agent — useful for training via repeated evaluation.\n\n### 3. Extractor\n\nPulls the specific piece of agent output to grade.\n\nBuilt-in extractors:\n- **last_assistant** — most recent assistant message\n- **memory block** (by label) — contents of a named memory block (e.g., `user_preferences`)\n- **tool_calls** — what tools the agent invoked\n- **tool_returns** — what the tools returned\n- Custom functions for anything else\n\n### 4. Grader\n\nScores the extracted output. Two types:\n\n**Tool-based grading** — deterministic, binary:\n- `contains` — does extracted text contain the ground truth?\n- Score: 1 if yes, 0 if no\n\n**Rubric-based grading** — uses a second LLM as judge:\n- Define weighted criteria (e.g., accuracy 50%, completeness 30%, format 20%)\n- Judge reads rubric + extracted output + ground truth\n- Returns 0–1 score with rationale\n\nAnything you can articulate in a rubric can be scored.\n\n### 5. Gate\n\nPass/fail threshold for CI integration. Throws an error when score falls below threshold.\n\n```python\ngate = EvalGate(min_score=0.67) # 67% of observations must pass\n```\n\nDesigned for GitHub Actions integration — fails the pipeline if agent quality degrades.\n\n### 6. Results\n\nMetrics: average score, standard deviation, error count, per-observation breakdown with scores and rationale.\n\n## Workflow: Build an Eval Suite\n\n### Step 1: Create Dataset\n\n```python\n# Single-turn CSV\ndataset = [\n {\"input\": \"What is my name?\", \"ground_truth\": \"Alice\"},\n {\"input\": \"What city do I live in?\", \"ground_truth\": \"Seattle\"},\n]\n\n# Or multi-turn: teach then test\ndataset = [\n {\n \"input\": [\n \"My name is Alice and I live in Seattle\",\n \"What is my name?\"\n ],\n \"ground_truth\": \"Alice\"\n }\n]\n```\n\n### Step 2: Define Target\n\n```python\nfrom letta_evals import AgentEvalTarget\n\ntarget = AgentEvalTarget(\n agent_file=\"agents/memory-agent.json\", # clones per observation\n # OR\n agent_id=\"agent-abc123\", # same live agent\n)\n```\n\n### Step 3: Configure Extractor\n\n```python\nfrom letta_evals import extractors\n\nextractor = extractors.last_assistant() # default\n# OR\nextractor = extractors.memory_block(\"user_info\") # specific memory block\n# OR\nextractor = extractors.tool_calls() # what tools were called\n```\n\n### Step 4: Configure Grader\n\n```python\nfrom letta_evals import graders\n\n# Simple containment check\ngrader = graders.contains()\n\n# LLM judge with rubric\ngrader = graders.rubric(\n criteria={\n \"accuracy\": {\"weight\": 0.5, \"description\": \"Is the answer factually correct?\"},\n \"completeness\": {\"weight\": 0.3, \"description\": \"Does it cover all required info?\"},\n \"format\": {\"weight\": 0.2, \"description\": \"Is it properly formatted?\"},\n },\n model=\"gpt-4o\",\n)\n```\n\n### Step 5: Run and Gate\n\n```python\nfrom letta_evals import EvalSuite, EvalGate\n\nsuite = EvalSuite(\n dataset=dataset,\n target=target,\n extractor=extractor,\n grader=grader,\n gate=EvalGate(min_score=0.75),\n)\n\nresults = suite.run()\nprint(f\"Score: {results.average_score:.2%}\")\nresults.to_csv(\"eval-results.csv\")\n```\n\n## CI/CD Integration\n\n```yaml\n# .github/workflows/agent-evals.yml\n- name: Run Agent Evals\n run: python run_evals.py\n env:\n LETTA_API_KEY: ${{ secrets.LETTA_API_KEY }}\n OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n```\n\nIf `EvalGate` threshold is not met, the script exits non-zero and the workflow fails.\n\n## Memory Block Testing Pattern\n\nTest that agents store information in the correct memory block:\n\n```python\ndataset = [\n {\n \"input\": [\"I prefer dark mode and always use vim keybindings\", \"check\"],\n \"ground_truth\": \"dark mode\", # should appear in user_preferences block\n }\n]\nextractor = extractors.memory_block(\"user_preferences\")\ngrader = graders.contains()\n```\n\n## Common Eval Mistakes\n\n| Mistake | Fix |\n|---------|-----|\n| Single-turn only | Add multi-turn: teach then probe |\n| Ground truth too strict | Use rubric grading for fuzzy matches |\n| Testing on training data | Hold out 20% of examples for final eval |\n| No gate in CI | Always wire a `min_score` threshold |\n| Grading full response | Use extractors to isolate what you're scoring |\n","html":"

Overview

\n

Stateful agent evals differ from stateless LLM evals. Agents accumulate memory, use tools, and behave differently across multi-turn conversations. The pipeline tests whether agents remember correctly, use tools appropriately, and produce quality responses.

\n

Pipeline: Dataset → Target → Extractor → Grader → Gate → Results

\n

The Six Concepts

\n

1. Dataset

\n

A list of observations to send to the agent. Formats: CSV (single-turn) or JSONL (multi-turn).

\n
input,ground_truth\n\"What is the capital of France?\",Paris\n\"What is the capital of Japan?\",Tokyo\n
\n

Multi-turn (JSONL):

\n
{\"input\": [\"Remember that my favorite city is Paris\", \"What's my favorite city?\"], \"ground_truth\": \"Paris\"}\n{\"input\": [\"My dog is named Biscuit\", \"What is my dog's name?\"], \"ground_truth\": \"Biscuit\"}\n
\n

2. Target

\n

The agent being evaluated. Three modes:

\n

| Mode | Behavior | Use Case |\n|------|----------|----------|\n| Agent file | Clones one fresh agent per observation | Reproducible regression testing |\n| Agent ID | Sends all observations to the same live agent | Training/learning loops |\n| Agent factory | Function that constructs agents programmatically | Parameterized testing |

\n

Agent file mode gives isolated, reproducible tests. Agent ID mode mutates the agent — useful for training via repeated evaluation.

\n

3. Extractor

\n

Pulls the specific piece of agent output to grade.

\n

Built-in extractors:

\n\n

4. Grader

\n

Scores the extracted output. Two types:

\n

Tool-based grading — deterministic, binary:

\n\n

Rubric-based grading — uses a second LLM as judge:

\n\n

Anything you can articulate in a rubric can be scored.

\n

5. Gate

\n

Pass/fail threshold for CI integration. Throws an error when score falls below threshold.

\n
gate = EvalGate(min_score=0.67)  # 67% of observations must pass\n
\n

Designed for GitHub Actions integration — fails the pipeline if agent quality degrades.

\n

6. Results

\n

Metrics: average score, standard deviation, error count, per-observation breakdown with scores and rationale.

\n

Workflow: Build an Eval Suite

\n

Step 1: Create Dataset

\n
# Single-turn CSV\ndataset = [\n    {\"input\": \"What is my name?\", \"ground_truth\": \"Alice\"},\n    {\"input\": \"What city do I live in?\", \"ground_truth\": \"Seattle\"},\n]\n\n# Or multi-turn: teach then test\ndataset = [\n    {\n        \"input\": [\n            \"My name is Alice and I live in Seattle\",\n            \"What is my name?\"\n        ],\n        \"ground_truth\": \"Alice\"\n    }\n]\n
\n

Step 2: Define Target

\n
from letta_evals import AgentEvalTarget\n\ntarget = AgentEvalTarget(\n    agent_file=\"agents/memory-agent.json\",  # clones per observation\n    # OR\n    agent_id=\"agent-abc123\",               # same live agent\n)\n
\n

Step 3: Configure Extractor

\n
from letta_evals import extractors\n\nextractor = extractors.last_assistant()           # default\n# OR\nextractor = extractors.memory_block(\"user_info\")  # specific memory block\n# OR\nextractor = extractors.tool_calls()               # what tools were called\n
\n

Step 4: Configure Grader

\n
from letta_evals import graders\n\n# Simple containment check\ngrader = graders.contains()\n\n# LLM judge with rubric\ngrader = graders.rubric(\n    criteria={\n        \"accuracy\": {\"weight\": 0.5, \"description\": \"Is the answer factually correct?\"},\n        \"completeness\": {\"weight\": 0.3, \"description\": \"Does it cover all required info?\"},\n        \"format\": {\"weight\": 0.2, \"description\": \"Is it properly formatted?\"},\n    },\n    model=\"gpt-4o\",\n)\n
\n

Step 5: Run and Gate

\n
from letta_evals import EvalSuite, EvalGate\n\nsuite = EvalSuite(\n    dataset=dataset,\n    target=target,\n    extractor=extractor,\n    grader=grader,\n    gate=EvalGate(min_score=0.75),\n)\n\nresults = suite.run()\nprint(f\"Score: {results.average_score:.2%}\")\nresults.to_csv(\"eval-results.csv\")\n
\n

CI/CD Integration

\n
# .github/workflows/agent-evals.yml\n- name: Run Agent Evals\n  run: python run_evals.py\n  env:\n    LETTA_API_KEY: ${{ secrets.LETTA_API_KEY }}\n    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n
\n

If EvalGate threshold is not met, the script exits non-zero and the workflow fails.

\n

Memory Block Testing Pattern

\n

Test that agents store information in the correct memory block:

\n
dataset = [\n    {\n        \"input\": [\"I prefer dark mode and always use vim keybindings\", \"check\"],\n        \"ground_truth\": \"dark mode\",  # should appear in user_preferences block\n    }\n]\nextractor = extractors.memory_block(\"user_preferences\")\ngrader = graders.contains()\n
\n

Common Eval Mistakes

\n

| Mistake | Fix |\n|---------|-----|\n| Single-turn only | Add multi-turn: teach then probe |\n| Ground truth too strict | Use rubric grading for fuzzy matches |\n| Testing on training data | Hold out 20% of examples for final eval |\n| No gate in CI | Always wire a min_score threshold |\n| Grading full response | Use extractors to isolate what you're scoring |

\n"}