The AI Agent Debugging Checklist

A 14-point triage sequence for when your AI agent gives wrong or inconsistent output.

This checklist walks through input validation, prompt clarity, output formatting, determinism checks, context window limits, tool failure modes, and grounding verification. It includes a ready-to-run evaluation prompt you can drop into your test suite.

• Input Integrity: verify data format, encoding, and schema before the agent sees it
• Prompt Specificity: check for ambiguous instructions, missing examples, or conflicting constraints
• Output Format: confirm the structure matches what downstream code expects
• Determinism: run the same input twice and compare outputs for variance
• Context Truncation: inspect token counts and ensure critical information sits in the context window
• Silent Tool Failures: log every tool call and check for empty or unexpected return values
• Hallucination Triage: compare claims against source documents or knowledge base entries
• Grounding Verification: validate that citations and references map to real data
• Retry and Fallback Behavior: test circuit breakers, exponential backoff, and degraded modes
• Logging and Observability: capture prompts, outputs, latencies, and error traces at each stage
• Golden Test Set: maintain a frozen set of inputs with known-good outputs for regression checks
• Versioning: track model versions, prompt versions, and dependencies in a reproducible log
• Evaluation Prompt: a copy-paste prompt template for running structured judgment against expected answers