The Shift to Agentic Engineering

Building AI agents manually is slow and unscalable. As organizations move toward deploying hundreds of agents, the human developer becomes the primary bottleneck in the development loop. The "Agentic AI Engineer" approach treats agent development as a continuous, automated loop—mirroring software engineering practices but applied to the non-deterministic nature of LLMs.

The Dual-Loop Lifecycle

The development process is split into two distinct, interconnected loops:

1. The Offline Loop (Build & Iterate)

This phase focuses on getting an agent to a "production-ready" state. It begins with Spec-Driven Development, where requirements, constraints, and success criteria are defined as a blueprint. This spec is decoupled from the implementation, allowing developers to switch frameworks or harnesses as the ecosystem evolves. Once the spec is set, the agent is built and subjected to Eval-Driven Development (the agentic equivalent of Test-Driven Development).

2. The Online Loop (Monitor & Optimize)

Once deployed, the agent enters the online phase. Here, the system continuously monitors traces for failures. Instead of manual log review, an autonomous Diagnostics Agent clusters failure modes and performs root cause analysis. These findings are fed back into the optimization loop, generating new evaluation criteria and code-level remedies, which then restart the cycle.

The Role of the Evaluator and Diagnostics Agents

To make this loop autonomous, two specialized agents are critical:

  • Evaluator Agent: Moves beyond simple score-based metrics (which often lack actionable feedback) toward binary, criteria-based evals. A useful eval must provide a clear "call to action" when it fails. These agents must be calibrated to handle the non-deterministic variance of LLM-as-a-judge.
  • Diagnostics Agent: Solves the cost and time problem of reading millions of traces. By using intelligent segmentation and "learned indicators" (code-checkable patterns), the agent identifies representative failure samples, performs recursive root-cause analysis, and suggests specific remedies in markdown format for the developer to apply.

Implementation Strategy

  • Decoupling: Keep the spec separate from the implementation to maintain flexibility as agent frameworks evolve.
  • Actionable Evals: Prioritize binary criteria over vague scoring. If an eval fails, the system should know exactly what to fix.
  • Intelligent Sampling: Do not attempt to read every trace. Use multi-tier filtering to focus on representative failure modes to keep costs lower than the execution itself.
  • Orchestration: Use an orchestrator to manage the hand-off between the diagnostic findings and the coding agent that applies the fix.

Key Takeaways

  • Automate the Loop: If you are building more than a few agents, human-in-the-loop evaluation will fail to scale. Build agents to evaluate and diagnose other agents.
  • Treat Evals as Code: Just as unit tests define software quality, evaluation suites define agent reliability. They should grow and evolve based on production failures.
  • Focus on Failure Modes: Don't just track if an agent failed; cluster failures by root cause (e.g., missing tool, prompt ambiguity) to create targeted remedies.
  • Start with a Spec: Even in an agentic workflow, a clear definition of responsibilities and boundaries is the necessary foundation for any automated build process.
  • Avoid Scoring Noise: Calibrate your LLM-as-a-judge to ensure consistency; otherwise, you cannot conclusively prove that an optimization actually improved performance.

Notable Quotes

  • "The bottleneck basically becomes the human review and the human building time... you can't scale especially if in your organization you're now planning to roll out hundreds of agents."
  • "As soon as you have a lot of features that you need to evaluate and experiment on then it suddenly becomes impossible to do this quickly or in parallel then the human essentially becomes the bottleneck."
  • "Unless your rubric is very well defined then score-based evals does not exactly tell you what to fix; in such cases using binary type of evals or criteria is preferred because there you have a kind of a call to action."
  • "If you have millions of agent traces trying to read all of these it actually costs more than the execution itself, so it's not the most efficient way of diagnosing the problems."