The most frequently used word in AI engineering discussions during the first half of 2026 was "loop." On June 7, Peter Steinberger said that instead of feeding a prompt into a coding agent, engineers should design a loop that feeds the agent prompts. That same week, Anthropic's Boris Cherny said on stage that he no longer writes prompts — he writes loops, and the loop does the work. Addy Osmani's "Loop Engineering," swyx's (Shawn Wang) "Loopcraft," and LangChain's "The Art of Loop Engineering" were all published in quick succession within June, and the word dominated the main stage at the AI Engineer World's Fair. The closing session on July 2 ended in an hour-long debate over whether the hype around loops had outrun what actually works.

The problem is that people talking about loops are pointing at different things. Arize AI recently published a blog post arguing that at least four distinct architectures are hiding under a single word, distinguishing them as execution loops, task loops, product loops, and system loops — and naming the outermost ring that governs all four the "oversight loop."

This article walks through Arize AI's definitions for each of the four loops — what they terminate on and where humans fit in — then looks at what signals are needed to confirm a loop is actually closing, and where Arize AX fits into that picture.

☑️ What Is a "Loop" in AI Engineering?

In AI engineering, a loop is a structure where an agent acts, receives the result of that action back as input, and repeats the next cycle. Arize AI explains that the word currently refers to four simultaneous architectures — execution loops, task loops, product loops, and system loops — each differing in what it repeats, what ends it, where a human intervenes, and what time scale it operates on.

A structure where feedback never flows back into the next cycle isn't a loop; it's a pipeline. Designing a loop, then, means deciding together what ends it and where that termination signal comes from.

Execution Loops: Act, Observe, Decide the Next Action

The unit an execution loop repeats is the steps within a single task. The signal that ends the loop is environmental feedback — test output, an API response, file contents. Humans generally don't appear mid-loop; they approve the plan or review results at the boundary. 

But this loop also ends the moment the agent itself decides it's done — regardless of whether the task is actually finished. The industry's first answer to this problem was to wrap the execution loop in another loop that doesn't simply accept the agent's own judgment.

Task Loops: Restarting the Agent Until Tests Pass and Specs Are Met

The task loop was the first to get a name, and Geoffrey Huntley's Ralph Loop is its representative example. The Ralph Loop repeatedly restarts a coding agent against the same spec, but allocates a completely fresh context window on every iteration and performs exactly one unit of work per loop.

What looks wasteful is the whole point. Re-feeding the full spec every time prevents context rot and compaction events — the gradual quality decline that plagues long-running sessions.

The unit a task loop repeats is a single deliverable, and the signal that ends it is spec compliance and passing tests. Humans write the spec and judge completion. Huntley assigns people one more role: watch the loop, find failure patterns, and fix things so those patterns don't recur. He compared this role to an engineer keeping a locomotive on its rails.

Product Loops: A Software Factory Running Against a Codebase and Backlog

The product loop got the most attention at the AI Engineer World's Fair. Factory's Tereza Tížková defined a software factory as the entire loop and entire lifecycle of developing software with autonomy. Warp's Zach Lloyd broke that lifecycle down into triage, spec, implementation, review, verification, deployment, and monitoring.

Zach Lloyd argues that software engineering is turning into factory engineering: instead of building the product, you build the system that builds the product. Warp applies this to its own open-source repositories under its factory platform Oz, describing an adoption path that starts with low-risk repos and pushes the auto-merged PR rate from roughly 20 percent toward 60 percent. Anthropic likewise reported that 65 percent of its own product team's code is now written through an internal version of Claude Tag, and Mike Krieger described his team's approach as delegated and proactive — rather than instructing an agent to fix a specific bug, it's given responsibility for an area of the codebase, told to monitor feedback channels, and left to pick up work on its own.

Execution loops and task loops have clear termination conditions, but the product loop runs continuously against a codebase and backlog. Its termination signals come from outside the codebase entirely — newly filed issues, production logs, user feedback, review outcomes. The human's role isn't fixed; it becomes configurable. Organizations choose which parts of the lifecycle to automate and where to insert a human checkpoint.

System Loops: Autoresearch That Improves Prompts, Evals, Harnesses, and the Model Itself

The system loop is what Introspection's Roland Gavrilescu calls "autoresearch." An inner loop is the base system doing user-facing work; an outer loop researches and maintains that base system. What the outer loop repeats against is the prompts, harness, model choice, and the evals themselves. He puts it simply: the loop is the product.

This pattern shows up at both small and large scale. The smallest example is the autoresearch system Andrej Karpathy released in March 2026 — roughly 630 lines of Python that ran 50 hypothesize-modify-evaluate experiments overnight on a single GPU. A production example is Meta's Brain2Qwerty v2, announced in late June, where an agent iteratively modified a codebase to produce a better decoding architecture, reportedly delivering a substantial improvement in word error rate — though the final training configuration was still chosen by a human.

The termination signal for a system loop is the hardest of the four to define. It requires evals, judges, and filtered product feedback, and Gavrilescu's design explicitly includes a tool that lets the agent ask a human — so it can accumulate tacit knowledge the way a new hire would.

☑️ Is Agentic MapReduce a Loop?

One pattern that drew attention around the same time is Cognition's Devin Security Swarm, which fans out boundary-scoped agents in parallel across a repository and aggregates the results — a structure Cognition calls Agentic MapReduce.

Arize AI does not consider this a loop. Dispatch, collection, and validation form a pipeline, because nothing feeds back into the next cycle. In this view, a loop without feedback is just a for-loop, and fan-out is a topology that can sit inside any of the four loops — not a separate loop in its own right.

The Oversight Loop: Where Humans Set Goals, Budgets, and Autonomy Levels

The outermost ring in swyx's loop diagram — above the loop that builds loops — has no name. The role written on that ring is to set goals, allocate resources, and prune work, and its termination condition is marked "none."

Arize AI names this ring the oversight loop: the place where goals are set, budgets allocated, and work triaged — the one ring where a human must always be present. Addy Osmani put it this way on stage: the inner loop is capability, the outer loop is agency, and agency is exactly what the oversight loop holds onto.

Opinions diverge on who should run this ring. Those who favor turning up the autonomy dial argue for deliberately chosen checkpoints, raising autonomy as trust accumulates. Others see a hard stop on that dial. Geoffrey Litt argued that anyone who delegates understanding gets replaced by the agent. Paul Bakaus flatly stated there is no such thing as "automatic," and never will be. HumanLayer's Dex Horthy said he isn't against loops in general — Kubernetes itself is built on a control loop, but it's a deterministic one — and voiced concern that the hype has outpaced the engineering.

The most concrete evidence in this debate came from inside Anthropic itself. Even the team running Claude Tag reported a bottleneck at the review stage — specifically, in a human's ability to conceptualize what the system is actually doing. The checkpoint humans left for themselves has now become the constraint.

How Do You Confirm a Loop Is Actually Closing? 

Autonomy is a separate dial for each of the four loops. You can run a fully autonomous execution loop inside a heavily supervised product loop, or leave a system loop to an agent while keeping all goal-setting with a human. The question Arize AI raises isn't which camp is right — it's what information you need to set each dial correctly.

The key point is that naming a signal and actually wiring it up are two different things. Arize AI notes that a loop whose signal isn't wired won't converge — it will simply keep running until something external stops it. Confirming that a loop is actually closing at production scale means continuously scanning traces and clustering failures, not spot-checking transcripts. That's the job Arize AX is designed to do.

☑️ How Arize AX Closes the Agent Improvement Loop

Arize AX is an AI observability and evaluation platform that records how LLM, RAG, and AI agent services execute as traces, measures the quality of that behavior with evals and judges, and connects those measurements to actual improvement work. In loop terms, what Arize AX handles is the termination signal itself: traces show what the agent did, evals and judges show whether that behavior was good, and experiments show whether a change was actually an improvement.

Production traffic surfaces edge cases that pre-release testing misses. Some failures — timeouts, tool errors, broken JSON — trip an alert. Others — retrieving the wrong context, skipping a required step, passing bad arguments to a tool, or producing a plausible-sounding answer built on faulty reasoning — only show up when you look at the trace itself. Arize AX ties this into a single engineering workflow, aiming to turn agent improvement into a repeatable process rather than a fresh investigation every time.

Mapped onto the four loops above, the role becomes clear. Execution-loop behavior is captured as traces and spans, so tool calls and decision paths can be inspected after the fact. The product loop's termination signals — issues, logs, review outcomes — are organized into investigable form through Signals and managed agents. The system loop's termination signals — evals and judges — are produced through Agent-as-a-Judge and agent experiments. And the oversight loop, where teams decide how far to raise autonomy, draws on all of these signals as evidence.

When Traces Outpace What Humans Can Review

Agent traffic scales; human review doesn't. You can spot a problem in a single trace, but finding patterns that repeat across thousands or millions of traces quickly becomes impossible.

Signal generates the product loop's termination signal. It's an always-on AI worker that continuously reviews incoming production traces, remembers issues it has already identified, and surfaces newly emerging failure patterns. It groups related traces into an investigation report with a summary, root-cause analysis, impact assessment, and suggested next steps — replacing reactive debugging that waits for an alert with a prioritized, evidence-backed issue list waiting at the start of the day.

From Signal to Investigation to Fix

Even after a failure is identified, a team still has to investigate the trace, work out what changed, find the root cause, decide on a fix, and verify the result — and most of that remains manual work.

Managed agents handle the step of turning a signal into an investigation and a proposed fix. They orchestrate long-running, repository-aware agents that inspect traces, reach into external systems, analyze code, produce investigation artifacts, and propose changes as pull requests. They can be configured for engineering workflows like regression triage, debugging production behavior, dataset curation, eval generation, security review, or code fixes. These workers aren't autonomous production agents that apply changes themselves — they investigate, gather evidence, and propose; engineers approve or reject.

Swarm observability gives a single view into the state, activity, trajectories, token usage, and cost of this growing population of AI workers — operational information that feeds directly into decisions about where to place checkpoints in the product and system loops.

Verifying That a Change Is Actually an Improvement

A production agent isn't just a prompt. It's a system made of tools, retrieval, routing, memory, a model, fallbacks, application code, and business logic — and a small change in any one layer can improve one behavior while breaking another.

Agent experiments correspond to the loop's verification step. A curated dataset is run through the entire agent system, and outputs, traces, and eval results are compared across runs to determine whether a change actually improved behavior or introduced a new regression. This is where teams judge whether tool use improved, whether latency shifted, whether retrieval quality held up, or whether a model upgrade improved final answer quality without degrading tool-call reliability.

When Unexpected Failure Modes Show Up

Traditional LLM judges work best when you already know what to look for. But agent failures are rarely that predictable — new failure modes emerge in production as agents interact with tools, users, and a changing environment.

Agent-as-a-Judge updates the system loop's termination signal — the eval itself — to match production behavior, helping the improvement loop adapt when failure modes appear that the team never anticipated. You describe what good behavior looks like, and an agentic judge inspects traces, identifies relevant spans, classifies issues, and generates labels that feed back into monitoring, evals, and experiments.

On top of this, voice agent support lets teams observe, search, replay, and evaluate audio sessions, transcripts, and multimodal traces. Bringing voice conversations into the same observability and evaluation workflow as text agents extends the improvement loop's reach to conversational AI systems as well.

☑️ Frequently Asked Questions

What is a loop in AI engineering?

A loop is a structure where an AI agent acts and receives the result back as input for the next cycle. Arize AI explains that the word currently refers to four simultaneous architectures — execution loops, task loops, product loops, and system loops — each differing in what it repeats, what signal ends it, and where a human fits in.

What's the difference between an execution loop and a task loop?

An execution loop repeats tool calls and result observation within a single task and ends on environmental feedback. A task loop repeatedly restarts an agent against the same spec and ends when tests pass and the spec is met. A task loop allocates a fresh context window on every iteration to prevent the context rot that builds up in long-running sessions.

Is Agentic MapReduce a loop?

Arize AI treats Agentic MapReduce as a pipeline, not a loop, because its dispatch-collect-validate structure has no feedback flowing back into the next cycle. Fan-out is a topology that can be placed inside any of the four loops.

How do you confirm a loop is actually closing?

You need to verify that the loop's termination signal is actually wired into the system. Arize AI notes that a loop whose signal isn't wired won't converge, and that at production scale this requires continuously scanning traces and clustering failures rather than spot-checking transcripts.

What role does Arize AX play in the agent improvement loop?

Arize AX provides a single workflow that records behavior as traces, generates signals through evals and judges, produces investigations and fixes through managed agents, and verifies the effect of changes through experiments. Signal, managed agents, swarm observability, agent experiments, Agent-as-a-Judge, and voice agent support are the features that make up this workflow.

☑️ Closing

The four loops take different forms, but the same underlying practice sits beneath all of them: raising the level of abstraction and pushing human judgment further up the stack. Arize AI's point here is that naming a termination signal and actually wiring it up are two different things — a loop whose signal isn't wired won't converge. Operationally, before raising autonomy levels, it's worth first confirming that the tracing and evaluation infrastructure needed to verify each loop is actually closing is already in place.

Figuring out which loop you're currently building, and whether its termination signal is actually wired up, is the starting point for operating AI agents. As an official Arize AI partner, Cloud Networks can work with clients to apply tracing and evaluation to LLM, RAG, and AI agent services running in production, and to help design, build, and operate an Arize AX–based agent improvement loop.

▶ Learn More About Arize AI

[Source : Arize AI, "What is a loop in AI engineering, anyway?", Arize AI, "Building the AI factory for self-improving agents: What's new in Arize AX"]