Your Agent Sees Its Own Mistakes But Not Your Users’ Fixes: What CopilotKit CLiF Captures

Every agent team eventually hits the same wall. The agent works in demos and struggles in production, so the team writes hundreds of hand-crafted evals, patches prompts, adds tools, and grinds the pass rate up until the thing can ship. Meanwhile the product's own users perform those same tasks every day, hundreds of times, and know exactly which steps in which order actually get the work done. Almost none of that knowledge is captured anywhere an agent can learn from.
CopilotKit's CLiF, released in early access as part of CopilotKit Intelligence, is built on a specific diagnosis of that gap, and the diagnosis is narrow enough to be checkable against any team's own experience: agent tracing shows where the agent went wrong, while the correct path lives in what users do in the product, which most stacks never record. CLiF's answer is the full-stack trajectory, the agent's trace and the user's actions captured on one timeline, then converted into skills, production evals and fine-tuning data. This article works from CopilotKit's own product page and release materials, and from the company's account of how the loop works, and it also asks the question a buyer should: when does this approach beat the alternatives.
The gap between two kinds of observability
Agent observability today is agent-centred. Tracing tools record prompts, tool calls and model outputs, which tells you precisely where the agent went wrong and nothing about what right looked like. Product analytics covers the other half, clicks, funnels, session replays, but in a form built for a human reading a dashboard rather than for teaching a machine a task.
Those two datasets live in separate systems in nearly every production stack, and the separation is the problem. An agent learns a product from prompts, skills, tool descriptions and exposed APIs, and it has to infer how to combine them for a real task. The people using the product solved that inference long ago through repetition, and their solution exists only in their clicks. Product analytics records those clicks without their meaning; session replays contain the meaning but at a density no human will watch. Closing the gap requires capturing both sides in one stream, at capture time, rather than stitching them together afterwards.
What a full-stack trajectory is
CopilotKit's primitive is the full-stack trajectory, and it is captured through AG-UI, the Agent-User Interaction Protocol the company created to connect agents to user-facing applications. Because AG-UI sits between the agent and the user on every surface, it can see both sides of a task at once.
What CLiF captures on the agent side: every run, tool call and state change, with what each returned. On the product side: what was on the user's screen, the buttons they clicked, the server requests those clicks triggered, and any conversation with the agent, all in the order it happened. Because AG-UI already streams agent events to the frontend, capturing user actions over the same protocol puts both halves in one stream from the start, with no second analytics pipeline to reconcile later.
The practical payoff is replay. Each user session becomes a single trajectory a team can open and step through, clicks, requests and agent messages interleaved as they occurred. That artefact is the unit of learning, and it is also the unit of evidence: any lesson derived from it can cite the exact sessions behind it.
The four-stage loop
CLiF's learning loop has four stages, and the sequencing is what makes it governable.
Capture. Trajectories stream into CopilotKit Intelligence and CLiF assigns them to learning containers, which scope learning to one kind of work such as billing support or customer onboarding. The runtime picks a container per session through a callback, sessions can be moved between containers later, and each container learns only from its own sessions. That scoping is the design decision that prevents the classic failure of continual learning in production: a model that learns a lesson from one workflow and misapplies it in another.
Find patterns. CLiF runs analysis on a schedule once a container passes a threshold of new sessions, or on demand. Each run works from a frozen snapshot and surfaces Insights, recurring requests, repeated corrections and the friction users quietly work around. CopilotKit treats that friction as the strongest signal, because it repeats across thousands of sessions and shows exactly what the agent has not learned. Every Insight cites the events behind it and links to the source trajectory, which is what makes the next stage reviewable.
Review. When the evidence supports a reusable lesson, CLiF proposes it as a skill written as a plain SKILL.md file, and CLiF cannot publish it without a person. The analysis service cannot publish anything on its own: a person approves or rejects each proposal, in the UI or through the CLI. Published skills are versioned and can be retired at any time. For a regulated industry, that human gate is not a nicety; it is the difference between a system an auditor can accept and one they cannot.
Deliver and export. CLiF delivers published skills to the agent without a redeploy, and stability-minded teams can pin an agent to a fixed skill revision. Skills can also be downloaded as plain files for any other agent or MCP server. Trajectories export as raw event data, and for early-access teams the same pipeline produces eval candidates and fine-tuning datasets.
The worked example that shows why this matters
CopilotKit's illustration is worth repeating because it captures the category's value better than any feature list. Imagine a project management platform with a one-click sprint close-out that moves every unfinished issue into the next sprint, documented exactly that way. An agent asked to close out a sprint does precisely that.
Thousands of captured trajectories then reveal what team leads actually do: they use the one-click option, then spend a few minutes repairing the result, moving already-carried-over issues back to the backlog and unassigning them, and commenting on blocked issues to tag whoever owns the blocker. Nobody designed that routine. Few users would describe it if asked. It exists only in the clicks that follow the one-click move, invisible to agent traces because users perform it themselves and opaque to product analytics because the clicks arrive without their meaning.
CLiF surfaces that routine as an Insight citing the sessions behind it, exactly the kind of finding CLiF was built to produce, proposes a skill for the full close-out, and after human review the built-in agent picks it up without a redeploy. The same skill ships with the MCP server so external agents follow it too. That last step is the strategically interesting one: the platform's tribal knowledge becomes an interface contract that every connected agent inherits.
What it produces, and how it fits existing stacks
The loop's outputs are deliberately plural, because different teams need different artefacts. Skills, for agents or MCP servers, built from patterns that recur across sessions. Production-based evals, built from tasks users actually performed, including edge cases nobody thought to write by hand, exportable to whatever eval platform the team already runs. Fine-tuning datasets drawn from real trajectories, aimed at moving high-volume workflows onto smaller, faster open models. And raw trajectories, exportable to any observability platform.
CLiF skill delivery runs through adapters for LangGraph in Python and TypeScript, Mastra, Google ADK, Microsoft Agent Framework, CopilotKit's own agent and others. The adapter adds a catalog and two tools, one to load a skill and one to read its files, and the model loads a skill when the task calls for it. Developer instructions keep precedence, so a learned skill cannot override the agent's role, safety rules or tool restrictions, which is the control that keeps a learning system from quietly becoming an unlearning system.
Deployment is self-hostable by design: CopilotKit Intelligence runs on a company's own Kubernetes cluster via Helm chart, or on AWS ECS and Fargate, with the team's own Postgres, Redis and identity provider. Trajectories and everything learned from them stay inside the VPC or on-premises, and traces can flow into an existing OpenTelemetry collector.
When this beats the alternatives
The honest comparison is against three things teams already do, and CLiF earns its place on a different axis against each.
Against hand-written evals, it wins on coverage of reality. Hand-written cases encode what engineers imagine users do; production trajectories encode what they did, including the repair steps nobody documented. The weakness is bias toward existing behaviour: if your users have a bad workaround for a broken feature, the system will learn the workaround. That is why the human review gate matters more than the automation.
Against prompt patching, it wins on context economy. Each agent mistake tends to add a rule to the system prompt or a tool to the agent, and context fills until behaviour becomes unpredictable and expensive. A skill catalog loaded on demand keeps learned behaviour out of the standing prompt, which is the same context-engineering argument this site has tracked since the inference caching guide and the model routing analysis.
Against fine-tuning from scratch, it wins on data provenance. Trajectories captured with both sides of the task are a better training signal than scraped examples, and the same corpus serves evals and fine-tuning, which means one investment feeds three needs. It complements rather than replaces the harness work on training environments: EnvHarness reshapes environments to teach a diagnosed weakness, CLiF diagnoses the weakness from production and supplies the correction.
What adopting a continual learning loop actually changes
For a team that decides the approach is worth trying, the practical changes are more organisational than technical, and it helps to name them before the first trajectory is captured.
The first change is that someone must own the containers. CLiF scopes learning per learning container, billing support here, onboarding there, and the routing callback decides which sessions land where. That makes container design an operating decision, not a technical one: get the scope wrong and either one workflow's lessons leak into another's behaviour, or the container is so narrow that no pattern ever accumulates. The teams that succeed will be the ones that treat containers the way a support organisation treats queues, as an owned taxonomy that gets reviewed.
The second change is that review becomes a standing job. Every proposed skill passes through a human, gets versioned, and can be retired. In practice that means a named reviewer, a cadence, and a definition of what makes a skill good enough to publish, which is a governance artefact no vendor can supply. The upside is that this same artefact is what an auditor or a risk committee will ask for, and having it from day one is cheaper than retrofitting it.
The third change is that evals stop being a one-time build. Production-based evals grow with usage, which is the point, but it also means the eval suite drifts as user behaviour drifts. A team that ran a fixed suite before now owns a living one, and someone has to notice when a regression is a real regression rather than a change in what users do. That is a genuinely new responsibility, and it is the one most likely to be underestimated.
None of these changes is exotic, and each maps to a familiar practice: queue design, code review, and test maintenance. The reason to name them is that continual learning systems fail on operations far more often than on algorithms, and the teams that plan for the operations get the benefit that the demo promised.
Where the claims stop
- CLiF is in early access. CopilotKit's product page notes that product trajectories are in early access while skill creation and Intelligence self-hosting are available today, so the parts most distinctive to this release are the least battle-tested.
- All described capabilities, including the four-stage loop, container scoping and the worked sprint example, come from CopilotKit's own materials and a sponsored account of the release. No independent deployment results, adoption numbers or measured improvements are published.
- The claim that insights surface “the rule nobody wrote down” is a design intention demonstrated on one illustrative example, not a measured success rate across customers.
- Cost and latency of running continual learning at production volume, including the analysis service's model usage, are not published. Teams should model the analysis spend against the value of the skills produced.
- The human review requirement limits throughput by design. Organisations expecting a fully automatic improvement loop will find the gate constraining, and that constraint is deliberate rather than a gap.
FAQ
What is CopilotKit CLiF?
It is a continual learning platform in early access from CopilotKit, part of CopilotKit Intelligence. It captures full-stack trajectories, which combine the agent's trace with the user's actions in the product on one timeline over the AG-UI protocol, and turns them into four outputs: skills for agents or MCP servers, production-based evals, fine-tuning datasets, and raw trajectory exports. Every proposed skill requires human approval, is versioned, and can be retired.
Why is user action data more useful than agent tracing alone?
Because agent tracing shows where the agent went wrong but not what right looks like, while the correct path is usually in what experienced users do. Product analytics captures those actions without their meaning, and session replays contain the meaning but nobody will watch thousands of hours. Putting both sides on one timeline makes the user's repair steps visible: the moves an experienced operator makes after the obvious action, which is exactly the knowledge an agent lacks and the knowledge nobody writes down.
Does CLiF replace my existing observability and eval tools?
No, and CopilotKit's positioning is explicitly the opposite: keep the eval and observability platforms you already use, and let CLiF supply the data they have been missing. Trajectories export as raw event data to any observability platform, production-based evals export to the eval platform of your choice, and skills download as plain files for any agent or MCP server. The platform is also self-hostable on Kubernetes or AWS ECS with your own Postgres, Redis and identity provider, so data can stay inside your network.
The short version
CopilotKit CLiF attacks a gap that most agent stacks have lived with quietly: tracing shows where an agent failed, product analytics shows what users clicked, and nobody joins the two, so the knowledge of how work actually gets done never reaches the agent. CLiF captures both sides as a full-stack trajectory over AG-UI, scopes learning into containers, surfaces Insights with citations, and proposes skills that a human must approve before they reach the agent. The outputs, skills, evals, fine-tuning data and raw trajectories, feed tools teams already run, and the platform self-hosts. It is early access, its most distinctive capability is its least proven, and its best argument is the one thing every agent team recognises: your users already know how to do the job, and until now your agent had no way to learn it from them.
Sources
- CopilotKit, CLiF product page, covering the full-stack trajectory, AG-UI foundation, self-hosting and early access status: https://www.copilotkit.ai/clif
- CopilotKit, the CLiF release announcement and the Automatic Learning documentation it builds on: https://www.copilotkit.ai/blog