Video · 2026-04-16 · 54m · 6 moments

How Capital One Delivers Multi-Agent Systems [Rashmi Shetty] - 765

✦ AI generated

timeline · colored by role

Mechanism◆

Multi-agentic architecture is needed when a large, complex goal must be broken down into discrete steps, with each step handled by a specific agent tasked with executing that narrow task before handing off to the next.

Rashmi explains the founding principle behind Capital One's use of multi-agent systems: complex, multifaceted goals get decomposed into steps, each owned by a dedicated agent.

transcript

Rashmi Shetty: We have a large complex goal which we have to break down into specific steps. And each step is basically narrowed to a specific agent. And that agent is tasked with the goal of executing that specific task. And then you move on to the next agent.

explains mechanism · 1gives example · 1

Example◆

Chat Concierge, a multi-agentic auto-dealership chat application with a human in the loop, was Capital One's beachhead project for deploying multi-agent systems, designed to seamlessly match customers to the right car and initiate follow-up actions like scheduling test drives.

Rashmi describes Chat Concierge, Capital One's first production multi-agent system, built to bridge auto dealers and car-buying customers through a personalized, agentic chat experience.

transcript

Rashmi Shetty: Chat Concierge is essentially auto dealership project or application that was deployed out to our auto dealers to basically bridge that experience between dealers and their customers and make it very seamless... it was a multi-agentic chat experience that was brought to the fore with a human in the loop.

Mechanism

Because Capital One operates deeply embedded within the banking industry's model risk framework, its platform bakes in risk and compliance as policies, guardrails, and security/cyber enforcement across every layer and thread boundary of its agentic systems.

Rashmi explains that Capital One's close partnership with its model risk office means compliance is embedded as guardrails and policy enforcement throughout the agentic platform, not bolted on afterward.

transcript

Rashmi Shetty: We are deeply embedded within the model risk framework that the banking industry gives you... we have all of the risk and compliance uh frameworks embedded within the platform, which appear as policies, as guardrails, as security enforcement, cyber enforcements across our different layers of the platform.

Mechanism◆

Observability in multi-agentic systems is more critical than in single-model systems because these are stochastic, interacting probabilistic systems, requiring the ability to replay agent actions and observe behavior across dimensions like tool invocation, reasoning process, and context propagation.

Rashmi explains that because multi-agent systems are stochastic and interacting, observability must extend to replaying agent actions, tool invocations, reasoning chains, and cross-system latency.

transcript

Rashmi Shetty: All the more important for us to be able to replay agentic actions and try to understand how it function. Agent behavior needs observability along many different dimensions in terms of what are the tools invoked? How what was the reasoning mechanism that led to that tool invocation? And overall, what was its context that passed across systems?

Mechanism

Effective evaluation for multi-agent systems must be end-to-end rather than per-agent, because evaluating an individual agent in isolation is meaningless unless it also works correctly upstream and downstream within the whole system.

Rashmi argues that eval frameworks for agentic systems must shift from evaluating individual agents to evaluating the whole pipeline end-to-end, since isolated agent performance doesn't guarantee system success.

transcript

Rashmi Shetty: Eval frameworks are basically tuned to do end-to-end evals rather than individual agent evals, because individual agent evals gives you nothing unless it works upstream and downstream for the whole system.

extends · 1

Claim◆

Anyone building enterprise agentic AI should treat it as a full system requiring governed data, risk controls baked into multiple layers, end-to-end latency optimization, and telemetry from post-production as a critical source of improvement.

As a closing lesson, Rashmi advises organizations earlier in their agentic journey to treat agentic AI holistically as a system, grounded in governed data, layered risk controls, and continuous post-production learning.

transcript

Rashmi Shetty: You really need to treat agentic AI as a system. It's truly a system. You have to start with governed data. You have to kind of put in that risk controls baked into multiple layers of your application or your system. You have to look at latency as something that needs to be optimized end-to-end... your biggest gains do come from post-production telemetries also critical.

gives example · 1

Highlight slides

Why Multi-Agent Architecture?✦ from: Multi-agentic architecture is needed when a large, complex goal must be broken down into discrete steps, with each step handled by a specific agent tasked with executing that narrow task before handing off to the next.Handoff-Driven Execution✦ from: Multi-agentic architecture is needed when a large, complex goal must be broken down into discrete steps, with each step handled by a specific agent tasked with executing that narrow task before handing off to the next.Chat Concierge: Capital One's Beachhead Multi-Agent System✦ from: Chat Concierge, a multi-agentic auto-dealership chat application with a human in the loop, was Capital One's beachhead project for deploying multi-agent systems, designed to seamlessly match customers to the right car and initiate follow-up actions like scheduling test drives.Purpose & Follow-Through✦ from: Chat Concierge, a multi-agentic auto-dealership chat application with a human in the loop, was Capital One's beachhead project for deploying multi-agent systems, designed to seamlessly match customers to the right car and initiate follow-up actions like scheduling test drives.Why Multi-Agent Systems Need Observability✦ from: Observability in multi-agentic systems is more critical than in single-model systems because these are stochastic, interacting probabilistic systems, requiring the ability to replay agent actions and observe behavior across dimensions like tool invocation, reasoning process, and context propagation.Key Observability Dimensions✦ from: Observability in multi-agentic systems is more critical than in single-model systems because these are stochastic, interacting probabilistic systems, requiring the ability to replay agent actions and observe behavior across dimensions like tool invocation, reasoning process, and context propagation.Treat Agentic AI as a Full System✦ from: Anyone building enterprise agentic AI should treat it as a full system requiring governed data, risk controls baked into multiple layers, end-to-end latency optimization, and telemetry from post-production as a critical source of improvement.Post-Production Telemetry Drives Gains✦ from: Anyone building enterprise agentic AI should treat it as a full system requiring governed data, risk controls baked into multiple layers, end-to-end latency optimization, and telemetry from post-production as a critical source of improvement.

Related episodes

Video · The Cognitive Revolution

Intelligence on the Edge: Liquid AI's Ramin Hasani on the Search for Device-Native Foundation Models

→

Video · 20VC

Should the US Ban Chinese Open-Source Models | OpenRouter's Chance To Sell | Stripe Buying PayPal

→

Video · My First Million

I spent 24hrs with a blue collar millionaire

→