Federated MCP vs. a Unified Knowledge Layer

Pete Griffiths

Director of AI Enablement, Atolio

A smarter, cheaper way for AI agents to use enterprise knowledge

Federated MCP gets agents connected to enterprise systems quickly, which makes it a common starting point for pilots. But at enterprise scale, a unified knowledge layer answers the same questions with up to 79% fewer tokens and enforces each person's permissions on every request. Here's what our data shows.

Bar chart of one question answered two ways. Federated MCP takes 13 turns, each bar taller than the last, while a unified knowledge layer takes three short turns.

Overview

Before an AI agent can answer a question about your company, it has to find the right information. How it finds that information decides what each answer costs, how long it takes, and whether the agent ends up showing someone a document they were never supposed to see.

There are two main ways to do that job today, plus two alternatives covered later in this guide. With federated MCP, the agent searches each connected system live, often several in parallel, and pieces the results together itself. With a unified knowledge layer, those same systems are indexed ahead of time with their permissions attached, so the agent asks once and gets back a ranked, filtered answer.

Both run on Model Context Protocol (MCP), the open standard Anthropic introduced in November 2024 to replace one-off integrations with a single way for AI apps to reach outside tools and data. An AI app runs an MCP client, each business system exposes an MCP server, and the model calls that server's tools to search, read, or act. MCP standardized how AI agents connect to business systems. It leaves open how an agent should find information spread across dozens of them, which is what this guide covers.

Many teams building agents route retrieval through MCP because it's the shortest path to a working demo. The servers exist, the client already speaks the protocol, and an agent can search Notion, then Drive, then Linear, reading each result before deciding what to ask next. With three tools and ten users, that logic holds up. With twelve systems and a few thousand users, the token bill and the permission surface grow faster than most budgets planned for, and faster than most IT teams can oversee.

Enterprise data has been around this loop before. From query federation in the 1970s, to service-oriented architecture in the 2000s, and then to data virtualization in the 2010s, each tried to query systems of record live but copy nothing. Each ran into the same problems. The source systems were built to serve their own users, and a flood of outside queries slowed them down. Every system added to the chain added more waiting. And access rules stayed behind in each source, so nothing downstream knew who was allowed to see what. Where data virtualization delivered, it delivered because of a cache.

Today, federated MCP is lap four.

MCP isn't the problem. It's being asked to work in the wrong tier of the architecture. Agents need two things: knowledge to ground their reasoning, and hands to act on systems of record. A unified knowledge layer is built for the first, serving high-volume reads across every source at low token cost. Direct MCP calls are built for the second: checking live state, filing the ticket, updating the record. Federated MCP stretches the action tier to do the knowledge tier's job.

This guide compares the two approaches in the four places where the differences show up in practice:

What the agent sees
How it finds the answer
What each question costs
Whether access holds

Along the way, you'll find plain definitions for terms vendors often use loosely, results from a 40-question pilot, the tradeoffs across four approaches, and a framework for deciding which one fits your workloads.

The Four Differences at a Glance

Side-by-side comparison of federated MCP and a unified knowledge layer on four points: what the agent sees, how it finds the answer, what each question costs, and whether access holds.

What is federated MCP, and what is a unified knowledge layer?

Federated MCP and a unified knowledge layer are two ways to retrieve enterprise information for an AI agent. Federated MCP sends the agent to search each connected system live, through that system's own MCP server. A unified knowledge layer answers from one permission-aware index built from all of those systems ahead of time. A handful of terms come up throughout this comparison, and vendors don't always use them the same way, so here's how this piece defines them.

Model Context Protocol (MCP)

Model Context Protocol (MCP) is an open standard, introduced by Anthropic in November 2024, that defines how an AI application (the host and its client) talks to an external system (the server). A server exposes tools the model can call and resources it can read. Anthropic contributed MCP to the Linux Foundation's new Agentic AI Foundation in December 2025, when the Linux Foundation counted more than 10,000 published MCP servers.

MCP server

An MCP server is the adapter for one system. The Jira MCP server knows how to search Jira, fetch an issue, and create one. It relies on Jira's own search and Jira's own access model.

Federated MCP

Federated MCP is a retrieval pattern where the agent connects to several MCP servers and searches each one itself at query time. Nothing is copied ahead of time. The model decides which source to query, reads the results, and decides whether to query again. People also call this live fan-out or point-to-point retrieval.

MCP gateway

An MCP gateway is a proxy that sits between agents and many MCP servers to centralize routing, authentication, and logging. A gateway changes how calls are managed, but the retrieval pattern underneath is still federated.

Curated index

A curated index is a hand-picked set of content copied into a store built for one assistant or agent. Claude Projects, Google Gemini Gems, and ChatGPT Projects work this way, as does a vector store loaded for a custom agent. The copy is managed apart from the source it came from, including its permissions.

Unified knowledge layer

A unified knowledge layer is a single, continuously updated index of structured and unstructured content from across an organization's systems, with each document's access controls mirrored from its source. The agent queries the index once, through an MCP or API call, instead of querying each system. Atolio's unified knowledge layer also includes a Collaboration Graph that models who works with whom and on what.

Context layer

Context layer is a newer term, still settling across the market. In this piece, it means a curated set of knowledge drawn from the unified knowledge layer and scoped to one use case, often a single agent. It gives that use case the right scope of context, and the asker's permissions still apply at run time.

How does federated MCP retrieval compare with a unified knowledge layer?

In federated MCP, the model does the searching for anything and everything that might address the task at hand. A unified knowledge layer instead runs the search, ranking, and permission checks first, then hands the model a scoped result.

In a federated setup, a cross-system question becomes a loop. The agent looks up which tools might help, searches the first source, reads the results into its context window, decides they're incomplete, and searches the next source. Every turn re-sends the growing conversation to the model, so each step costs more than the one before it. Anthropic's engineering team describes the same mechanic: when an agent calls tools directly, every intermediate result passes through the model. Retrieval quality is also capped by each system's native search, because an MCP server can only return what that system's search finds. Stitching together weak results from five tools gives the model a longer answer with the same gaps, a point Atolio co-founder and CTO Gareth Watts covers in Enterprise MCP: Limitations, Security Risks, and Scale Considerations.

Everyone has made this call. The phone menu handles the simple things, like checking a balance or resetting a password. Ask for anything off the menu and you're passed from department to department, proving who you are and re-telling your story at every transfer, to people who can each see only their own screen. Federated MCP puts your agent on that call. Every system is another transfer, every transfer another credential check, and every turn re-tells the whole story so far. Now and then you reach the veteran who's worked there for twenty years. They know where everything lives and who owns what, check who you are once, and tell you only what you're entitled to know. A unified knowledge layer does that job on every call.

Diagram of two paths. With federated MCP, an agent is passed from Slack to Drive to Jira to Salesforce, proving who it is and re-telling a longer story at each stop. With a unified knowledge layer, it makes one call to a veteran, is checked once, and gets a short, scoped answer.

Running those searches in parallel helps, but only with part of the problem. Modern MCP clients can send several tool calls at once, so a first pass across several systems takes about as long as the slowest one. What comes back doesn't change. Every result still lands in the context window, the model still has to read and reconcile it, and when the first pass misses, the next round depends on what the last one returned, so retries run in turn. Parallel fan-out also tends to go wider: an agent that can query every source simultaneously often does, which pulls more content into context and sends more traffic to source systems, where it counts against their rate limits. The failure modes stay the same too. A call that times out in a parallel batch goes as unreported as one in a sequence.

In a unified knowledge layer , the content from those same systems is already indexed together, with metadata, relationships, and per-document access controls attached. A single user request can span every indexed source. The system then ranks results across all of them, filters them to what the asker can see, and returns a short, cited set. Fast-changing transactional data is a different job. Inventory or a live ticket status would go stale in an index, so that data is better read live from the source through MCP.

Animation of six agents and six source systems. Under federated MCP, every agent connects to every system, which means many connections to maintain. Under a unified knowledge layer, each connects once to one permission-aware index, so there are few.

Federated MCP multiplies connections as agents and systems grow, and every question fans out across them. A unified knowledge layer connects each source once, and then serves every agent from the same permission-aware index.

‍

Dimension	Federated MCP	Unified knowledge layer
Where content sits at query time	In each source system, queried live	In one index, synced from each source
Freshness	Live on every call	Kept current through streaming ingest; data that changes by the minute stays with live MCP calls
Search quality	Inherits each system's native search, one system at a time	One ranking across all sources, using keyword, semantic, and relationship signals
Token cost per question	Grows with every source consulted and every turn, since context is re-read each step	Designed to stay roughly flat as sources are added; one retrieval, one scoped result
Latency	Parallel calls shorten the first pass to the slowest source, but each retry depends on what came back, so follow-up rounds still run in turn	One query against an index that's already built, with no waiting on each source
Permissions	Enforced by each source, under whatever credential the connection holds	Mirrored per document at ingest, kept current as access changes, and enforced against the asker on every request
Setup time	Quick for each system: connect an existing MCP server	Longer: connectors, indexing, and identity mapping come first, with security and permissions set once for every source
Time to first result	Depends on how many systems the agent searches, and how many times	Expected on par or faster: one query returns a ranked answer
Ongoing ownership	One server per system, each with its own auth, rate limits, and version changes	One index to operate, with connectors maintained centrally
Audit trail	Spread across each server's logs	One retrieval log, with trace and lineage for every call

‍

Two rows deserve a closer look. On permissions, federated MCP is only as safe as the credential behind each connection. The MCP specification makes authorization optional, so access control depends on how each server is built and configured. Per-user OAuth lets the source enforce the right boundaries on every call. A shared service account doesn't, and it's common in pilots because it's faster to set up. In call-center terms, it's a department that never asks who you are at all. Broad access also widens the opening for indirect prompt injection, where instructions hidden in retrieved content steer the model. It's the first entry, LLM01, on the OWASP Top 10 for LLM Applications 2025.

On freshness, the two land close together for knowledge content. Federated MCP reads live on every call, and a unified knowledge layer built on streaming ingest reflects changes shortly after they happen at the source, including revoked access and deleted files. For data that changes by the minute, live MCP is the right tool, and it sits alongside the index rather than competing with it.

Any retrieval system will be wrong sometimes. With a secure-by-design unified knowledge layer, the philosophy is to build the system such that the errors fall on the side of showing too little.

A missing document is a support ticket. An exposed one is an incident.

Federated setups also tend to turn into indexes by accident. Retrieval gets slow or the bill arrives, someone adds a cache, and the team now owns a copy of company knowledge with no rule for when it goes stale or who may see it. If that cache hands a document to someone who shouldn't see it, that's a data leak, and nobody may notice until someone reports it. Failed calls make things worse. When three of twelve calls time out, the model still answers from what came back, and nothing in the response says a quarter of the sources never replied.

How many tokens does federated MCP use compared with a unified knowledge layer?

In our 40-question pilot, federated MCP used 68.1 million tokens. Atolio's unified knowledge layer, reached through a single MCP call, used 14.5 million – that's 79% fewer. Atolio's figure includes its own backend processing, so the comparison counts the work the index does in the background.

‍

79%

fewer tokens across all 40 questions

53%

fewer on the median question

86%

fewer on one synthesis question

‍

We built the test to answer one narrow question: for the same real questions over the same sources, how many tokens does each retrieval pattern consume end to end?

How the pilot worked

40 real questions across five types: single-system lookups, questions spanning several systems, time-based questions, long synthesis tasks, and questions the data can't answer
Two setups over the same sources. In the federated setup, Claude connected to Gmail, Google Drive, Notion, and Linear through tuned MCP connections and searched each one itself. In the index setup, Claude made one MCP call to Atolio's unified knowledge layer of those same sources
Runs paired question by question, so neither total includes a retry the other method didn't make
One model (Claude Opus 5), one run, token usage only

What we found

‍

Measure	Federated MCP	Atolio unified knowledge layer
Total tokens, all 40 questions	68.1M	14.5M (79% fewer)
Median question (paired)	Baseline	53% fewer
One synthesis question, start to finish	13 steps, ~443,000 tokens	3 steps, ~64,000 tokens (86% fewer)

‍

The gap between the 79% total and the 53% median question shows where the cost sits. The five most expensive questions accounted for half of federated MCP's total spend. Hard, cross-system questions are where federated retrieval runs away, because each extra step re-sends everything retrieved so far.

That shape has a budget consequence. An index is a fixed cost that tracks corpus size, which grows slowly. Federated retrieval is a variable cost that scales with how curious users are and how many agents they run, so unit economics get worse as adoption improves.

There's a second cost the token bill doesn't show. Agents are built to do work: write the code, update the record, run the workflow. Every token spent finding knowledge is a token not spent reasoning about the task or carrying it out, and a context window crowded with half-relevant results leaves less room for both.

The breakdown of where tokens went supports that reading:

Federated MCP spent most of its budget carrying retrieved content and conversation history back through the model on every turn, with tool lookup a small share.
On the index side, lookup and retrieval barely register; nearly all of the spend is Atolio's backend ranking, filtering, and assembling the scoped result for each question.

An example makes the mechanism more concrete: asked to summarize the main points of an internal slide deck, the federated agent searched Notion, Gmail, Drive, and Linear repeatedly, looked up how to read a slide deck, and opened Drive files, with each step costing more than the last. The index path asked once and wrote the answer.

Paired bar chart of 40 questions sorted by federated MCP token cost. A few hard questions drive half the federated spend, while the unified knowledge layer bars stay low throughout.

For now, this round ran on one model, Claude Opus 5, across four sources and measured tokens only. Additional tests will include additional models, sources, and questions, though the method above is specific enough for any team to rerun on its own stack.