Agentic RAG in Rust

May 10, 2026

Model Context Protocol (MCP) is an open API standard that allows AI models to interact with external tools. Simply put, MCP enables calling functions that can perform tasks, search for information, and retrieve data — without having to stuff everything into a prompt.

When I started diving into LLMs, I ran into a problem: a local model couldn't just scan my project. It's not often talked about, but the size of the model itself isn't the most important factor for Video Random Access Memory (VRAM). Context is what consumes memory. It's what gives the model an understanding of what to do and where to get knowledge from.

Context

The model along with its context must fit entirely in VRAM — otherwise, expect slow performance. There are various approaches that attempt to solve this problem:

KV-cache gets offloaded to regular RAM or even an SSD — context grows, but speed drops. Sliding window simply discards old tokens — fast, but the model forgets the beginning. Token dropping intelligently decides what to throw out, but sometimes loses important information. Ring Attention spreads the cache across multiple GPUs — powerful, but not for local setups. State-space models like Mamba don't store a KV-cache at all the way transformers do — but they still lag behind in quality. There's no silver bullet.

Agentic RAG

How can we give a model a complete understanding of a project of any size or provide it with other necessary information without using VRAM? Through MCP, of course. Databases — fast and reliable, running on the CPU — don't care about the size of your video memory. Agentic RAG is currently the most practical approach: instead of loading everything into context, the model calls tools and only retrieves the data it needs.

How will the model search for data in the database? It all depends on the tools that MCP provides to the model. When passing a set of tools, we describe them. Based on the description, the model understands why it needs a given tool and how to call it.

And here's where it gets interesting. There's MCP with a complex query system — not all models are smart enough to build highly complex search queries. Your MCP will simply be useless for them. MCP tool descriptions are also context. And if you create a huge set of tools with extremely detailed descriptions, you can end up creating a context problem rather than solving one.

The model will search for data using simple queries — pretty much the same way a human does. Give me this, give me that. Models can make mistakes, translate from one language to another, call a nonexistent tool. What kind of database can deliver a relevant answer to text-based queries? Obviously, you need a search database.

Testing

And another important question — how are we going to test MCP? What do we want to learn from the tests?

The model should call the right tool. When we ask it to show a file, it shouldn't start indexing. This requires concise yet informative tool descriptions that don't compete with each other.

The model should get a response even if it hallucinates a tool. For example, instead of the existing get_files, it calls get_project_files. This is our problem, and it can be solved at the MCP level. And we need to make sure the model gets a response no matter what.

The model should get a response even if it didn't understand the request format. Even a simple interface causes problems for models. For instance, get_files has a limit parameter, and the model passes a negative value or null — which it shouldn't have done. This is also an MCP problem — we can't leave the model without data. It's solved through parameter normalization.

It's impossible to predict what the model will do. We need a tool for real model interaction and MCP calls. If we mock the requests and just test the MCP functionality the way we envision it — there's a high chance the model will see it differently, and your perspectives will diverge. The MCP will be non-functional.

Choosing Implementation Tools

Let's say we want to implement Agentic RAG. What language should we use? Generally speaking, MCP can be implemented in any language with the right skills. Somewhere it'll be easier, somewhere harder. But I would consider the choice of language not based on development complexity, but on speed of execution and ease of delivering the project to the user's PC. To me, Rust seems like the ideal option — simple delivery from crates.io with the best performance. Why choose anything else?

There aren't that many alternatives to Elasticsearch. But Elasticsearch is no longer Apache 2.0. Why use it when OpenSearch exists with Apache 2.0? We need full-text search with multilingual analyzers — OpenSearch fully covers this task.

Choosing Implementation Frameworks

Once we've settled on the basic tools, we can look at the available development resources: frameworks for implementing MCP, database queries, testing.

For Rust developers, there's an excellent crate called rmcp that makes it easy to implement an MCP server — the #[tool] and #[tool_router] macros handle all the boilerplate: you describe the parameter structure, add a description, and the tool is ready.

For querying OpenSearch, a regular HTTP client is enough — no special SDK is needed, OpenSearch communicates via REST API. The reqwest crate easily handles this task.

Testing MCP requires a special approach — not mocking requests, but actually running the model through the tools, because it's impossible to predict the model's behavior. But now there's the vibe-tests crate, which allows testing MCP with real LLMs.

Architecture

To implement Agentic RAG, we'll need project analysis. We can't take an entire file and return it in an MCP response. A file can be large, and our context is limited. But fortunately, the model doesn't need the entire file — we can rely on the Abstract Syntax Tree (AST). Extract all the important information the model needs to understand the project. We can also provide the full file when necessary — but that's an option, not the default format.

AST alone isn't enough for good search. We can "enrich the AST" during the indexing process using a model. This significantly improves the search for files the model needs through MCP. In fact, enrichment holds many more strengths than just tags and a general description for search — but that's a topic for another article.

Indexing with enrichment is not a fast task. So incremental indexing is a must — analyze only what has changed. Enrichment is handled by the model, and if you add a cluster of models, indexing can be accelerated horizontally — more PCs, faster analysis. This won't require a GPU that costs as much as Chelsea Football Club.

Another architectural feature is protection against model hallucinations. Request normalization and creating aliases for tools. Normalization is straightforward — make the interface as simple as possible and tweak the request when needed. But aliases aren't so simple. The first idea that comes to mind is to add more tools for different scenarios. But we have to remember that tools are passed to the model, and their descriptions are also context. Another important detail — the more tools there are, the more confusion in the model's head. The alias problem needs to be solved at the routing level — the model doesn't need to know about them. Let it hallucinate freely, but always get what it asked for.

User ↔ LLM → Anti-hallucination → MCP → OpenSearch
        ↑                                │
        │                     ┌──────────┴──────────┐
        │                     ▼                     ▼
        │                    AST              Enrichment
        │                     │                     │
        │                     └──────────┬──────────┘
        │                                ▼
        └─────────────────────────── Response

The End Result

In the Vibe Analyzer project, available on crates.io, I implemented this architecture. The project provides an MCP for Agentic RAG: 11 tools for searching code and documentation, incremental indexing with enrichment via an Ollama cluster, hallucination protection with normalization and 160 aliases. The project runs locally — OpenSearch + Ollama on your own hardware, no data leaks outside.

The project has been thoroughly tested on real-world use cases. I conducted tests with models starting from 3B. Hallucinations don't interfere with MCP operation. 120 end-to-end tests on real models confirm that the architecture works. And three dedicated crates — vibe-ast, vibe-fs, vibe-tests — can be used in your own projects independently of Vibe Analyzer.

In addition to MCP, the project can also generate reports for projects and knowledge bases written in Markdown. This allows not only analyzing projects and using a local MCP, but also passing context to open chats. The use case is simple: scan a project, pass the resulting report into a chat — and the model understands the project's context and navigates it perfectly.

Conclusion

Model Context Protocol is still a bit raw at the moment. There are a large number of protocols, and they change frequently. Not all models are well-trained to use them. But by providing the necessary level of protection and simplicity of interaction, you can get a working MCP for your tasks — even on 3B models.

Rust allows you to achieve maximum analysis performance and MCP speed. Let's not forget about the memory safety that Rust provides. The choice of language was a good one — yes, I had to implement missing crates, but now they exist, and working with MCP in Rust will become even easier.

Good luck exploring LLMs! It's not scary — they're not capable of replacing people, and your job is safe. Maybe not in the same form, but we're human, and we adapt to any external environment — that's our main skill, one not available to LLMs.