Building LLM-Powered Apps in Rust with Rig

If you're building anything with LLMs right now, you're probably writing Python. LangChain, LlamaIndex, the OpenAI SDK — the entire ecosystem gravitates towards Python. And for good reason. It's fast to prototype in and every provider ships a Python client first.

But if you've ever tried to take an LLM-powered app to production, you've felt the cracks. Type safety is an afterthought. Error handling is messy. Performance under load isn't great. And deploying Python services at scale comes with its own baggage.

This is where Rig comes in. It's an open-source Rust library for building LLM applications, and it's genuinely good.

What Is Rig?

Rig gives you a unified API for working with LLMs across providers. Instead of learning a different SDK for OpenAI, Anthropic, Gemini, and the rest, you get one consistent interface. Swap the provider, keep the same code.

use rig::client::CompletionClient;
use rig::completion::Prompt;
use rig::providers::openai;

#[tokio::main]
async fn main() -> Result<(), anyhow::Error> {
    let client = openai::Client::from_env();

    let agent = client
        .agent("gpt-4")
        .preamble("You are a helpful assistant.")
        .build();

    let response = agent.prompt("What is Rust?").await?;
    println!("{response}");

    Ok(())
}

That's a working LLM call in about 10 lines. If you want to switch to Anthropic, you change the provider import and the model name. Everything else stays the same.

Why Rust for LLM Apps?

The honest answer is you probably don't need Rust for a simple chatbot wrapper. Python is fine for that. But once your LLM application starts doing real work — processing documents at scale, running multiple agents concurrently, serving hundreds of requests — Rust's advantages start compounding.

Type safety matters more than you think. LLM responses are unpredictable by nature. Having the compiler enforce the shape of your data, your tool definitions, and your agent configurations catches entire categories of bugs before they hit production.

Concurrency is where Rust shines. Running multiple LLM calls in parallel, streaming responses, coordinating agents — Tokio and Rust's async model handle this naturally. No GIL, no threading hacks.

Memory and performance. If you're building something that processes thousands of documents for RAG, or running an agent loop that makes dozens of LLM calls per request, the overhead of Python adds up. Rust doesn't have that tax.

The Key Abstractions

Rig is built around a few core concepts that keep things clean.

Agents

The Agent type is the main building block. At its simplest, it wraps a model with a system prompt. But it can also hold tools, a knowledge base for RAG, and custom configuration.

let agent = client
    .agent("claude-sonnet-4-5-20250929")
    .preamble("You are a code review assistant. Be concise.")
    .temperature(0.3)
    .build();

Tools

Tools let your agent call functions. You define a struct, implement the Tool trait, and Rig handles the function-calling protocol with the LLM provider.

use rig::tool::Tool;

#[derive(serde::Deserialize, serde::Serialize)]
struct SearchParams {
    query: String,
}

struct SearchTool;

impl Tool for SearchTool {
    const NAME: &'static str = "search";

    type Args = SearchParams;
    type Output = String;
    type Error = anyhow::Error;

    async fn call(&self, args: Self::Args) -> Result<Self::Output, Self::Error> {
        // Your search logic here
        Ok(format!("Results for: {}", args.query))
    }
}

The type safety here is significant. Your tool's input and output types are checked at compile time. No more runtime crashes because the LLM returned a field name you didn't expect.

Extractors

One of Rig's more useful features. Extractors let you pull structured data out of unstructured text using LLMs, with the output typed to a Rust struct.

use rig::extractor::Extractor;

#[derive(serde::Deserialize, schemars::JsonSchema)]
struct ContactInfo {
    name: String,
    email: Option<String>,
    company: Option<String>,
}

You define the shape you want, and Rig handles the prompting and parsing. The result is a proper Rust struct, not a loosely-typed dictionary.

Vector Stores and RAG

Rig supports 10+ vector store backends — MongoDB, Qdrant, SQLite, LanceDB, and others. Building a RAG pipeline means connecting an embedding model, indexing your documents, and attaching the vector store to an agent as a knowledge base.

The key insight is that all of this uses the same trait interfaces. Swap SQLite for Qdrant by changing a few lines, not rewriting your pipeline.

Provider Support

This is where Rig really delivers. Out of the box, it supports over 20 providers:

OpenAI, Anthropic, Gemini, Deepseek, Groq, Mistral, Ollama, Cohere, Together, xAI, Perplexity, OpenRouter, and more.

Running models locally with Ollama? Same API. Hitting a cloud provider? Same API. That consistency is something the Python ecosystem still doesn't have cleanly.

Getting Started

Add it to your project:

cargo add rig-core

Set your API key as an environment variable (e.g., OPENAI_API_KEY), and you're ready to go. The GitHub repo has examples covering agents, RAG, tool use, streaming, and multi-provider setups.

Where It Sits Today

Rig is at version 0.30 and under active development. The API is stabilising but still evolving. Production users include organisations in genomics, blockchain, and AI agent platforms, so it's not just a toy.

The WASM compatibility is worth noting too. You can compile Rig-powered logic to WebAssembly and run it in the browser or at the edge. Try doing that cleanly with LangChain.

Should You Use It?

If you're already writing Rust and want to add LLM capabilities to your application, Rig is the obvious choice. The ergonomics are surprisingly good for a Rust library, and the multi-provider support means you're not locked in.

If you're coming from Python specifically to build LLM apps, the value depends on your use case. For prototyping and experimentation, Python is still faster to iterate in. For production services that need performance, type safety, and reliable concurrency — Rust with Rig is a compelling option.

The AI tooling space moves fast. Most of it moves in Python. But Rig is proof that Rust has a real seat at the table.