tokio-prompt-orchestrator

Overview

Tokio-prompt-orchestrator is a production-grade LLM orchestration framework built in Rust. It provides advanced features including bounded-backpressure DAG processing, deduplication, circuit breakers, rate limiting, prompt injection detection, and provider arbitrage. The system exposes MCP (Model Context Protocol) integration for Claude Desktop, along with REST, WebSocket, and other APIs. It supports multiple LLM backends including Anthropic, OpenAI, llama.cpp, and vLLM.

Try asking AI

After installing, here are 5 things you can ask your AI assistant:

• create_ab_test

  Create or replace an A/B testing experiment for prompt variants

  Try:Create an A/B test comparing two different system promptsTry:Set up a new experiment to test temperature settings
• get_ab_test

  Retrieve details of a specific A/B test experiment

  Try:Show me the results for the customer support prompt testTry:Get the status of the pricing experiment
• create_session

  Create a new conversation session with optional system prompt

  Try:Start a new session for customer support inquiriesTry:Create a session with 'You are a helpful assistant' system prompt
• append_message

  Add a message to an existing conversation session

  Try:Add a user message to session 123Try:Append an assistant response to our conversation
• get_session

  Retrieve metadata and messages from a session

  Try:Show me the full conversation from session 456Try:Get the current state of our support ticket session
• delete_session

  Remove a conversation session entirely

  Try:Clean up old session 789Try:Delete the completed sales consultation session
• render_template

  Render a prompt template with variable substitution

  Try:Render the 'summarize_email' template with the given textTry:Use the 'product_description' template with product variables
• register_template

  Store a new named prompt template for later use

  Try:Create a template called 'blog_intro' with {{topic}} variableTry:Save the 'system_prompt' template for reuse
• list_templates

  Show all available prompt templates

  Try:What templates are available?Try:List all registered prompt templates
• query_audit_log

  Search and filter the audit log of LLM requests

  Try:Show me recent requests to GPT-4 that took longer than 5 secondsTry:Find all failed requests from yesterday
• get_audit_stats

  Get aggregate statistics from the audit log

  Try:Show me cache hit rates for the last weekTry:Get average latency by model
• export_audit_log

  Download all audit log entries as JSONL

  Try:Export all audit logs for the past monthTry:Get the complete audit log for compliance review

Note: Inferred tool names from REST API endpoints and documentation. The README shows API endpoints but doesn't explicitly list them as MCP tools.

Comparable tools

Installation

tokio-prompt-orchestrator = "1.9"

{
  "mcpServers": {
    "tokio-prompt-orchestrator": {
      "command": "cargo",
      "args": ["run", "--release", "--package", "tokio-prompt-orchestrator", "--", "--mcp"]
    }
  }
}

FAQ

What LLM providers are supported?

The system supports Anthropic, OpenAI, llama.cpp, vLLM, and any custom backend through its extensible architecture.

How does the deduplication feature work?

It uses SHA-256 hashing to identify identical in-flight requests, coalescing them to a single backend call with a 30-second TTL to prevent memory leaks.

On Hacker News

Recent discussion from the developer community.

• Show HN: Tokio-prompt-orchestrator – LLM pipeline orchestration in Rust▲ 2
  Story by Shmungus · 2026-02-24
• 24 Simultaneous Claude Code agents on local hardware▲ 2💬 1
  Story by Shmungus · 2026-02-21

Compare tokio-prompt-orchestrator with