tokio-prompt-orchestrator

概述

Tokio-prompt-orchestrator 是一个用 Rust 构建的生产级 LLM 编排框架。它提供高级功能，包括有界回压 DAG 处理、去重、断路器、速率限制、提示注入检测和提供商仲裁。该系统支持 MCP（模型上下文协议）集成，用于 Claude Desktop，同时提供 REST、WebSocket 和其他 API。它支持多种 LLM 后端，包括 Anthropic、OpenAI、llama.cpp 和 vLLM。

试试问 AI

装完之后，这里有 5 个你可以让 AI 做的事：

• create_ab_test

  Create or replace an A/B testing experiment for prompt variants

  试试:Create an A/B test comparing two different system prompts试试:Set up a new experiment to test temperature settings
• get_ab_test

  Retrieve details of a specific A/B test experiment

  试试:Show me the results for the customer support prompt test试试:Get the status of the pricing experiment
• create_session

  Create a new conversation session with optional system prompt

  试试:Start a new session for customer support inquiries试试:Create a session with 'You are a helpful assistant' system prompt
• append_message

  Add a message to an existing conversation session

  试试:Add a user message to session 123试试:Append an assistant response to our conversation
• get_session

  Retrieve metadata and messages from a session

  试试:Show me the full conversation from session 456试试:Get the current state of our support ticket session
• delete_session

  Remove a conversation session entirely

  试试:Clean up old session 789试试:Delete the completed sales consultation session
• render_template

  Render a prompt template with variable substitution

  试试:Render the 'summarize_email' template with the given text试试:Use the 'product_description' template with product variables
• register_template

  Store a new named prompt template for later use

  试试:Create a template called 'blog_intro' with {{topic}} variable试试:Save the 'system_prompt' template for reuse
• list_templates

  Show all available prompt templates

  试试:What templates are available?试试:List all registered prompt templates
• query_audit_log

  Search and filter the audit log of LLM requests

  试试:Show me recent requests to GPT-4 that took longer than 5 seconds试试:Find all failed requests from yesterday
• get_audit_stats

  Get aggregate statistics from the audit log

  试试:Show me cache hit rates for the last week试试:Get average latency by model
• export_audit_log

  Download all audit log entries as JSONL

  试试:Export all audit logs for the past month试试:Get the complete audit log for compliance review

说明：Inferred tool names from REST API endpoints and documentation. The README shows API endpoints but doesn't explicitly list them as MCP tools.

可对比工具

安装

tokio-prompt-orchestrator = "1.9"

{
  "mcpServers": {
    "tokio-prompt-orchestrator": {
      "command": "cargo",
      "args": ["run", "--release", "--package", "tokio-prompt-orchestrator", "--", "--mcp"]
    }
  }
}

FAQ

支持哪些 LLM 提供商？

该系统通过其可扩展架构支持 Anthropic、OpenAI、llama.cpp、vLLM 以及任何自定义后端。

去重功能如何工作？

它使用 SHA-256 哈希识别相同的飞行中请求，将它们合并为单个后端调用，并设置 30 秒 TTL 以防止内存泄漏。

Hacker News 讨论

开发者社区最近的相关讨论。

• Show HN: Tokio-prompt-orchestrator – LLM pipeline orchestration in Rust▲ 2
  帖子 by Shmungus · 2026-02-24
• 24 Simultaneous Claude Code agents on local hardware▲ 2💬 1
  帖子 by Shmungus · 2026-02-21

tokio-prompt-orchestrator 对比