Respan Rewrites Its LiteLLM-Based AI Gateway in Rust, With Every Feature a Plugin
Software / news
Respan Rewrites Its LiteLLM-Based AI Gateway in Rust, With Every Feature a Plugin
The Y Combinator company says a 100,000-line wrapper pushed it off LiteLLM. Its own benchmark table is vendor-supplied, and the launch thread left cost accounting unanswered.
Respan, a Y Combinator W24 company, has open-sourced an AI gateway written in Rust that replaces a heavily modified LiteLLM fork. In the Launch HN post, a Respan employee posting as Andydy25 called the project "pre-alpha", which means configuration and APIs may still change.
The repository is respanai/respan-gateway, dual-licensed MIT or Apache-2.0. It showed 288 stars, 30 forks, 415 commits on main, 13 open issues and 13 open pull requests when fetched on Oct. 25.
Why Respan left LiteLLM
The post says Respan built on LiteLLM from 2024 and that its wrapper code grew past 100,000 lines. A LiteLLM upgrade in January caused connection errors and a rollback, and the team has since backported upstream fixes by hand.
The replacement keeps a small core that only dispatches events. Providers such as OpenAI, Anthropic and Gemini are plugins. So are protocols (chat, realtime voice, MCP), modalities (images, video) and gateway features such as smart routing, fallbacks and PII redaction. AI Weekly compared the design to Nginx.
The README says every feature is built on "one trait with four hooks" and lists 16 providers plus any compatible endpoint. Plugins run in registration order. Each can pick a provider, filter a stream, make an extra call or refuse the request.
The benchmark table is Respan's own
The README carries a comparison against Bifrost, Portkey and a single-worker LiteLLM. All figures below are vendor-supplied, and the Launch HN post itself gives no benchmarks. The README names no independent party that ran them.
| Gateway | Added latency (p50) | Throughput (requests/s) | Memory |
|---|---|---|---|
| Respan | 0.07 ms | 56,734 | 31 MB |
| Bifrost | 0.16 ms | 18,182 | 488 MB |
| Portkey | 0.17 ms | 4,187 | 350 MB |
| LiteLLM (1 worker) | 4.34 ms | 268 | 418 MB |
- Respan57K req/s
- Bifrost18K req/s
- Portkey4187 req/s
- LiteLLM, 1 worker268 req/s
Source: respanai/respan-gateway README, accessed 2026-10-25
The LiteLLM row is a single worker, the weakest configuration of the four. A gateway sits in front of a model call that takes hundreds of milliseconds or more, so a 4 ms difference in added latency matters mostly to teams running very high request volumes.
What the launch thread pushed back on
Commenters raised four problems, and the maintainers answered only some.
- Plugin security: a commenter named verdverm argued that compiled crates still carry supply-chain risk and pointed to custom OAuth code in the MCP module. Plugins are compiled into the application, not downloaded as binaries, so a custom plugin means building your own copy.
- Install friction: commenters wrote that there was no Docker image at launch. Andydy25 said one was pulled before launch and that GitHub Releases provide binaries. The README lists
pip install respan-gateway,cargo install respan-gatewayand Docker, but its Releases section showed no entries when fetched. - Isolation: commenters suggested WASM or out-of-process plugins. Andydy25 said WASM was rejected because plugins often need database access.
- Cost tracking: several asked how usage is counted across providers that report it inconsistently, including retries. AI Weekly cites Groq's custom usage field as an example. The thread showed no Respan answer.
Andydy25 also said Claude generated much of the Rust but that they reviewed every line. The post gives no full name or title for the author.
The business behind the open core
Respan sells a hosted platform with observability, evals and unified billing on top of the gateway. The post says thousands of organisations use the LiteLLM-based version, a figure Respan supplied and The Terminal could not check.
For anyone running LiteLLM today, the open question is the one the thread left hanging. Whether a gateway can bill correctly across retries and odd usage fields decides whether it is a drop-in. Respan had not answered it in the thread, and it is the first thing to test before swapping a production gateway.
The pattern is familiar from this week's other launches: Laminar's flow-1 also shipped open code with the commercial product alongside it, and HumanLayer did the same with a homepage that lagged its repository.
Sources
More in Software
- 01Corgi and JetBrains' ThinkRail Both Want to Be Where You Run Coding Agents. One Is a Multiplexer, One an IDE.Corgi (MIT, 141 stars) wraps Claude Code and Codex in a terminal; ThinkRail (Apache-2.0, 1.2k stars) builds an IDE around the pi agent. What each README says and leaves out.
- 02Coolify 4.5.0 Encrypts Stored Secrets and Breaks Three Things Self-Hosters May Rely OnThe self-hosted PaaS shipped 4.5.0 on Oct. 23, two days after a security release for PHP2Shell. The changelog buries the API and Compose changes under about 100 items.
- 03Astral Adds uv check, and ty Will Read PEP 723 Script MetadataThe integration is experimental, and environment setup for scripts is opt-in because installing dependencies can run third-party code.
- 04TornadoVM 7.3 Drops JNI for Java's FFM API and Ships Three Releases in Nine DaysInfoQ's write-up of the Java GPU framework gives no benchmark figures, while the project's own pages supply speedups of 6x, 99x and 117 tokens a second.