Respan Rewrites Its LiteLLM-Based AI Gateway in Rust, With Every Feature a Plugin
Software / news
Respan Rewrites Its LiteLLM-Based AI Gateway in Rust, With Every Feature a Plugin
The Y Combinator company says a 100,000-line wrapper pushed it off LiteLLM. Its own benchmark table is vendor-supplied, and the launch thread left cost accounting unanswered.
Respan, a Y Combinator W24 company, has open-sourced an AI gateway written in Rust that replaces a heavily modified LiteLLM fork. In the Launch HN post, a Respan employee posting as Andydy25 called the project "pre-alpha", which means configuration and APIs may still change.
The repository is respanai/respan-gateway, dual-licensed MIT or Apache-2.0. It showed 288 stars, 30 forks, 415 commits on main, 13 open issues and 13 open pull requests when fetched on Oct. 25.
Why Respan left LiteLLM
The post says Respan built on LiteLLM from 2024 and that its wrapper code grew past 100,000 lines. A LiteLLM upgrade in January caused connection errors and a rollback, and the team has since backported upstream fixes by hand.
The replacement keeps a small core that only dispatches events. Providers such as OpenAI, Anthropic and Gemini are plugins. So are protocols (chat, realtime voice, MCP), modalities (images, video) and gateway features such as smart routing, fallbacks and PII redaction. AI Weekly compared the design to Nginx.
The README says every feature is built on "one trait with four hooks" and lists 16 providers plus any compatible endpoint. Plugins run in registration order. Each can pick a provider, filter a stream, make an extra call or refuse the request.
The benchmark table is Respan's own
The README carries a comparison against Bifrost, Portkey and a single-worker LiteLLM. All figures below are vendor-supplied, and the Launch HN post itself gives no benchmarks. The README names no independent party that ran them.
| Gateway | Added latency (p50) | Throughput (requests/s) | Memory |
|---|---|---|---|
| Respan | 0.07 ms | 56,734 | 31 MB |
| Bifrost | 0.16 ms | 18,182 | 488 MB |
| Portkey | 0.17 ms | 4,187 | 350 MB |
| LiteLLM (1 worker) | 4.34 ms | 268 | 418 MB |
- Respan57K req/s
- Bifrost18K req/s
- Portkey4187 req/s
- LiteLLM, 1 worker268 req/s
Source: respanai/respan-gateway README, accessed 2026-10-25
The LiteLLM row is a single worker, the weakest configuration of the four. A gateway sits in front of a model call that takes hundreds of milliseconds or more, so a 4 ms difference in added latency matters mostly to teams running very high request volumes.
What the launch thread pushed back on
Commenters raised four problems, and the maintainers answered only some.
- Plugin security: a commenter named verdverm argued that compiled crates still carry supply-chain risk and pointed to custom OAuth code in the MCP module. Plugins are compiled into the application, not downloaded as binaries, so a custom plugin means building your own copy.
- Install friction: commenters wrote that there was no Docker image at launch. Andydy25 said one was pulled before launch and that GitHub Releases provide binaries. The README lists
pip install respan-gateway,cargo install respan-gatewayand Docker, but its Releases section showed no entries when fetched. - Isolation: commenters suggested WASM or out-of-process plugins. Andydy25 said WASM was rejected because plugins often need database access.
- Cost tracking: several asked how usage is counted across providers that report it inconsistently, including retries. AI Weekly cites Groq's custom usage field as an example. The thread showed no Respan answer.
Andydy25 also said Claude generated much of the Rust but that they reviewed every line. The post gives no full name or title for the author.
The business behind the open core
Respan sells a hosted platform with observability, evals and unified billing on top of the gateway. The post says thousands of organisations use the LiteLLM-based version, a figure Respan supplied and The Terminal could not check.
For anyone running LiteLLM today, the open question is the one the thread left hanging. Whether a gateway can bill correctly across retries and odd usage fields decides whether it is a drop-in. Respan had not answered it in the thread, and it is the first thing to test before swapping a production gateway.
The pattern is familiar from this week's other launches: Laminar's flow-1 also shipped open code with the commercial product alongside it, and HumanLayer did the same with a homepage that lagged its repository.
Sources
More in Software
- 01OmniDiff, a Rust Diff Tool, Reports 60 Percent Exact Matches Against a 90 Percent GoalThe README publishes its own scorecard on 1,419 hand-labelled fixtures. It also says VS Code cannot yet use it as the built-in diff view.
- 02Enlightenment 0.28.0 Adds NetworkManager and PipeWire Support 658 Days After 0.27The window manager's first release since January 2025 needs EFL 1.29.0 and adds NetBSD support, a clipboard history and battery charge limits.
- 03Lawson's 12-Framework Test Puts the AI Cost of One Pokémon App Between $2.67 and $4.97Nolan Lawson had Claude Opus 5.5 build the same app in each stack. Angular cost 1.86 times as much as Hono+htmx, and plain Vite beat eight frameworks on price.
- 04Python 3.15 Ships a Built-In sentinel After Five Years of PEP 661The new type replaces the object() idiom with a one-line constructor that survives copying, pickling and type checking, but a Python 3.15 floor is the price of using it.