A deliberately small gateway for LLM traffic. Forward requests, attach attribution, meter tokens, and apply token limits.
Start with a local configuration file. Add Git-based configuration distribution if you need several instances. Billing tags are supported on selected provider paths; invoice reconciliation is outside the current scope.
chosen
One data plane, one control plane. A team can run the data plane alone, from a file on disk, and never install the second.
bounded
The gateway keeps runtime state in memory and can read configuration from a local file. Its request path makes no database calls.
measured
Measured on the WASM path in a local benchmark. Per-event streaming hooks are off by default. This is an extension measurement, not overall request latency.
measured
Recorded from live traffic and replayed to compare token estimates with provider-reported usage. A small sample, not a guarantee for every model.
01 · Shape
Start with a gateway and a configuration file. The optional control plane distributes configuration from Git when you have several instances to manage.
01
The request proxy
runs alone
Forwards requests to model providers, resolves caller attribution, records token usage, and applies token limits. Supported adapters read streaming responses as they arrive. Run it with a local configuration file.
02
Optional configuration management
Renders configuration from Git, distributes snapshots to gateway instances, and tracks which version each instance is running. This code is available when several gateways need shared configuration.
The standalone gateway needs no control plane. With fleet management enabled, gateways retain their current configuration if that connection drops. Shared token limits have separate partition behavior, described in the deployment notes.
02 · Path
For supported streaming responses, the gateway reads usage while forwarding bytes to the caller. Token estimates help enforce limits before a provider sends its final usage count.
Providers use different streaming formats, including server-sent events and binary event frames. The configured adapter parses the supported format.
holds: Support is checked per provider and request shape
Adapters expose content and usage through a shared event model. Replay tests check that splitting the same response into different network chunks produces the same events.
holds: Chunk boundaries should not change the parsed events
Token estimates support live enforcement. A provider's final usage count replaces the estimate when available. Missing usage and interrupted responses have documented limits.
holds: Token usage is not a billed dollar amount
An exhausted token allowance can end a stream with the operator's configured terminal event. The local estimate and any distributed allowance affect when that happens.
holds: Limits are measured in tokens
03 · Measured
A claim with no stated failure mode has usually not been tested hard enough to have found one. Each row here carries the method that produced it and the part that still does not work, in the same weight of type.
claim
how it was checked
Seventeen recorded transcripts replayed against the provider's own reported usage.
what is still wrong
All but one stream. The misses are structural rather than random: tool-call scaffolding and very short responses are where the estimate is worst.
claim
how it was checked
The control plane allocates token shares to gateways. Tests exercise enforcement and measure overshoot when a gateway loses its connection.
what is still wrong
Token usage can exceed the configured allowance during a network partition. The documented bound depends on the configuration. These are token limits, not dollar budgets.
claim
how it was checked
Extensions run sandboxed with no ambient access, on a fuel budget, interruptible mid-execution. They must be signed.
what is still wrong
A local benchmark measured about eleven microseconds per call. Per-event hooks are off by default. Sandboxing does not establish that every extension or workload is safe.
claim
how it was checked
Git is the source of truth, and every rollout is a rendered snapshot tied to the commit that produced it.
what is still wrong
Temporary overrides can differ from Git until they expire. This describes configuration history, not a durable record of every request.
04 · The project
The focus is forwarding model requests, identifying who made them, recording token usage, and applying token limits. Provider billing tags are included where the API supports them. Invoice reconciliation and dollar allocation sit outside this project.
This is an experimental project by Anton Braverman, with no stable release or production support commitment yet. Bug reports, small fixes, and clearer examples are welcome. The Gateway Baseline is a related comparison of attribution features.