Monitoring and Protecting AI Agent API Costs with Stoke

Practical Summary of Stoke

Stoke is a self-hosted, 5.5 MB Rust binary that acts as a gateway for AI agent traffic. It is designed to enforce strict cost controls and routing policies before any request reaches an external provider. By sitting between your agents and your inference providers, Stoke provides a circuit breaker that can reject loops, clamp fan-out, and hold concurrent stream reservations to prevent budget overruns. It is intended for operators who need to manage local Ollama nodes alongside approved cloud providers, ensuring that sensitive routes remain off unapproved providers and that runaway agents do not trigger unexpected financial costs.

Prerequisites and Installation

To use Stoke, you must have an environment capable of running the prebuilt binary. The tool is designed to be installed in approximately 60 seconds. You can retrieve the latest stable version using the following command:

curl -sSf https://stokegate.com/install | sh

Stoke works with any OpenAI-compatible agent by pointing the base_url to localhost:8787/v1. For Claude Code, you must configure the ANTHROPIC_BASE_URL to point to the gateway. The gateway is self-hosted and does not require a hosted control plane or external account, making it suitable for infrastructure you control directly.

Managing Runaway Costs and Loops

Stoke prevents runaway agent costs by implementing a circuit breaker before the provider call is made. It detects loops and applies rate limits to refuse repeated traffic. To manage fan-out, Stoke uses pre-dispatch ceilings that limit the number of billed provider calls a single request can generate. Furthermore, for streamed requests, Stoke holds concurrent stream reservations to ensure that a burst of activity does not overshoot the configured budget cap for a specific key. These mechanisms ensure that the gateway decides whether to proceed before any money is spent on inference.

Routing and Capacity Management

Stoke allows you to route traffic based on model support, warm state, health, load, and policy. It discovers exact Ollama model IDs, inventory, and health through the /v1/nodes endpoint. The gateway filters eligible capacity based on your route policy and prefers healthy, warm nodes with the lowest live load. You can connect multiple machines by adding a second Stoke gateway as an authenticated remote provider, allowing both machines to sit behind the same client URL. This enables a unified capacity layer where you can keep Ollama private on each machine while centralizing policy enforcement.

Egress Policy and Credential Hygiene

To maintain security, Stoke supports per-route allowlists. By setting allowed_tiers to specific values like ["local", "remote"], you can deny cloud fallback before any provider is contacted. If no approved provider is available, Stoke fails closed with a 403 error rather than escalating to unapproved cloud capacity. Additionally, the gateway performs credential hygiene by redacting known credentials before dispatching requests to a provider. This ensures that sensitive information is not leaked to external services, providing an extra layer of protection for your agent infrastructure.

Monitoring via the Control Room

Every running Stoke gateway includes an embedded control room accessible at /ui. This panel provides a real-time view of decisions, budget state, routing, and node status. It allows operators to watch enforcement in action, such as seeing when a request is allowed, blocked due to a budget cap, or held due to concurrent stream reservations. The panel is authenticated and provides an inspectable decision feed, which helps in auditing why specific requests were routed to particular nodes or rejected by the gateway's policy engine.

Limitations and Operational Scope

Stoke is not a hosted service and does not provide its own internal retry policy; it only propagates valid, unambiguous Retry-After and x-should-retry hints from upstream providers. Regarding caching, only eligible non-streaming named routes can use the exact cache policy, which is bounded and opt-in. The optional local Headroom feature is default-off and only accepts a smaller, validated, lossless JSON-whitespace transformation; it does not summarize or rewrite reasoning. If a worker is unavailable or rejects a transformation, Stoke retains the original outputs to ensure reliability.

Choosing When to Use Stoke

Stoke is ideal for organizations that need to enforce strict cost governance over AI agents while maintaining the flexibility to use both local Ollama nodes and cloud providers. It is particularly useful for teams concerned about runaway agent loops, fan-out costs, or the accidental exposure of credentials to third-party providers. By providing a single endpoint for diverse capacity, it simplifies topology management. For detailed configuration, routing design, and security documentation, operators should consult the official Stoke documentation and the project's README on GitHub.