Free LLM balancer combines multiple local inference machines with cloud fallback

Your machines run the models. The cloud only fills the gaps. Call every model through one OpenAI format — self-hosted, and yours to own.

LLMrPro is a self-hosted LLM router. It exposes one OpenAI-compatible API and serves each request from a pool of machines you own — desktops, workstations, servers — running local inference engines. When no local worker can serve a request, it transparently falls back to the cloud provider you configured for that tier. One endpoint, one request format, your hardware first, your keys, your data.

One OpenAI-compatible API for everything. Point any OpenAI client or SDK at the balancer URL — POST /{frontier,mini,nano}/v1/chat/completions with a standard Chat Completions body. Swap models and providers behind it without changing a line of caller code. Pools your own machines as inference workers. A lightweight desktop agent runs on each machine, connects outbound over a websocket (no inbound ports, no firewall changes), and pulls inference jobs. Each worker runs a full model on a local engine — LM Studio, Ollama, vLLM, llama.cpp, or the bundled MLX engine on macOS. Automatic per-tier cloud fallback. Three tiers — frontier , mini , nano — each with its own cloud backend (OpenAI, Anthropic, Google, Azure OpenAI). If the local pool can't serve a tier, the request falls back to that tier's cloud provider, with configurable retry/backoff. Single-tenant and self-hosted. One balancer per organization. Your keys, your data, your infrastructure — nothing leaves your control unless a request falls back to a cloud provider you configured. Live pool management. A built-in admin UI — device pairing, agent revocation, tier config, dispatch log, and health — backed by real-time pub/sub. Security built in. Per-source rate limiting, an egress guard (SSRF protection), provider keys encrypted at rest, signed agent tokens with expiry, and a hardened install/deploy path. How it works OpenAI-compatible ┌───────────────────────────────┐ client / app ───────────▶ │ LLMrPro │ (Bearer key) │ balancer (router + admin) │ │ tier: frontier / mini / nano │ └───────┬────────────────┬───────┘ websocket │ dispatch │ fallback (job pull) ▼ (local first) ▼ (per tier) ┌───────────────────────┐ ┌───────────────────┐ │ desktop agents │ │ cloud providers │ │ (your machines) │ │ OpenAI/Anthropic │ │ MLX / Ollama / vLLM │ │ / Google / Azure │ └───────────────────────┘ └───────────────────┘ A client sends a standard OpenAI Chat Completions request to a tier endpoint with the bearer key. The balancer picks an eligible paired agent and dispatches the job; the agent runs it on its local engine and streams the result back. If no agent can serve the tier, the balancer calls that tier's configured cloud provider and returns the result in the same OpenAI format. X-Balancer-* response headers report who served the request and why. Why LLMrPro is different Plenty of projects touch part of this; as of 2026 no single open-source project combines all of it.

The distinctive combination — federate a fleet of end-user desktops via a signed agent + tiered cloud fallback + single-tenant self-hosting, behind one OpenAI-compatible API — is what LLMrPro is for.

LLMrPro/ ├── balancer/ # the router: Meteor server + MongoDB + React admin UI ├── agent/ # the desktop worker: Electron + React app ├── docs/ # operator documentation (deploy, API, agent install, troubleshooting) ├── infra/ # provisioning + deploy scripts (Debian VM, reverse proxy, systemd) ├── DESIGN.md # full architecture, wire protocol, routing algorithm └── SECURITY.md # security model + how to report a vulnerability Quick start LLMrPro has two installable pieces: the balancer (you run one, server-side) and the agent (installed on each worker machine).

Requirements: Node.js 22+, MongoDB 6+ (a replica set is required for Meteor 3 change-stream reactivity), Meteor 3.4+, and — in production — a TLS-terminating reverse proxy (nginx / Caddy / Traefik).

git clone https://github.com/gysho/LLMrPro.git cd LLMrPro/balancer meteor npm install cp .env.production.example .env.production # then fill in the secrets (see below) npm run dev # inject-secrets + meteor run on :3500 Open the admin UI at http://localhost:3500/admin .

Production — provision a fresh host (Debian 12; installs Node, a MongoDB replica set, nginx, a systemd unit, and ufw):

sudo bash infra/azure/setup-vm.sh balancer.example.com Production — deploy a release (build → rsync → atomic symlink swap → restart):

cd balancer ./scripts/deploy.sh https://balancer.example.com deploy@your-host curl -i https://balancer.example.com/healthz # expect: 200 ok Full walkthrough: docs/operator-runbook.md .

Secrets are never committed. settings.json holds ${VAR} placeholders; real values live in .env.production and are injected into settings.runtime.json at boot by inject-secrets.sh . You provide: a platform bearer key (stored only as a SHA-256 hash), an agent-token HMAC signing key, the bootstrap admin password, and one cloud API key per tier. See docs/operator-runbook.md §2 .

Each worker machine runs the agent, which connects outbound to the balancer (no inbound ports) and forwards jobs to a local inference engine.

End users — one-line install (serves the right artifact for the OS/arch and verifies its SHA-256):

curl -fsSL https://balancer.example.com/install.sh | sh Inspect it first with curl -fsSL https://balancer.example.com/install.sh.txt . On first launch the agent opens a Pairing screen — paste the balancer URL and a pairing code (an admin generates these in the balancer UI), and the machine joins the pool.

cd agent npm install npm run build:linux # .AppImage / .deb / .rpm (on a Linux host) npm run build:mac # .dmg + bundled MLX engine (on a macOS host) The desktop agent is distributed unsigned by design — see SECURITY.md .

curl https://balancer.example.com/frontier/v1/chat/completions \ -H " Authorization: Bearer <platform-key> " \ -H " Content-Type: application/json " \ -d ' { "model": "gpt-5", "messages": [{ "role": "user", "content": "Hello" }] } ' Endpoints: POST /{frontier,mini,nano}/v1/chat/completions — the tier is the URL path. Body: standard OpenAI Chat Completions . Streaming, tools, and structured outputs pass through. Auth: Authorization: Bearer <platform-key> (required on every tier). The response is OpenAI-formatted whether served by a local agent or a cloud fallback. Full contract: docs/api-contract.md . Configuration All server config lives in balancer/settings.json (template: balancer/settings-example.json ). Key blocks:

Secrets in settings.json are ${VAR} placeholders resolved from the environment at boot — never commit real values.

Operator runbook — deploy, configure tiers, rotate keys, monitor, update, roll back Architecture · full design API contract Agent install: macOS · Linux Troubleshooting · FAQ Security The security model, the unsigned-agent distribution tradeoff, and how to report a vulnerability are in SECURITY.md . Provider keys are encrypted at rest, agent tokens are signed with expiry, and the balancer ships with rate limiting and SSRF egress protection.

LLMrPro is released under the MIT License . © 2026 Gysho.