LLM gateway

Your models stay put. Your apps still reach them.

A local model is cheap, private and fast — and completely unreachable from a browser. Gluer turns any inference server into an action your clients can call, without a public address, a tunnel or a rewrite.

Start for free Read the docs

Ollama

http://localhost:11434

Default choice on laptops and workstations. One worker exposes every pulled model.

llama.cpp server

http://localhost:8080/v1

OpenAI-compatible endpoint for GGUF models on CPU or Metal.

vLLM

http://gpu-box:8000/v1

High-throughput serving on your own GPUs, batched behind one action.

LM Studio

http://localhost:1234/v1

Local server mode for desktop experimentation.

Streaming

Tokens as they are generated

Publish to a channel from the worker, subscribe in the client. No SSE endpoint, no long-lived HTTP request, no proxy buffering surprises.

worker

import * as gluer from "gluer-nodejs";

gluer.register_plugin("llm", "stream", async ({ prompt, channel }) => {
  const res = await fetch("http://localhost:11434/api/generate", {
    method: "POST",
    body: JSON.stringify({ model: "llama3.1", prompt, stream: true }),
  });

  for await (const chunk of res.body) {
    const { response, done } = JSON.parse(chunk.toString());
    // Every subscriber of this channel receives the token immediately.
    gluer.sendToChannel(channel, "llm:token", { token: response, done });
  }

  return { started: true };
});

gluer.connect(process.env.GLUER_BACKEND_KEY);

client

import gluer from "gluer-js";

gluer.setup({ project: "4f2c9ab7e1d05c83" });
gluer.connect();

gluer.subscribe("chat-42", (msg) => append(msg.data.token));

await gluer.sendMsg("llm:stream", {
  prompt: "Explain our refund policy",
  channel: "chat-42",
});

Routing

Local first, hosted when you need it

Clients call llm:chat. Where inference happens is a worker decision — so you can move between models, providers and hardware without shipping a new frontend.

import * as gluer from "gluer-nodejs";

const LOCAL = process.env.OLLAMA_URL ?? "http://localhost:11434/v1";

async function complete(url, key, body) {
  const res = await fetch(`${url}/chat/completions`, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      ...(key ? { Authorization: `Bearer ${key}` } : {}),
    },
    body: JSON.stringify(body),
  });
  if (!res.ok) throw new Error(`upstream ${res.status}`);
  return res.json();
}

// Same action name, local first, hosted as a safety net.
gluer.register_plugin("llm", "chat", async ({ messages }) => {
  try {
    return await complete(LOCAL, null, { model: "llama3.1", messages });
  } catch {
    return await complete(
      "https://api.openai.com/v1",
      process.env.OPENAI_API_KEY,
      { model: "gpt-4o-mini", messages },
    );
  }
});

gluer.connect(process.env.GLUER_BACKEND_KEY);

FAQ

Local models on Gluer

Do prompts leave my network when I use Gluer with a local model?

The prompt travels from the client to the Gluer mesh and then to your worker, which calls the model on localhost or inside your network. The model host itself is never exposed, and you choose what the worker sends back. For fully offline setups, run the single-binary mesh locally so nothing leaves the machine at all.

How do I stream tokens to the browser?

Publish each token to a channel with sendToChannel and subscribe to that channel in the client. The tokens arrive over the WebSocket the client already has open, with no SSE endpoint to host.

Can I mix local and hosted models?

Yes. Because clients call an action name rather than a provider URL, the worker decides where inference happens — local first with a hosted fallback, per-tenant routing, or A/B testing between models — without any client change.

Can an AI agent use Gluer as its tool transport?

Yes. Register the tools an agent may call as plugin actions and give the agent runtime a project key. It reaches your internal systems through one audited channel instead of receiving network access or long-lived credentials.

Connect your first two systems today

Create a project, drop the SDK into your app, and point a worker at it. Free while you build, no credit card, no infrastructure to provision.

Start for free Read the docs