inferL2 inference

The model, on the same machine.

B3OS serves its own small instruct model and Kokoro text-to-speech on the node's GPU, behind OpenAI-compatible paths. Point the OpenAI SDK at /v1/infer and your agent's model calls are served on the machine that runs its tools.

Get API key Read the docs

API: /v1/infer
CLI: b3os infer
MCP tools: infer_*

[01/03] What it does

Chat completions

Streaming, tool calling and JSON-schema output on the OpenAI wire format.

POST /v1/infer/chat/completions

A model table

Each model lists its context length, quantization, tool and JSON-schema support, image input and your per-minute token limit.

GET /v1/infer/models

Speech with word timings

Text to speech with per-word start and end times, the same timings render uses for captions.

POST /v1/infer/audio/speech

No silent fallback

An unknown model returns model_not_available. Opt into route: gateway to send the call to the B3IQ gateway instead; Server-Timing reports which route served it.

route: "gateway"

Which model answers

On the node
Models in GET /v1/infer/models answer on the node's GPU.
Unknown model
Returns model_not_available. Nothing is rerouted on your behalf.
route: gateway
Opt in to send the call to the B3IQ gateway instead.
Server-Timing
infer_route tells you which path served each call.

1MODEL=$(curl -sS https://beta-api.b3os.org/v1/infer/models \
2  -H "Authorization: Bearer $B3OS_API_KEY" | jq -r '.data[0].id')
3
4curl -sS https://beta-api.b3os.org/v1/infer/chat/completions \
5  -H "Authorization: Bearer $B3OS_API_KEY" \
6  -H "Content-Type: application/json" \
7  -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Say hi in five words.\"}]}"

MODEL=$(curl -sS https://beta-api.b3os.org/v1/infer/models \
  -H "Authorization: Bearer $B3OS_API_KEY" | jq -r '.data[0].id')

curl -sS https://beta-api.b3os.org/v1/infer/chat/completions \
  -H "Authorization: Bearer $B3OS_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"Say hi in five words.\"}]}"

1import OpenAI from "openai";
2
3const client = new OpenAI({
4  baseURL: "https://beta-api.b3os.org/v1/infer",
5  apiKey: process.env.B3OS_API_KEY,
6});
7
8const models = await client.models.list();
9const reply = await client.chat.completions.create({
10  model: models.data[0].id,
11  messages: [{ role: "user", content: "Say hi in five words." }],
12});
13console.log(reply.choices[0].message.content);

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://beta-api.b3os.org/v1/infer",
  apiKey: process.env.B3OS_API_KEY,
});

const models = await client.models.list();
const reply = await client.chat.completions.create({
  model: models.data[0].id,
  messages: [{ role: "user", content: "Say hi in five words." }],
});
console.log(reply.choices[0].message.content);

1import os
2
3from openai import OpenAI
4
5client = OpenAI(
6    base_url="https://beta-api.b3os.org/v1/infer",
7    api_key=os.environ["B3OS_API_KEY"],
8)
9
10model = client.models.list().data[0].id
11reply = client.chat.completions.create(
12    model=model,
13    messages=[{"role": "user", "content": "Say hi in five words."}],
14)
15print(reply.choices[0].message.content)

import os

from openai import OpenAI

client = OpenAI(
    base_url="https://beta-api.b3os.org/v1/infer",
    api_key=os.environ["B3OS_API_KEY"],
)

model = client.models.list().data[0].id
reply = client.chat.completions.create(
    model=model,
    messages=[{"role": "user", "content": "Say hi in five words."}],
)
print(reply.choices[0].message.content)

Tokens are metered and free during the beta, under per-model token-per-minute limits listed in GET /v1/infer/models.
Frontier models elsewhere still work with every B3OS tool; only their model turns leave the machine.

[02/03] Measured

What we measure for infer

Values appear here only from published benchmark runs, each with its raw data. A row that says measuring has no published run yet.

Time to first token

Streamed chat completion on the co-located model, with B3OS idle and at budgeted load.

Measuringp50, client side

Time per output token

The same stream, per output token.

Measuringp50, client side

How we measure, and what others publish

[03/03] Beta limits

Free during the beta, with caps

Caps per organization

Token-per-minute limits are set per model and listed by GET /v1/infer/models.

Metered units

Input tokens: infer.input_tokens
Output tokens: infer.output_tokens
Speech: infer.audio_chars

Pricing details