# Private Model Gateway

**AI agents: fetch https://llm.hisavage.com/docs.md** (plain markdown, no auth, no JS). Index: https://llm.hisavage.com/llms.txt

This is a private model, served through an **OpenAI-compatible** API. It has three tiers: `default` (balanced, general purpose), `smart` (strongest reasoning) and `fast` (quickest, lightest). There are also pinned versions of each tier. Any client or SDK built for the OpenAI Chat Completions API works if you change the base URL and the API key.

- **Base URL:** `https://llm.hisavage.com/v1`
- **Auth header:** `Authorization: Bearer <YOUR_API_KEY>` (keys are issued by the gateway owner; never put a key in code you share)
- **Raw markdown of this page (for agents):** `https://llm.hisavage.com/docs.md`

## Instructions for AI agents

1. Use the OpenAI Chat Completions format against the base URL above. Read the key from an env var such as `GATEWAY_API_KEY`.
2. Pick a `model` from the table below. If you are asked for a **Claude model**, use its mapped neutral name from the **Claude equivalent** column (for example "Claude Sonnet 4.5" → `default-4.5`, "Claude Opus" → `smart`). If you are asked for an **OpenAI model**, use its mapped neutral name from the **OpenAI equivalent** column (for example "gpt-4o-mini" → `fast`, "gpt-5" → `default`, "o3" → `smart`). If unsure, use `default`.
3. If you're unsure, use `default`. Common OpenAI- and Claude-style model names are routed to the matching tier automatically (see [Model name routing](#model-name-routing)). Anything unrecognised falls back to `default`.
4. You can send text and images (see [Vision / images](#vision-images)), OpenAI-style `tools` (see [Tool calling](#tool-calling)) and `response_format` for JSON (see [JSON / structured output](#json-structured-output)). Don't send audio or files.
5. On a `503` or `429`, retry with backoff (see Errors).

## Endpoints

| Method | Path | Purpose |
|---|---|---|
| GET | `/v1/models` | List available model names |
| POST | `/v1/chat/completions` | Chat completion (set `"stream": true` for SSE streaming) |
| GET | `/health` | Liveness check, no auth needed |

Supported request fields: `model`, `messages` (roles `system`, `developer`, `user`, `assistant` (including `tool_calls`), `tool`; string content, or arrays of `text` and `image_url` parts), `stream`, `stream_options.include_usage`, `tools`, `tool_choice`, `parallel_tool_calls`, `response_format`. Other fields such as `temperature` and `max_tokens` are accepted but ignored.

## Models

Floating names (`default`, `smart`, `fast`) always point to the newest model of that tier. Versioned names are pinned.

| Model | Tier | Good for | Claude equivalent | OpenAI equivalent |
|---|---|---|---|---|
| `smart` | smart | Hardest reasoning, complex coding, long careful analysis (slowest) | Claude Opus 5.5 (latest Opus) | ≈ `gpt-6-astra` (flagship) |
| `smart-5.5` | smart | Same as `smart`, pinned | Claude Opus 5.5 | ≈ `gpt-6-astra` |
| `smart-5` | smart | Pinned older top-tier model | Claude Opus 5 | ≈ GPT-5.x flagship / `gpt-5-pro` class |
| `smart-4.7` | smart | Pinned older top-tier model | Claude Opus 4.7 | ≈ GPT-5 (high reasoning) / o3 class |
| `smart-4.6` | smart | Pinned older top-tier model | Claude Opus 4.6 | ≈ GPT-5 (high reasoning) / o3 class |
| `smart-4.5` | smart | Pinned older top-tier model | Claude Opus 4.5 | ≈ GPT-5 / o3 class |
| `default` | default | General-purpose: writing, coding, chat, summarising. Best balance | Claude Sonnet 5.5 (latest Sonnet) | ≈ `gpt-6.1-sol` (main balanced model) |
| `default-5.5` | default | Same as `default`, pinned | Claude Sonnet 5.5 | ≈ `gpt-6.1-sol` |
| `default-5` | default | Pinned older general model | Claude Sonnet 5 | ≈ `gpt-5.4` class |
| `default-4.6` | default | Pinned older general model | Claude Sonnet 4.6 | ≈ GPT-5 / `gpt-4.1` class |
| `default-4.5` | default | Pinned older general model | Claude Sonnet 4.5 | ≈ GPT-5 / `gpt-4.1` class |
| `fast` | fast | Quick, light tasks: classification, extraction, short replies | Claude Haiku 5.5 (latest Haiku) | ≈ `gpt-6-luna` (efficient tier) |
| `fast-5.5` | fast | Same as `fast`, pinned | Claude Haiku 5.5 | ≈ `gpt-6-luna` |
| `fast-4.5` | fast | Pinned older fast model | Claude Haiku 4.5 | ≈ `gpt-5.4-mini` / `gpt-5-mini` class |

The equivalents are only a rough guide, so any agent or developer can pick a tier quickly. The OpenAI column is an approximate capability match, not a benchmark claim.

Mapping rule: when asked for a Claude model, use its neutral name. Opus → `smart*`, Sonnet → `default*`, Haiku → `fast*`. Use the version suffix if it's listed, otherwise use the floating name. When asked for an OpenAI model, use the tier from the OpenAI column. The API always reports the neutral name in the `model` field.

### Model name routing

You can also send a third-party model name directly. The gateway maps it to a tier, and the response's `model` field shows the neutral name it used. The first matching rule wins:

| If `model` looks like… | Routed to |
|---|---|
| A neutral name (`default`, `smart-4.7`, …), any case | itself |
| `claude-opus-*`, `claude-sonnet-*`, `claude-haiku-*` | `smart` / `default` / `fast`, or the pinned version if it exists (e.g. `claude-opus-4-5` → `smart-4.5`, `claude-sonnet-4-5-20250929` → `default-4.5`) |
| Contains `mini`, `nano`, `luna`, `flash`, `lite`, `small`, `instant`, or is `gpt-3.5*` (e.g. `gpt-4o-mini`, `gpt-5.4-nano`, `o4-mini`, `gpt-6-luna`) | `fast` |
| o-series (`o1`, `o3`, `o4`…), `*-pro`, `astra`, or names with `reason`, `thinking`, `ultra`, `max`, `deep` (e.g. `gpt-5-pro`, `gpt-6-astra`) | `smart` |
| Other `gpt-*` / `chatgpt-*` / `chat-latest` / `sol` (e.g. `gpt-4o`, `gpt-4.1`, `gpt-5`, `gpt-6.1-sol`) | `default` |
| Anything else | `default` |

## Examples

### curl

```bash
curl https://llm.hisavage.com/v1/chat/completions \
  -H "Authorization: Bearer $GATEWAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"default","messages":[{"role":"system","content":"Be concise."},{"role":"user","content":"Hello!"}]}'
```

Streaming (Server-Sent Events, ends with `data: [DONE]`):

```bash
curl -N https://llm.hisavage.com/v1/chat/completions \
  -H "Authorization: Bearer $GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"fast","stream":true,"messages":[{"role":"user","content":"Count to 5"}]}'
```

### Python (`pip install openai`)

```python
import os
from openai import OpenAI

client = OpenAI(base_url="https://llm.hisavage.com/v1",
                api_key=os.environ["GATEWAY_API_KEY"], timeout=330, max_retries=3)

r = client.chat.completions.create(model="default",
        messages=[{"role": "user", "content": "Write a haiku about autumn."}])
print(r.choices[0].message.content)

for chunk in client.chat.completions.create(model="fast", stream=True,
        messages=[{"role": "user", "content": "Count to 5"}]):
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
```

### Node (`npm i openai`)

```js
import OpenAI from "openai";
const client = new OpenAI({ baseURL: "https://llm.hisavage.com/v1",
  apiKey: process.env.GATEWAY_API_KEY, timeout: 330_000, maxRetries: 3 });

const r = await client.chat.completions.create({ model: "default",
  messages: [{ role: "user", content: "Hello!" }] });
console.log(r.choices[0].message.content);

const stream = await client.chat.completions.create({ model: "fast", stream: true,
  messages: [{ role: "user", content: "Count to 5" }] });
for await (const c of stream) process.stdout.write(c.choices[0]?.delta?.content ?? "");
```

## Vision / images

**Every model** (`default`, `smart`, `fast` and all pinned versions) accepts images, using the standard OpenAI chat format. Put a content array in a `user` message with `text` parts and `image_url` parts. `image_url.url` can be:

- a **data URL**: `data:image/png;base64,iVBORw0...`, or
- a public **https:// (or http://) URL**, which the gateway downloads server-side.

`detail` is accepted and ignored. Images are numbered in order across the conversation, so you can refer to them as "the first image", "image 2", and so on.

**Image limits**

- Formats: PNG, JPEG, GIF, WebP (detected from the file bytes).
- Up to **10 MB per image** and **10 images per request**. The whole request body can be up to 25 MB (base64 adds about 33%).
- Remote URLs must be publicly reachable. Private, loopback, link-local and internal addresses are refused, and so is this server itself. Downloads time out after 15 s and follow at most 3 redirects. URLs with credentials aren't allowed.
- Image errors return `400` (or `413` if the image is too large) with `code: "invalid_image"` and a clear message.

### curl (data URL)

```bash
IMG=$(base64 -w0 photo.png)   # macOS: base64 -i photo.png
curl https://llm.hisavage.com/v1/chat/completions \
  -H "Authorization: Bearer $GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"default","messages":[{"role":"user","content":[
        {"type":"text","text":"What does this image show?"},
        {"type":"image_url","image_url":{"url":"data:image/png;base64,'"$IMG"'"}}]}]}'
```

### curl (remote URL)

```bash
curl https://llm.hisavage.com/v1/chat/completions \
  -H "Authorization: Bearer $GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"fast","messages":[{"role":"user","content":[
        {"type":"text","text":"Read any text in this image."},
        {"type":"image_url","image_url":{"url":"https://example.com/receipt.jpg"}}]}]}'
```

### Python

```python
import base64, os
from openai import OpenAI
client = OpenAI(base_url="https://llm.hisavage.com/v1", api_key=os.environ["GATEWAY_API_KEY"], timeout=330)

b64 = base64.b64encode(open("screenshot.png", "rb").read()).decode()
r = client.chat.completions.create(model="default", messages=[{"role": "user", "content": [
    {"type": "text", "text": "Transcribe the text in this screenshot."},
    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
    {"type": "image_url", "image_url": {"url": "https://example.com/second.jpg"}},
]}])
print(r.choices[0].message.content)
```

### Node

```js
import fs from "node:fs";
import OpenAI from "openai";
const client = new OpenAI({ baseURL: "https://llm.hisavage.com/v1", apiKey: process.env.GATEWAY_API_KEY, timeout: 330_000 });

const b64 = fs.readFileSync("chart.png").toString("base64");
const stream = await client.chat.completions.create({ model: "smart", stream: true, messages: [{ role: "user", content: [
  { type: "text", text: "Summarise this chart." },
  { type: "image_url", image_url: { url: `data:image/png;base64,${b64}` } },
] }] });
for await (const c of stream) process.stdout.write(c.choices[0]?.delta?.content ?? "");
```

Streaming and non-streaming both work with images.

## Tool calling

Standard OpenAI **function calling** works on every model: `tools` (type `function` with a JSON Schema `parameters`), `tool_choice` (`"auto"` (default), `"none"`, `"required"`, or `{"type":"function","function":{"name":"..."}}`) and `parallel_tool_calls` (default `true`). Responses use the normal OpenAI shape: `message.tool_calls[]` with `id` (`call_...`), `type: "function"`, `function.name` and `function.arguments` as a **JSON string**, plus `finish_reason: "tool_calls"`. Send results back as `role: "tool"` messages with the matching `tool_call_id`. The gateway never runs your functions; your code does.

Tested with the official `openai` Python SDK (including `chat.completions.stream` and `parse`), `langchain-openai` (`bind_tools`) and `openai-agents` (`Agent` + `function_tool` via `OpenAIChatCompletionsModel`).

### Python: full round trip

```python
import json, os
from openai import OpenAI
client = OpenAI(base_url="https://llm.hisavage.com/v1", api_key=os.environ["GATEWAY_API_KEY"], timeout=330)

tools = [{"type": "function", "function": {
    "name": "get_weather", "description": "Get current weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"},
                   "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}},
                   "required": ["city"], "additionalProperties": False}}}]

messages = [{"role": "user", "content": "What's the weather in Paris and Tokyo?"}]
r = client.chat.completions.create(model="default", messages=messages, tools=tools)
msg = r.choices[0].message            # finish_reason == "tool_calls", two parallel calls
messages.append(msg.model_dump(exclude_none=True))
for call in msg.tool_calls:
    args = json.loads(call.function.arguments)
    messages.append({"role": "tool", "tool_call_id": call.id,
                     "content": json.dumps({"city": args["city"], "temp_c": 17})})
final = client.chat.completions.create(model="default", messages=messages, tools=tools)
print(final.choices[0].message.content)
```

### Node

```js
import OpenAI from "openai";
const client = new OpenAI({ baseURL: "https://llm.hisavage.com/v1", apiKey: process.env.GATEWAY_API_KEY, timeout: 330_000 });
const tools = [{ type: "function", function: { name: "get_weather", description: "Get weather for a city",
  parameters: { type: "object", properties: { city: { type: "string" } }, required: ["city"] } } }];
const r = await client.chat.completions.create({ model: "fast", tools, tool_choice: "required",
  messages: [{ role: "user", content: "Weather in Lagos?" }] });
console.log(r.choices[0].message.tool_calls);  // [{ id: "call_...", function: { name: "get_weather", arguments: "{\"city\":\"Lagos\"}" } }]
```

Streaming (`stream: true`) also works. Tool calls arrive as standard `delta.tool_calls` chunks (`index`, `id`, `function.name`, then `function.arguments` fragments), which the SDKs assemble for you.

## JSON / structured output

`response_format` works on every model:

- `{"type": "json_object"}`: the reply `content` is a single JSON object.
- `{"type": "json_schema", "json_schema": {"name": "...", "schema": {...}, "strict": true}}`: the reply is JSON that matches your schema. It's validated, and code fences or stray prose are stripped.

```python
from pydantic import BaseModel
class City(BaseModel):
    name: str
    country: str
    landmarks: list[str]

r = client.chat.completions.parse(model="default", response_format=City,
        messages=[{"role": "user", "content": "Describe Paris."}])
print(r.choices[0].message.parsed)      # City(name='Paris', country='France', landmarks=[...])
```

```bash
curl https://llm.hisavage.com/v1/chat/completions -H "Authorization: Bearer $GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"fast","response_format":{"type":"json_object"},"messages":[{"role":"user","content":"Give a JSON object with keys name and age."}]}'
```

### Caveats (tool calling and JSON)

- Both are **emulated by the gateway** through prompting, not native model features. They're reliable in testing, but:
- **Slightly slower:** the reply is generated in full before it's returned. With `stream: true` the chunks arrive all at once at the end rather than token by token. If the output is invalid, the gateway retries once, which adds a few seconds.
- Arguments and JSON are validated against a **practical subset of JSON Schema**: `type`, `properties`, `required`, `additionalProperties`, `items`, `enum`, `const`, `anyOf`/`oneOf`, `minimum`/`maximum`, and local `$ref`. Keep schemas simple.
- If output is still invalid after the retry, you get `502 invalid_model_output`. The exception is non-strict `json_schema`, where you get the best valid JSON produced. With `tool_choice: "auto"` you get the plain-text reply as `content`.
- Tool calls and tool results in the history are shown to the model as text. Very large tool results count against the context like any other text.

## Limits

- **Text and images.** Images are supported on every model (see [Vision / images](#vision-images)). Audio and file parts are dropped. Tool calling and JSON output are supported but emulated (see their sections).
- **Concurrency:** 2 requests run at a time and up to 20 more wait in a queue. Beyond that you get `429`.
- **Timeout:** 5 minutes per request (returns `504`). Set your client timeout a little above that, for example 330 s.
- **Request body:** 25 MB maximum (including base64 images).
- Each request is stateless, so send the full conversation in `messages` every time.
- `usage` token counts are approximate.

## Errors and retries

Errors use the OpenAI shape: `{"error":{"message":"...","type":"...","code":"..."}}`.

| Status | code / type | Meaning | What to do |
|---|---|---|---|
| 400 | `invalid_request_error` / `invalid_image` | Bad JSON, missing `messages`, or a bad/blocked/unsupported image | Fix the request (read `message`) |
| 401 | `invalid_api_key` | Missing, wrong or revoked key | Don't retry. Get a valid key |
| 413 | `request_too_large` / `invalid_image` | Body over 25 MB, or an image over 10 MB | Shrink or compress the input |
| 429 | `rate_limit_error` | Queue full | Retry after 2–10 s with exponential backoff |
| 429 | `rate_limit_exceeded` (type `rate_limit_error`) | Usage limit reached. `message` gives the reset time (ISO 8601) and the `Retry-After` header gives seconds until reset | Don't retry before `Retry-After`. Requests fail immediately until then. Tell the user, or wait |
| 502 | `invalid_model_output` | The model didn't produce valid tool-call/JSON output, even after a retry | Retry once, or simplify the schema |
| 503 | `backend_unavailable` | Backend temporarily unavailable | Retry with backoff (up to about 3 times), then report |
| 504 | `timeout` | Took longer than 5 min | Retry with a smaller task or use `fast` |

If a stream fails mid-response, an `error` object is sent as a `data:` event, followed by `data: [DONE]`.

## Quick self-test

```bash
export GATEWAY_API_KEY=...   # your key
B=https://llm.hisavage.com/v1
curl -s -o /dev/null -w "%{http_code}\n" $B/models                                   # expect 401
curl -s -H "Authorization: Bearer $GATEWAY_API_KEY" $B/models                        # expect model list
curl -s -H "Authorization: Bearer $GATEWAY_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"fast","messages":[{"role":"user","content":"Reply with: ok"}]}' $B/chat/completions
```

The last call should return `choices[0].message.content` of about "ok", with `"model":"fast"`.