The Decisions API Is Generally Available: When a Bounded Answer Beats a Chat Completion

What the endpoint actually returns

Requests go to POST /v1/decisions and use gpt-6-luna, the only supported model at launch. The shape is simple: a shared input (text, images, or JSON), then an array of questions. Each question has a type. A predicate answers "is this condition true" with a probability between 0 and 1. A choice picks one of the values you supplied, and also returns a distribution plus a separate confidence field. A score rates an input against ordered levels and returns the probability-weighted average of the level indices, which is why a severity score can land between levels instead of snapping to an integer.

There is a refusal answer type, and how you handle it is the difference between a careful integration and an outage. A refusal is not a category. If your router treats every non-null answer as a route, a refusal becomes whatever the default branch happens to be. Handle it explicitly, as in the example below, and send it to a human queue.

The boundary with the rest of the API is worth stating plainly, because teams get it wrong in both directions. When you need generated prose, extracted fields, or an object matching your own schema, that is Responses with Structured Outputs. When you need a model to choose a tool and emit arguments, that is function calling. Decisions is for the third case: a bounded question asked thousands of times where latency and cost dominate.

# POST /v1/decisions, model gpt-6-luna (the only supported model at GA).
# The shape is deliberately boring: shared input, then questions with bounded answers.

curl https://api.openai.com/v1/decisions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-6-luna",
    "input": "I was charged twice for my order and nobody replied to my email.",
    "questions": [{
      "type": "choice",
      "name": "department",
      "instructions": "Which department should handle this complaint?",
      "choices": [
        {"value": "billing", "description": "Payments, invoices, and refunds."},
        {"value": "technical", "description": "Problems using the product."},
        {"value": "shipping", "description": "Delivery and tracking."},
        {"value": "other", "description": "Requests outside these categories."}
      ]
    }]
  }'

# Python equivalent. Note what you get back: an answers array whose entries are
# typed. A choice answer carries the chosen value, the distribution, and a
# separate confidence number. Handle the refusal type explicitly or your router
# will silently treat a refusal as a category.
from openai import OpenAI

client = OpenAI()
decision = client.decisions.create(
    model="gpt-6-luna",
    input="I was charged twice for my order and nobody replied to my email.",
    questions=[{
        "type": "choice",
        "name": "department",
        "instructions": "Which department should handle this complaint?",
        "choices": [
            {"value": "billing", "description": "Payments, invoices, and refunds."},
            {"value": "technical", "description": "Problems using the product."},
            {"value": "shipping", "description": "Delivery and tracking."},
            {"value": "other", "description": "Requests outside these categories."},
        ],
    }],
)

answer = decision.answers[0]
if answer.type == "refusal":
    route = "human_queue"
elif answer.type == "choice":
    route = answer.choice if answer.confidence > 0.6 else "human_queue"
    print(f"route={route} confidence={answer.confidence:.3f}")

Do the arithmetic before the architecture

Billing is input-only: $0.10 per million input tokens, $0.01 per million cached input tokens, and no charge for output tokens. That last detail matters more than the headline number. In a chat completion, the answer you pay for is generated token by token, and a verbose run of reasoning can cost more than the prompt that triggered it. Here there is no generated answer to pay for.

Caching is what turns the price from cheap into strange. The API places an implicit cache breakpoint at the end of the shared input, and a reusable prefix has to reach at least 1,024 tokens. Content inside questions, including question-specific images, sits outside that prefix, so a long policy document stays cheap while a per-request variant does not. You can partition reuse with prompt_cache_key, or mark boundaries yourself with an explicit mode, but the default behaviour is what most teams should measure first.

Check the real numbers rather than trusting the model of them: usage.input_tokens_details.cached_tokens and cache_write_tokens are both reported, and both are already included inside usage.input_tokens. Adding them to the total is a common reporting bug that makes dashboards look 20% wrong.

# Price the workload before you build the retry logic. Decisions bills input only:
# $0.10 per million input tokens, $0.01 per million cached input tokens, and no
# output-token charge. That last part changes the arithmetic of small, frequent calls.

def monthly_cost(calls_per_day, tokens_per_call, cache_hit_rate, cached_tokens=1024):
    billed_prompt = tokens_per_call - (cached_tokens * cache_hit_rate)
    billed_prompt = max(billed_prompt, 0)
    uncached = billed_prompt * (1 - cache_hit_rate)
    cached = billed_prompt * cache_hit_rate
    per_call = (uncached * 0.10 + cached * 0.01) / 1_000_000
    return round(calls_per_day * 30 * per_call, 2)

# A support router at 200k calls/day with a 2,000-token prompt costs about $1.1k
# a month with no cache, and about $90 a month if the shared prefix is cached and
# your prompt is mostly shared. Same token budget, different order of magnitude.
print("no cache   ", monthly_cost(200_000, 2_000, 0.0))
print("half cached", monthly_cost(200_000, 2_000, 0.5))
print("all cached ", monthly_cost(200_000, 2_000, 0.95))

# Caching has a floor worth remembering: the reusable prefix must be at least
# 1,024 tokens, and question content never counts toward that threshold.

The evaluation that decides whether you ship

The commercial argument for switching is latency and price. The engineering argument against switching is silent disagreement. A router that is 90% cheaper and 3% wrong is not cheaper, it is a support queue with extra steps. So the harness below does something deliberately unglamorous: it runs the same labeled rows through both paths and reports agreement, refusals, and mean latency.

Read the output in that order. Agreement tells you whether the cheap path is a substitute. Refusals tell you how often you still need a human, which is a cost you should model in the same spreadsheet as tokens. Latency tells you whether the user-visible win is real or whether your own network hop dominates it. Vendor benchmarks are averages over someone else's traffic; your labeled set is the only one that describes yours.

Then read twenty disagreements by hand. They cluster. Ours usually fall into two shapes: an input that genuinely belongs to two categories, and a category description that two people on the team would define differently. The first needs a threshold. The second needs you to rewrite the choice descriptions, which is a five-minute fix that no amount of prompt engineering on a chat model would have surfaced.

# An evaluation harness that answers the only question that matters: does the
# cheap path disagree with the expensive path in a way you can live with?

import json, time
from openai import OpenAI

client = OpenAI()

def as_decisions(text, choices):
    d = client.decisions.create(
        model="gpt-6-luna",
        input=text,
        questions=[{"type": "choice", "name": "label",
                    "instructions": "Pick the correct label.",
                    "choices": [{"value": c} for c in choices]}],
    )
    a = d.answers[0]
    return None if a.type == "refusal" else a.choice

def as_responses(text, choices):
    r = client.responses.create(
        model="gpt-6-luna",
        input=f"Reply with exactly one of: {', '.join(choices)}\n\n{text}",
    )
    return r.output_text.strip().lower()

labelled = [json.loads(line) for line in open("labelled.jsonl")]
agree = refusals = 0
t_dec = t_res = 0.0

for row in labelled:
    t0 = time.perf_counter()
    d = as_decisions(row["text"], row["choices"])
    t_dec += time.perf_counter() - t0

    t0 = time.perf_counter()
    r = as_responses(row["text"], row["choices"])
    t_res += time.perf_counter() - t0

    refusals += d is None
    agree += (d == r)

n = len(labelled)
print(f"agreement {agree/n:.1%}  refusals {refusals/n:.1%}")
print(f"p50-ish mean latency decisions {t_dec/n*1000:.0f} ms vs responses {t_res/n*1000:.0f} ms")

# Disagreement is not automatically failure. Print the rows where they differ and
# read twenty of them by hand; that is where the threshold you need actually lives.

Run the old path and the new path over the same labeled rows before you move traffic.

Putting it in the hot path without losing the guardrails

The pattern that holds up is two gates, not one. The first asks the routing question. The second asks whether this request is allowed to be routed automatically at all — a predicate like "clearly within a standard category, with no legal, safety, or account-deletion implications". Only when both the predicate probability and the choice confidence clear your floors does the answer act.

This is where Decisions earns its keep inside agent loops. The published customer numbers are all about placing a decision in a place that was previously too slow to tolerate a model: Box reports roughly 5x faster route selection on 30 certificate-of-insurance cases at the same accuracy as the Responses API, Datadog scores agent-generated shell commands before execution with a 40x faster p95, Ramp attributes AI spend 4x faster and 30% cheaper, and Sierra runs orchestration choices under 280 ms p95 at about 90% lower cost per call.

Treat those as existence proofs, not benchmarks. Every one of them ran on that customer's data with that customer's thresholds. What generalises is the pattern: cheap bounded decisions on the inside of a loop, and an explicit escape hatch to a bigger model or a human when the bounded answer is not confident.

# Wire it into the hot path with two rules: a confidence floor, and a predicate
# that decides whether the cheap answer is even allowed to act.

def should_auto_route(decision):
    gate = client.decisions.create(
        model="gpt-6-luna",
        input=decision["text"],
        questions=[{"type": "predicate", "name": "safe_to_auto",
                    "instructions": ("Is this request clearly within a standard "
                                     "category, with no legal, safety, or account "
                                     "deletion implications?")}],
    )
    p = gate.answers[0].probability
    return p >= 0.9 and decision["confidence"] >= 0.7

# The important line is the one you do not write: there is no fallback that
# quietly defaults to a category. Low probability means a human or a bigger model,
# not a coin flip.

def route(ticket):
    d = client.decisions.create(
        model="gpt-6-luna",
        input=ticket["text"],
        questions=[{"type": "choice", "name": "queue",
                    "instructions": "Which queue owns this ticket?",
                    "choices": [{"value": "billing"}, {"value": "technical"},
                                {"value": "shipping"}, {"value": "other"}]}],
    )
    a = d.answers[0]
    if a.type == "refusal":
        return "human_queue"
    entry = {"text": ticket["text"], "choice": a.choice, "confidence": a.confidence}
    return a.choice if should_auto_route(entry) else "human_review"

A rollout checklist you can defend in review

Four checks before production, all cheap. First, endpoint sanity: one predicate call in CI, so a credential or model-name change fails the build instead of the router. Second, cache verification: three identical requests with a shared prefix above 1,024 tokens, printing cached_tokens each time. If the number stays at zero, your prefix is not identical, and your cost model is fiction.

Third, a budget guard expressed as tokens per call rather than a monthly total nobody reads. Fourth, a rollback you can execute in one command. A pilot with a one-command rollback is an experiment; a migration without one is a bet, and bets belong in a strategy document.

One caveat to keep visible. gpt-6-luna is the only supported model today, which means your latency and quality are tied to a single model's roadmap. Price that in the same way you price a vendor lock anywhere else: keep the routing question and its answer schema in your own configuration, so changing the implementation is a config change rather than a rewrite. Model endpoints change faster than routing logic does.

#!/usr/bin/env bash
# Pre-flight checks. Attach the output to the pull request instead of saying
# "it felt faster in the playground".
set -euo pipefail

echo "== endpoint sanity =="
test -n "${OPENAI_API_KEY:-}" || { echo "no key"; exit 1; }
curl -s https://api.openai.com/v1/decisions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-6-luna","input":"ping","questions":[{"type":"predicate","name":"ok","instructions":"Is this input short?"}]}' \
  | python -c 'import json,sys; d=json.load(sys.stdin); print(d["answers"][0])'

echo "== cache actually hitting? =="
python - <<'PY'
from openai import OpenAI
c = OpenAI()
shared = "policy text " * 400          # >1,024 tokens, identical every request
for i in range(3):
    r = c.decisions.create(
        model="gpt-6-luna",
        input=shared,
        questions=[{"type": "predicate", "name": "p", "instructions": "Is this policy text?"}],
    )
    u = r.usage.input_tokens_details
    print(i, "in", r.usage.input_tokens, "cached", u.cached_tokens, "written", u.cache_write_tokens)
PY

echo "== budget guard =="
python - <<'PY'
budget = 250          # currency units per month
calls_per_month = 200_000
cap_per_call = budget / calls_per_month
print(f"cap per call: {cap_per_call:.6f} -> keep prompts under ~{cap_per_call/0.10*1_000_000:,.0f} tokens")
PY

echo "== rollback =="
test -n "${LEGACY_ROUTE_COMMAND:-}" || echo "no documented fallback -> stop here"

A pilot with a one-command rollback is an experiment. A migration without one is a bet.