​

Serverless Inference

• Develop AI applications and agents without signing up for a hosting provider or self-hosting a model.
• Try the supported models in the Weave Playground.

• Prerequisites
  • Additional prerequisites for using the API via Python
• API specification
  • Endpoint
  • Available methods
    • Chat completions
    • List supported models
• Usage examples
• UI
  • Access the Inference service
  • Try a model in the Playground
  • Compare multiple models
  • View billing and usage information
• Usage information and limits
• API errors

​

Prerequisites

1. A W&B account. Sign up for an account.
2. A W&B API key. Create an API key at User Settings.
3. A W&B project.
4. If you’re using the Inference service through Python, see Additional prerequisites for using the API via Python.

​

Additional prerequisites for using the API via Python

pip install openai weave

​

API specification

• Endpoint
• Available methods
• Usage examples

​

Endpoint

https://api.inference.wandb.ai/v1

​

Available methods

• Chat completions
• List supported models

​

Chat completions

• The Inference service base URL https://api.inference.wandb.ai/v1
• Your W&B API key <your-api-key>
• Your W&B entity and project names <your-team>/<your-project>
• The ID for the model you want to use, one of:
  • meta-llama/Llama-3.1-8B-Instruct
  • deepseek-ai/DeepSeek-V3-0324
  • meta-llama/Llama-3.3-70B-Instruct
  • deepseek-ai/DeepSeek-R1-0528
  • meta-llama/Llama-4-Scout-17B-16E-Instruct
  • microsoft/Phi-4-mini-instruct

curl https://api.inference.wandb.ai/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your-api-key>" \
  -H "OpenAI-Project: <your-team>/<your-project>" \
  -d '{
    "model": "<model-id>",
    "messages": [
      { "role": "system", "content": "You are a helpful assistant." },
      { "role": "user", "content": "Tell me a joke." }
    ]
  }'

import openai

client = openai.OpenAI(
    # The custom base URL points to Serverless Inference
    base_url='https://api.inference.wandb.ai/v1',

    # Create an API key at https://wandb.ai/settings
    # Consider setting it in the environment as OPENAI_API_KEY instead for safety
    api_key="<your-api-key>",

    # Team and project are required for usage tracking
    project="<your-team>/<your-project>",
)

# Replace <model-id> with any of the following values:
# meta-llama/Llama-3.1-8B-Instruct
# deepseek-ai/DeepSeek-V3-0324
# meta-llama/Llama-3.3-70B-Instruct
# deepseek-ai/DeepSeek-R1-0528
# meta-llama/Llama-4-Scout-17B-16E-Instruct
# microsoft/Phi-4-mini-instruct

response = client.chat.completions.create(
    model="<model-id>",
    messages=[
        {"role": "system", "content": "<your-system-prompt>"},
        {"role": "user", "content": "<your-prompt>"}
    ],
)

print(response.choices[0].message.content)

​

List supported models

curl https://api.inference.wandb.ai/v1/models \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <your-api-key>" \
  -H "OpenAI-Project: <your-team>/<your-project>" \

import openai

client = openai.OpenAI(
    base_url="https://api.inference.wandb.ai/v1",
    api_key="<your-api-key>",
    project="<your-team>/<your-project>"
)

response = client.models.list()

for model in response.data:
    print(model.id)

​

Usage examples

• Basic example: Trace Llama 3.1 8B with Weave
• Advanced example: Use Weave Evaluations and Leaderboards with the Inference service

​

Basic example: Trace Llama 3.1 8B with Weave

• You define a @weave.op()-decorated function, run_chat, which makes a chat completion request using the OpenAI-compatible client.
• Weave records your traces and associates them with your W&B entity and project project="<your-team>/<your-project>".
• Weave automatically traces the function, logging its inputs, outputs, latency, and metadata (like model ID).
• The result prints in the terminal, and the trace appears in your Traces tab at https://wandb.ai under the specified project.

import weave
import openai

# Set the Weave team and project for tracing
weave.init("<your-team>/<your-project>")

client = openai.OpenAI(
    base_url='https://api.inference.wandb.ai/v1',

    # Create an API key at https://wandb.ai/settings
    api_key="<your-api-key>",

    # Required for W&B inference usage tracking
    project="wandb/inference-demo",
)

# Trace the model call in Weave
@weave.op()
def run_chat():
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.1-8B-Instruct",
        messages=[
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Tell me a joke."}
        ],
    )
    return response.choices[0].message.content

# Run and log the traced call
output = run_chat()
print(output)

1. Navigate to https://wandb.ai.
2. Select the Traces tab to view your Weave traces.

​

Advanced example: Use Weave Evaluations and Leaderboards with the Inference service

import os
import asyncio
import openai
import weave
from weave.flow import leaderboard
from weave.trace.ref_util import get_ref

# Set the Weave team and project for tracing
weave.init("<your-team>/<your-project>")

dataset = [
    {"input": "What is 2 + 2?", "target": "4"},
    {"input": "Name a primary color.", "target": "red"},
]

@weave.op
def exact_match(target: str, output: str) -> float:
    return float(target.strip().lower() == output.strip().lower())

class WBInferenceModel(weave.Model):
    model: str

    @weave.op
    def predict(self, prompt: str) -> str:
        client = openai.OpenAI(
            base_url="https://api.inference.wandb.ai/v1",
            # Create an API key at https://wandb.ai/settings
            api_key="<your-api-key>",
            # Required for W&B inference usage tracking
            project="<your-team>/<your-project>",
        )
        resp = client.chat.completions.create(
            model=self.model,
            messages=[{"role": "user", "content": prompt}],
        )
        return resp.choices[0].message.content

llama = WBInferenceModel(model="meta-llama/Llama-3.1-8B-Instruct")
deepseek = WBInferenceModel(model="deepseek-ai/DeepSeek-V3-0324")

def preprocess_model_input(example):
    return {"prompt": example["input"]}

evaluation = weave.Evaluation(
    name="QA",
    dataset=dataset,
    scorers=[exact_match],
    preprocess_model_input=preprocess_model_input,
)

async def run_eval():
    await evaluation.evaluate(llama)
    await evaluation.evaluate(deepseek)

asyncio.run(run_eval())

spec = leaderboard.Leaderboard(
    name="Inference Leaderboard",
    description="Compare models on a QA dataset",
    columns=[
        leaderboard.LeaderboardColumn(
            evaluation_object_ref=get_ref(evaluation).uri(),
            scorer_name="exact_match",
            summary_metric_path="mean",
        )
    ],
)

weave.publish(spec)

• Navigate to the Traces tab to view your traces.
• Navigate to the Evals tab to view your model evaluations.
• Navigate to the Leaders tab to view the generated leaderboard.

​

UI

​

Access the Inference service

• Direct link
• From the Inference tab
• From the Playground tab

​

Direct link

​

From the Inference tab

1. Navigate to your W&B account at https://wandb.ai/.
2. From the left sidebar, select Inference. A page with available models and model information displays.

​

From the Playground tab

1. From the left sidebar, select Playground. The Playground chat UI displays.
2. From the LLM dropdown list, hover over Serverless Inference. A dropdown with available Serverless Inference models displays to the right.
3. From the Serverless Inference models dropdown, you can:
   • Click the name of any available model to try it in the Playground.
   • Compare one or more models in the Playground.

​

Try a model in the Playground

• Customize model settings and parameters
• Add, retry, edit, and delete messages
• Save and reuse a model with custom settings
• Compare multiple models

​

Compare multiple models

• Access the Compare view from the Inference tab
• Access the Compare view from the Playground tab

​

Access the Compare view from the Inference tab

1. From the left sidebar, select Inference. A page with available models and model information displays.
2. To select models for comparison, click anywhere on a model card (except for the model name). The border of the model card highlights in blue to indicate the selection.
3. Repeat step 2 for each model you want to compare.
4. In any of the selected cards, click the Compare N models in the Playground button (N is the number of models you’re comparing. For example, when you select 3 models, the button displays as Compare 3 models in the Playground). The comparison view opens.

​

Access the Compare view from the Playground tab

1. From the left sidebar, select Playground. The Playground chat UI displays.
2. From the LLM dropdown list, hover over Serverless Inference. A dropdown with available Serverless Inference models displays to the right.
3. From the dropdown, select Compare. The Inference tab displays.
4. To select models for comparison, click anywhere on a model card (except for the model name). The border of the model card highlights in blue to indicate the selection.
5. Repeat step 4 for each model you want to compare.
6. In any of the selected cards, click the Compare N models in the Playground button (N is the number of models you’re comparing. For example, when you select 3 models, the button displays as Compare 3 models in the Playground). The comparison view opens.

​

View billing and usage information

1. In the W&B UI, navigate to the W&B Billing page.
2. In the bottom right corner, the Inference billing information card displays. From here, you can:
   • Click the View usage button in the Inference billing information card to view your usage over time.
   • If you’re on a paid plan, view your upcoming inference charges.

​

Usage information and limits

​

Geographic restrictions

​

Concurrency limits

• Prevent misuse and protect API stability.
• Ensure access for all users.
• Manage infrastructure load effectively.

​

Pricing

​

API errors